JP6802923B2 - Object detection device and object detection method - Google Patents

Description

The present invention is an object detection that realizes robust object detection even when the installation state of the camera changes or the appearance of the detection target changes due to the movement of the camera and the detection target. The present invention relates to an apparatus and an object detection method.

There is a great need for an object detection technology that detects an object to be detected (for example, a person, cargo, vehicle, etc.) from image information acquired by an imaging device such as a surveillance camera. As a general object detection technique, a background image in which the object to be detected does not exist is prepared in advance, and by comparing the input captured image with the background image, background subtraction for detecting the object and a frame of the image are obtained. There is an optical flow that detects a moving object by the difference of feature points between them. However, in these methods, since all moving objects in the image are detected, it is not possible to detect only a specific target in the image, for example.

Therefore, there is a need for a technique for detecting a specific object by using the contour information of the object and the appearance information such as the color and shape that can be read from the appearance.

For example, in Patent Document 1, paragraph 0034 states, "A person is determined from the contour information obtained from the feature amount based on the appearance by HOG, and the foreground (moving state or static state) is determined from the feature amount based on the spatiotemporal feature by the pixel state analysis. An identification unit that "determines an image determined to be in a state) from a person is described. Including this description, Patent Document 1 describes a means for extracting contour information of a person from a learning sample consisting of an image including a person and an image not including the person, and generating a classifier for distinguishing between the person and a person other than the person. A technique for realizing person detection is disclosed by using a means for determining whether or not a person exists in a predetermined area on an image using the above.

Further, in Patent Document 2, paragraph 0016 states, "The deformation detection region 2100 on the surveillance image 2000 reflects information on each parameter of the camera device, and as shown in FIG. 3, the distortion of the surveillance image 2000 is taken into consideration. The object recognition device 1 (1a) extracts the feature amount of the image information 100 of the deformation detection area 2100 created as the area including the recognition target deformed by distortion or the like, and extracts the feature amount of the object to be recognized. Whether or not it is determined. " Including this description, in Patent Document 2, as an applied technique of Patent Document 1, a person exists by using a discriminator on the assumption that the detection target on the image is deformed by the influence of the lens distortion peculiar to the camera. A technique for improving the detection rate by deforming a predetermined area to be input before determining whether or not to use the device is disclosed.

Japanese Unexamined Patent Publication No. 2009-181220 Japanese Unexamined Patent Publication No. 2012-22437

In Patent Document 1, a person image of a specific posture (for example, an image of an upright posture taken from the front) is trained by a classifier as a learning sample, and the person is detected using this classifier to detect a person in a specific posture. The rate is increasing.

However, in the actually captured image, the posture (appearance) of the person changes greatly due to the relative positional relationship between the camera device and the person and the lens distortion of the camera device, so the learning sample and the person in the captured image If the contour information is different, the classifier of Patent Document 1 has a problem that the accuracy of person detection is lowered.

Further, in Patent Document 2, as shown in FIG. 13 and the like of the same document, the contour information of the person to be input to the classifier is specified in advance from the parameter information of the camera device and the positional relationship between the detection target and the camera device. By transforming (normalizing) the posture so that it becomes the same as the posture of the person, the detection rate by the discriminator can be maintained even when the posture of the person changes within a certain range.

However, when the appearance of the detection target is significantly different from the assumption, or when a part of the detection target is hidden behind a shield, the detection rate is significantly reduced by the object recognition method of Patent Document 2. There is a problem. For example, even if a person can be easily detected from an image that includes all of the person's head, arms, body, and legs, an image of the person taken from directly above or the lower half of the body is hidden by a shield. When an image of a person is used, the person detection rate in the image is significantly reduced by the classifier of Patent Document 2 because the leg cannot be detected from the image.

In order to solve such a problem, in the present invention, when a photographed image including a person in a posture that does not correspond to the classifier or an image captured in a state where a part of the human body is hidden in the shadow of an obstacle is used. Another object of the present invention is to provide an object detection device capable of realizing highly accurate person detection.

The object detection device according to the present invention is an object detection device that determines whether or not a detection target exists within the measurement range, and acquires three-dimensional information within the measurement range based on an input from the image pickup device. A three-dimensional information acquisition unit, an identification candidate area extraction unit that extracts an identification candidate area in which the detection target may exist, a classifier used for detecting the detection target, and a classifier information acquisition that acquires information on the classifier. Based on the unit, the image conversion method determination unit that determines the parameters for virtually performing viewpoint conversion processing on the three-dimensional information in the identification candidate area, and the three-dimensional information in the identification candidate area that has been virtually subjected to viewpoint conversion processing. An image conversion execution unit that generates a converted image, an identification unit that detects the detection target using the classifier based on the converted image, and image information within the measurement range based on the input from the imaging device. The image conversion method determining unit includes an image acquisition unit to be acquired, and the image conversion method determining unit generates the converted image optimal as an input of the classifier by using the image information, the three-dimensional information, and the information of the classifier. was shall to determine the parameters to be.

According to the object detection device of the present invention, even when an image in which the relative position of the camera device and the object is significantly different from the assumption or an image in which a part of the object is shielded is used, the object to be detected can be detected with high accuracy. Can be detected.

It is a figure which shows the structural example of the object detection apparatus of Example 1. FIG. It is a figure which shows the detail of the identification candidate area extraction part of Example 1. FIG. It is a figure which shows the detail of the identification candidate area information management part of Example 1. FIG. It is a figure which shows the identification candidate area in a two-dimensional image. It is a figure which shows the identification candidate area in a three-dimensional photographing space. It is a figure which shows the detail of the classifier of Example 1. FIG. It is a figure which shows the detail of the image conversion method determination part of Example 1. FIG. It is a figure explaining the processing content of the viewpoint conversion part of Example 1. FIG. It is a figure explaining the effect of the image conversion method determination part. It is a figure explaining the effect of the image conversion method determination part. It is a figure which shows the detail of the identification part of the configuration example of Example 1. FIG. It is a figure which shows the processing flow example in Example 1. FIG. It is a figure which shows the structural example of the object detection apparatus of Example 2. It is a figure explaining the process of the image conversion method determination part of Example 2. FIG. It is a figure explaining the processing flow of the image conversion method determination part of Example 2. FIG. It is a figure explaining the detail of the processing flow of FIG.

Hereinafter, examples of the present invention will be described in detail with reference to the drawings as appropriate. In the following, an example in which the detection target is a person will be described, but the detection target is not limited to the person, and may be a cargo, a vehicle, or the like. Further, an example of detecting a detection target from image information taken by an image pickup device such as a camera will be described, but the information including the detection target is not limited to the image information taken by the image pickup device, and is a heat map acquired by a thermo sensor. There may be.

The object detection device 2a of the first embodiment will be described with reference to FIGS. 1 to 9.

FIG. 1 is a block diagram showing an outline of an object detection device 2a of the present embodiment connected to an image pickup device 1 such as a stereo camera. The object detection device 2a realizes robust detection of the detection target even when the appearance of the detection target on the captured image of the image pickup device 1 changes due to a change in the relative position between the image pickup device 1 and the detection target. It is an object detection device.

In the object detection device 2a shown in FIG. 1, 3 is an image acquisition unit that acquires image information within the measurement range based on the input from the image pickup device 1, and 4 is a tertiary within the measurement range based on the input from the image pickup device 1. A three-dimensional information acquisition unit that acquires original information, 5 is an identification candidate area extraction unit that extracts an identification candidate area, which is an area in which a detection target can exist by using image information and three-dimensional information, from a measurement range, and 6 is an object. The classifier information acquisition unit that acquires the information of the classifier 64 used in the detection device 2a, 7a determines a method of converting the discrimination candidate area into the optimum image as the input of the classifier 64 using the information of the classifier. 8 is an image conversion unit that acquires a converted image from an identification candidate area based on the determined image conversion method, and 9 is an identification unit that determines whether or not a detection target is included in the converted image. is there. A part or all of the image acquisition unit 3 to the identification unit 9 does not necessarily have to be dedicated hardware, and is stored in a program stored in a main storage device such as a semiconductor memory or in an auxiliary storage device such as a hard disk. It may be realized by processing the data in a computing device such as a CPU.

Hereinafter, the image pickup device 1, the identification candidate region extraction unit 5, the classifier information acquisition unit 6, the image conversion method determination unit 7a, the image conversion unit 8, and the identification unit 9 shown in FIG. 1 will be described in detail individually.
<Imaging device>
The image pickup apparatus 1 is an apparatus capable of acquiring image information and three-dimensional information of a measurement range. Here, the image information is the brightness information in the digital image data, and the three-dimensional information is the coordinate information of the three-dimensional point cloud in the measurement range (three-dimensional space).

The image pickup device 1 may be a stereo camera composed of two or more cameras, or a combination of one camera and a distance sensor capable of acquiring three-dimensional information. For example, a stereo camera measures the distance from a camera to an object by using the principle of triangulation by photographing the same object with two or more cameras, and both image information and three-dimensional information. Can be obtained. In addition, the distance sensor measures the distance to the target by calculating the time it takes for the projected light to be reflected by the target and return to the distance sensor from the phase difference between the projected light and the reflected light. By combining with a matched camera, it is possible to obtain the three-dimensional information and the image information in association with each other.
<Identification candidate area extraction unit>
FIG. 2 shows the details of the identification candidate region extraction unit 5. The identification candidate area extraction unit 5 extracts the identification candidate area 55 in which the detection target may exist by using the image information and / or the three-dimensional information acquired by the image acquisition unit 3 and the three-dimensional information acquisition unit 4. The image processing unit 51 that extracts the identification candidate area 55 using the image information, the three-dimensional information processing unit 52 that extracts the identification candidate area 55 using the three-dimensional information, and one or more extracted identification candidates. It includes an identification candidate area ID assigning unit 53 that assigns an ID to the area 55, and an identification candidate area information management unit 54 that acquires and manages identification candidate area information indicating the position of the identification candidate area 55. Hereinafter, the image processing unit 51, the three-dimensional information processing unit 52, the identification candidate area ID assigning unit 53, and the identification candidate area information management unit 54 will be described in detail.

The image processing unit 51 extracts the identification candidate region 55 by performing image processing on the image information acquired by the image pickup apparatus 1. As the image processing executed here, for example, there is a background subtraction in which a background image obtained by shooting a shooting space in a state where a detection target does not exist is acquired in advance and the difference between the background image and the shot image is calculated. However, the method is not particularly limited as long as it is a means capable of extracting a region to be detected by image information, such as detection using color information such as skin color detection.

The three-dimensional information processing unit 52 extracts the identification candidate region 55 by performing three-dimensional processing on the three-dimensional information acquired by the imaging device 1. As the three-dimensional processing executed here, for example, the background three-dimensional information of the shooting space in the state where the detection target does not exist is acquired in advance, and the difference between the background three-dimensional information and the newly acquired three-dimensional information is obtained. There is a method of calculating, but it is not particularly limited as long as the identification candidate area 55 is acquired by performing three-dimensional processing.

Next, the identification candidate area ID assigning unit 53 and the identification candidate area information management unit 54 will be described with reference to FIGS. 3A to 3C.

The identification candidate area ID assigning unit 53 assigns an ID to each of the identification candidate areas 55 extracted by the image processing unit 51 and the three-dimensional information processing unit 52. Further, the identification candidate area information management unit 54 adds the position information of the identification candidate area to the ID and manages it as the identification candidate area information 54_n. The position information is an image position indicating the start point and the end point in the two-dimensional image of the identification candidate area, and a three-dimensional position indicating the start point and the end point of the identification candidate area in the three-dimensional photographing space.

FIG. 3A exemplifies n identification candidate area information 54_n managed by the identification candidate area information management unit 54, and each identification candidate area information 54_n includes an ID and a corresponding identification candidate area 55. It shows that the image position and the three-dimensional position, which are the position information, are recorded. FIG. 3B specifically shows the image position of the identification candidate area information 54_1, and 56a and 56b are the start point (x1, y1) and the end point (x1) of the rectangular identification candidate area 55 in the captured image of the image pickup apparatus 1. ', y1') is shown. Similarly, FIG. 3C specifically shows the three-dimensional position of the identification candidate region information 54_1, and 57a and 57b are the start point (X1, Y1, Z1) and the end point (X1) of the rectangular parallelepiped identification candidate region 55. ', Y1', Z1') is shown. Although the rectangular parallelepiped identification candidate region 55 is illustrated in FIGS. 3B and 3C, an identification candidate region having another shape may be used as long as the position of the identification candidate region 55 can be specified. In this case, it goes without saying that the information on the image position and the three-dimensional position in FIG. 3A is also expressed according to the identification candidate region of the other shape.
<Identifier information acquisition unit>
Next, the classifier information acquisition unit 6 will be described with reference to FIG. The classifier information acquisition unit 6 selects an appropriate classifier 64 from a plurality of classifiers 64, and extracts the classifier information 65 corresponding thereto. In addition, 67_n is a classifier ID given to manage the classifier 64_n.

The classifier 64 is used for a discrimination process for discriminating whether or not a detection target is included in the captured image of the image pickup apparatus 1, and each of the classifier 64_n has a high discriminating ability for a detection target having a different posture. is there. By learning a large number of images including the detection target and images not including the detection target (learning samples) by the machine learning method, each classifier 64_n can have different characteristics. As the machine learning method, Support Vector Machine is generally used, but other machine learning methods may be used.

The classifier information 65_n indicates an input image in which the classifier 64_n exerts a particularly high discriminating ability. In FIG. 4, as the classifier information, a template 66_1 that is strong in identifying a front-viewed person image, a template 66_2 that is strong in identifying a top-viewed person image, and a template 66_n that is strong in identifying a side-viewed person image are illustrated. , Color information, feature quantity expressing contour, brightness information, gradient information, etc., if it is an image suitable for input of classifier 64_n or classifier information expressing a method of generating an image, record other information. You can keep it.
<Image conversion method determination unit>
Next, the processing flow of the image conversion method determination unit 7a will be described with reference to FIG. The image conversion method determination unit 7a is a conversion method (parameters, etc.) for converting to an optimum image as an input to the classifier 64 based on the three-dimensional information in the rectangular parallelepiped identification candidate region 55 illustrated in FIG. 3C. Is what determines. As the processing flow of the image conversion method determination unit 7a, first, the viewpoint conversion parameters are determined (S51), and the viewpoint conversion image is generated using the parameters (S52). Then, the similarity with the converted image is calculated with reference to the classifier information 65 held by each of the plurality of classifiers 64 (S53), and if the similarity is higher than the threshold value, the process is terminated, and if it is less than the threshold value, step S51. Return to and change the parameter to another value (S54). Hereinafter, steps S51, S52, S53, and S54 will be described in detail.

In step S51, the parameters α, β, and γ necessary for generating the viewpoint-transformed image are determined. The details of each parameter will be described in detail later. As a method for determining the parameters α, β, and γ in step S51, there is a method of comprehensively varying the parameters.

In step S52, the process shown in FIG. 6 is performed. In FIG. 6, 82 is a viewpoint for observing the identification candidate region 55, 83, 84, and 85 are created by viewpoint conversion in the x-axis, y-axis, z-axis, 86_1, and 86_2 in the coordinate system of the three-dimensional space set in the measurement range. An example of the converted image to be performed is shown. In step S52, using the parameters α, β, and γ determined in step S51, the three-dimensional information included in the rectangular parallelepiped identification candidate region 55 is divided into α centered on the x-axis 83 and β centered on the y-axis 84. The converted image 86 is acquired by rotating the z-axis 85 by γ to convert the viewpoint to the state observed from an arbitrary viewpoint and projecting the identification candidate region 55 after the viewpoint conversion onto the image.

As a method of viewpoint conversion, a conversion formula such as Equations 1 to 3 is generally used, but other viewpoint conversion methods may be used.

識別候補領域５５を変換画像86_nに投影する方法としては、透視投影が一般的な方法であるが、他の方法を用いても良い。例えば、直立する人物の三次元情報を含む識別候補領域５５を、視点８２の方向に設置した撮像装置１で撮影した場合、視点変換せずに、識別候補領域５５を投影することで、人物を上面視した変換画像86_1を取得できる。これに対し、同じ撮像装置１で撮影した識別候補領域５５を、α＝０°、β＝０°、γ＝９０°だけ回転する視点変換を実施し、視点８２に対して識別候補領域５５を画像に投影すると、人物を側面視した変換画像86_2を取得できる。

Perspective projection is a general method for projecting the identification candidate region 55 onto the converted image 86_n, but other methods may be used. For example, when the identification candidate area 55 including the three-dimensional information of an upright person is photographed by the imaging device 1 installed in the direction of the viewpoint 82, the person is projected by projecting the identification candidate area 55 without changing the viewpoint. The converted image 86_1 viewed from above can be obtained. On the other hand, the identification candidate area 55 captured by the same image pickup apparatus 1 is subjected to viewpoint conversion in which the identification candidate area 55 is rotated by α = 0 °, β = 0 °, and γ = 90 °, and the identification candidate area 55 is set with respect to the viewpoint 82. When projected onto an image, a converted image 86_2 with a side view of a person can be obtained.

Further, in step S53 of FIG. 5, the optimization process is performed to determine the image conversion method to the image most suitable for the classifier 64. As a method of determining the image conversion method, for example, the template 66 is acquired with reference to the classifier information 65, and the degree of similarity with the converted image 86_n acquired by performing viewpoint conversion on the identification candidate area 55 is calculated. There are methods and so on. As a method for calculating the degree of similarity, for example, pattern matching such as Normalized Cross-Correlation is generally used, but other methods may be used. At this time, by designing an evaluation function in which the evaluation function is the similarity and the parameters α, β, and γ are variables, and solving the optimization problem that maximizes this evaluation function, the similarity is obtained with respect to the classifier 64_n. Get the maximum image conversion method. As illustrated in FIG. 4, when two or more classifiers 64 are present, the degree of similarity with the converted image 86_n acquired for the identification candidate area 55 is calculated for each classifier, and the degree of similarity is calculated. Obtain the ID of the classifier 64_n that maximizes.

In step S54, the similarity between the converted image 86_n calculated in step S53 and the classifier information 65 is compared with the threshold value, and if the similarity is greater than or equal to the threshold value, the process is terminated, and if it is less than the threshold value, the process returns to step S51 and different parameters are used. After changing to, repeat the same process. The threshold value used in step S54 may be arbitrarily set by the installer of the object detection device 2a, but by feeding back the accuracy of the object detection when the object detection by the object detection device 2a is executed by the predetermined threshold value. , The threshold value may be changed to an appropriate value. For example, there is a method of changing the threshold value to a higher value when it is determined that the accuracy of the object detection device 2a using a certain threshold value is insufficient.

When determining the parameters in step S51, the image conversion method determination unit 7a creates a matrix map in which the aspect ratios of the converted images generated by the parameters α, β, and γ are recorded in advance, and refers to the matrix map. You may decide. Alternatively, the imaging space may be divided into a plurality of regions, a matrix map holding parameters α, β, and γ that are approximately valid for each region may be prepared and determined by referring to the matrix map. At that time, if a parameter more suitable than the parameters α, β, and γ held by the matrix map is found, a method of updating may be used. Alternatively, a method of determining substantially effective parameters α, β, and γ by acquiring camera parameters and acquiring information on the installation state of the imaging device 1 may be used.

Further, the image conversion method determination unit 7a determines whether or not to continue the process of changing the parameters α, β, and γ to calculate the similarity, and if it continues, returns to step S51, and if it does not continue, the process. May be terminated. The criterion for determining whether or not to execute the process may be determined, for example, by whether or not the number of times the parameters α, β, and γ are changed exceeds the number of times set in advance. Alternatively, the process may be terminated when the similarity calculated in step S53 is equal to or less than the preset minimum value. By terminating the process even when the similarity does not exceed the threshold value, it is possible to prevent waste in which the object detection process of the object detection device 2a is repeatedly performed when the object to be detected is not included in the identification candidate area 55. Can be done.

Next, the effect of the image conversion method determining unit 7a will be described with reference to FIGS. 7A and 7B. In FIG. 7A, 82a, 82b, and 82c indicate the viewpoints (installation position / direction) of the imaging device 1, and 87a, 87b, and 87c indicate rectangular images including a person extracted from the captured images of the respective viewpoints.

When a person is detected using the classifiers 64_1 and 64_2 shown in FIG. 7B based on the rectangular images 87a, 87b and 87c obtained by photographing an upright person, the rectangular image 87a photographed from the viewpoint 82a is a template 66_1 of the classifier 64_1. The rectangular image 87c taken from the viewpoint 82c has a high degree of similarity to the template 66_2 of the classifier 64_2. Therefore, for the rectangular image 87a and the rectangular image 87c, a person can be easily detected by using the classifier 64_1 or the classifier 64_2.

On the other hand, the person in the rectangular image 87b taken from the viewpoint 82b is deformed (deviation from the template) due to the inclination of the line of sight of the image pickup apparatus 1, and has a degree of similarity to both the template 66_1 and the template 66_2. Is low, so the classifier 64_1 and the classifier 64_2 cannot identify a person. For this reason, it has been difficult to detect a person in the field where only the image pickup device 1 of the viewpoint 82b is installed.

Even in such a case, by using the image conversion method determining unit 7a of this embodiment, it is possible to detect the deformed detection target in the rectangular image 87b. The procedure for detecting a deformed person photographed from the viewpoint 82b will be described below.

First, the parameters α, β, and γ are determined, and the converted image 86b is created by performing virtual viewpoint conversion on the rectangular image 87b using the parameters α, β, and γ as inputs. Then, the similarity between the converted image 86b and the template 66_1 and the template 66_2 is calculated, and if there is a classifier 64_n indicating the similarity equal to or higher than the threshold value, the ID 67 of the classifier is acquired. If there is no classifier 64_n showing similarity above the threshold, the parameters α, β, and γ are determined again, and the same processing is performed. Although the rectangular image 87b is deformed due to the tilt of the camera, the image information and the three-dimensional information of the front of the person can be acquired. Therefore, when the rectangular image 87b taken from the viewpoint 82b is virtually converted to the viewpoint 82a, a converted image 86b similar to the rectangular image 87a can be obtained, and an image suitable for input to the classifier 64_1 can be obtained. It becomes.

Similarly, when the rectangular image 87b taken from the viewpoint 82b is virtually converted to the viewpoint 82c, a converted image 86b similar to the rectangular image 87c can be obtained, and an image suitable for input to the classifier 64_2 can be obtained. It will be possible.

Here, the advantages of the image conversion method determining unit 7a in a situation where an obstacle exists between the viewpoint 82b and the person and a part of the human body (for example, the legs) is not reflected in the rectangular image 87b will be described. When the rectangular image 87b is converted to the viewpoint 82a, the converted image 86b also lacks the legs as in the rectangular image 87b, so that the classifier 64_1 that requires leg detection cannot detect a person. On the other hand, when the rectangular image 87b is converted to the viewpoint 82c, the converted image 86b also lacks the legs as in the rectangular image 87b, but the classifier 64_2, which does not require leg detection, detects a person. Can be done. That is, even when a rectangular image 87b in which a part of the human body is missing is input, if an appropriate image conversion method is determined by the image conversion method determination unit 7a and the classifier 64 corresponding to the image conversion method is selected, Accurate person detection can be realized.
<Image conversion unit>
The image conversion unit 8 converts the identification candidate region 55 according to the image conversion method determined by the image conversion method determination unit 7a, and acquires a converted image 86 suitable for input to the classifier 64. As the image conversion method, as in step S52, for example, a conversion formula such as Equations 1 to 3 can be used, but other methods may be used.
<Identification unit>
FIG. 8 shows the details of the identification unit 9. The identification unit 9 determines whether or not a detection target is included in the converted image 86_n acquired by the image conversion unit 8, and includes a classifier recording unit 91 that records at least one classifier 64_n. It includes an identification processing execution unit 92 that performs identification processing on the converted image 86_n using the classifier 64_n, and an identification result output unit that outputs the result of the identification processing. Hereinafter, the identification processing execution unit 92 and the identification result output unit 93 will be described in detail.

The identification processing execution unit 92 performs identification processing on the converted image 86_n output by the image conversion unit 8 by using the identification device 64_n recorded in the identification device recording unit 91. When two or more classifiers 64_n are recorded in the classifier recording unit 91, the discriminator processing execution unit 92 acquires the ID of the classifier 64_n selected by the image conversion method determination unit 7a and corresponds to the ID. After selecting the classifier 64_n to be used, the image conversion unit 7 outputs the converted image 86_n to perform the identification process.

The identification result output unit 93 outputs the identification processing result of the identification processing execution unit 92 to the outside. For example, when the object detection device 2a is connected to a display device such as a monitor, an image of the shooting space may be displayed on the display device. Then, when the identification processing execution unit 92 determines that the conversion image 86_n includes the detection target, the identification candidate area information 54_n of the identification candidate area 55 that is the basis of the converted image 86_n is referred to, and the image pickup apparatus 1 The image position of the identification candidate area 55 in the captured image is acquired. Then, a rectangular detection window or the like may be displayed at a position corresponding to the detection target in the captured image displayed on the display device, or a message indicating that the detection target has been detected may be displayed.
<Processing flow>
Next, the processing flow of object detection in the object detection device 2a of this embodiment will be described with reference to FIG.

In step S91, first, the image pickup apparatus 1 acquires the image information and the three-dimensional information corresponding to the measurement range and outputs them to the object detection apparatus 2a. The image acquisition unit 3 acquires image information based on the input from the image pickup device 1, and the three-dimensional information acquisition unit 4 acquires the three-dimensional information based on the input from the image pickup device 1.

In step S92, the identification candidate region 55 is extracted using the identification candidate region extraction unit 5. Specifically, after the rectangular area extracted by the image processing unit 51 and the rectangular parallelepiped area extracted by the three-dimensional information processing unit 52 are set as the identification candidate area 55, the identification candidate area ID is used for the extracted identification candidate area 55. An ID is assigned by the granting unit 53.

In step S93, one identification candidate area 55 to be subjected to the identification process is selected from the extracted identification candidate areas 55.

In step S94, the viewpoint conversion is performed on the selected identification candidate area 55, and the converted image 86_n is acquired by projecting it onto the image. By the optimization process, the image conversion method that maximizes the similarity to the classifier information 65 is acquired. When two or more classifiers 64_n exist, for example, the ID of the classifier 64_n having the maximum similarity between the converted image 86_n and the template 66 acquired by performing viewpoint conversion on the discrimination candidate area 55 is acquired. Then determine the appropriate image conversion method for the corresponding classifier 64_n.

In step S95, image conversion is performed on the selected identification candidate area 55 by the conversion method determined in step S94, and the converted image 86_n is acquired.

In step S96, the converted image 86_n acquired in step S95 is subjected to identification processing using the classifier 64_n.

In step S97, as a result of the identification process, it is determined whether or not the detection target is included in the converted image. If it is included, step S98 is performed, and if it is not included, step S99 is performed.

In step S98, when it is determined that the detected object is included in the converted image as a result of the identification process, the identification result is output. When the object detection device 2a is connected to a display device such as a monitor, an image of the shooting space is displayed on the display device, and a rectangular detection window is displayed at a position corresponding to the identification candidate area 55 in the image. , A message indicating that the detection target has been detected may be displayed. The position corresponding to the identification candidate area 55 is acquired by referring to the position information recorded in the identification candidate area information management unit 54.

In step S99, after the identification process for the selected identification candidate area 55 is completed, it is determined whether or not the identification process has been performed on all the identification candidate areas 55 extracted in step S92. Then, if there is an identification candidate area 55 for which the identification process has not been performed, step S93 is executed, and if there is no identification candidate area 55 for which the identification process has not been performed, the object detection process is terminated.

As described above, in the object detection device 2a of the first embodiment, the extracted identification candidate area 55 is converted into an image suitable for input to the classifier by virtual viewpoint conversion, and then the detection target is detected. By doing so, even if the appearance of the detection target in the image is different from the template of the classifier, or even if a part of the detection target on the screen is hidden by a shield, the detection target is detected with high accuracy. can do.

Next, the object detection device 2b of the second embodiment will be described with reference to FIGS. 10 to 13. The points common to the first embodiment are omitted from the duplicate description.

FIG. 2 is a block diagram showing an outline of the object detection device 2b of the present embodiment connected to the image pickup device 1 such as a stereo camera. In the object detection device 2a of the first embodiment, the image conversion method determination unit 7a for comprehensively changing the parameters α, β, and γ for rotating the three-dimensional information was used, but in the object detection device 2b of the present embodiment, An image conversion method determination unit 7b capable of efficiently determining the parameters α, β, and γ is used. Hereinafter, the image conversion method determination unit 7b will be described in detail.

First, an outline of processing in the image conversion method determination unit 7b will be described with reference to FIG. 11 showing a state in which an upright person is photographed from the viewpoint 82d. In FIG. 11, Xc, Yc, and Zc are the x-axis, y-axis, and z-axis of the camera coordinate system, and 204 shows the optical axis of the image pickup device 1 installed at the viewpoint 82d. Here, the camera coordinate system is a three-dimensional coordinate system representing a shooting space, with the optical center of the camera of the image pickup apparatus 1 as the origin, the z-axis (Zc) being aligned with the direction of the optical axis 204 of the camera, and the x-axis. (Xc) and the y-axis (Yc) are taken parallel to the horizontal direction and the vertical direction of the image projection surface 205. Further, 205 is an image projection surface, 206 is an image taken from the viewpoint 82d, 207 is three-dimensional information acquired from the viewpoint 82d, 208 is a straight line indicating the posture direction of the detection target, and 209 and 210 are the identification candidate areas 55 and the straight line 208. Indicates the intersection coordinates (Xct, Yct, Zct), (Xcb, Ycb, Zcb) in the camera coordinate system of, and 211 and 212 are the intersection coordinates of the identification candidate area 55 after the virtual viewpoint conversion and the straight line 208 in the camera coordinate system. (Xct', Yct', Zct'), (Xcb', Ycb', Zcb') are shown.

In the image conversion method determining unit 7b of the present embodiment, as shown in the lower figure of FIG. 11, a straight line 208 inclined with respect to the y-axis (Yc) is converted into a straight line 208'parallel to the y-axis (Yc). The parameters α, β, and γ of are calculated. Then, by generating a three-view drawing of the conversion candidate region 55', the optimum converted image 86 is acquired as an input to the classifier 67.

FIG. 12 shows a processing flow of the image conversion method determining unit 7b including the determination processing of the above parameters α, β, and γ. The processing flow will be outlined below.

First, the viewpoint transformation parameter β is set to 0 ° (S121), and then the straight line 208 is acquired (S122). Then, after setting arbitrary parameters α and γ for rotating the straight line 208 (S123), a three-view drawing to be detected is generated using the set parameters α, β and γ (S124). Then, after selecting one of the three views as the converted image 86 (S125), the similarity between the selected converted image 86 and the classifier information 65 is calculated (S126), and if the similarity is equal to or more than the threshold value, the selection is being made. The converted image 86 of the above is determined as the input image to the classifier 64, and the process is terminated. On the other hand, if the similarity is less than the threshold value, the process proceeds to step S128 (S127). In step S128, it is determined whether all of the generated three views are selected as converted images, and if not all are selected, the process proceeds to step S125, and the similarity is calculated for all three views when β = 0 °. If so, the process proceeds to step S129 (S128). In step S129, the parameter β is changed, that is, the identification candidate region 55 is rotated around the y-axis (Yc), and then the process returns to step S124 (S129) to obtain a converted image 86 having a similarity equal to or higher than the threshold value. The process is repeated until the result is obtained. Hereinafter, particularly important steps S122, S123, S124, and S129 will be described in detail.

In step S122, the straight line 208 is acquired. As an example of how to obtain the straight line 208, it is assumed that the three-dimensional information of the identification candidate region 55 is referred to and a straight line connecting two points having the maximum Euclidean distance between each point of the three-dimensional point cloud is taken. This is because when the detection target is an upright person, the identification candidate area 55 including the person can be predicted to be a rectangular body long in the vertical direction, and the direction in which the Euclidean distance is maximized is the straight line 208 indicating the posture direction of the person. This is because it can be estimated to be. Further, the principal component analysis may be performed on the three-dimensional point cloud of the identification candidate region 55, and a straight line taken in the direction of the first component may be used. Alternatively, if the floor surface existing in the space to be photographed can be detected by a general floor surface estimation method, the straight line 208 is drawn by using the information of the direction orthogonal to the floor surface and one point corresponding to the head. You can take the method of deciding.

In step S123, the parameters α and γ are determined so that the x value and the z value of the intersection coordinate 211 and the intersection coordinate 212 are both equal, that is, Xct'= Xbt'and Zct'= Zbt'. When the identification candidate area 55 is rotated so that Xct'and Xcb' are equal, the rotation angle around the z-axis (Zc) corresponds to the parameter γ, and the identification candidate area 55 is made equal to Zct'and Zcb'. The angle of rotation around the x-axis (Xc) when the is rotated corresponds to the parameter α. Since the parameter β is set to 0 ° in step S121, the parameters α, β, and γ can be determined by the above processing.

Next, the process of step S124 will be described with reference to FIG. In step S124, a three-view view of the identification candidate area 55 is acquired after the virtual viewpoint conversion. In FIG. 13, the viewpoints 82e, 82f, 82g are viewpoints for generating a three-view view, and the converted images 86e, 86f, 86g show the converted images 86 generated from each viewpoint. After determining the parameters α and γ that make the straight line 208 parallel to the y-axis (Yc), when a three-view drawing is generated while changing the parameter β, the conversion shown in FIG. 13 occurs when the predetermined parameter β is obtained. As shown in the image 86e, the viewpoint can be virtually converted to the viewpoint from the front of the person in the identification candidate area 55, and the person can be detected by using the classifier 64 having the corresponding template.

However, in the actual environment, the appearance of the detection target from a specific direction may not be suitable for identification due to shielding or the like. Therefore, by obtaining the converted images 86f and 86g from the viewpoints 82f and 82g from the side surface and the upper surface in addition to the converted image 86e from the viewpoint 82e from the front, the number of candidate classifiers 64 can be increased and the object can be obtained. The accuracy of detection can be improved. It should be noted that the viewpoint 82f for viewing the side surface can be set by further rotating each parameter with respect to the viewpoint 82e by α = 0 °, β = 90 °, and γ = 0 °, and each parameter is further changed with respect to the viewpoint 82e. The viewpoint 82g for viewing the upper surface can be set by rotating α = 90 °, β = 0 °, and γ = 0 °, and the converted image 86e can be set by perspectively projecting the identification candidate region 55 at the viewpoints 82e, 82f, and 82g. 86f and 86g can be obtained efficiently, and efficient person detection can be realized by using these as a three-view drawing.

In the object detection device of the second embodiment described above, the parameters α, β, and γ can be determined more efficiently than in the first embodiment, and when the person is deformed in the image or when shielding occurs. Can also perform highly accurate person detection.

1 image pickup device, 2a, 2b object detection device, 3 image acquisition unit, 4 three-dimensional information acquisition unit, 5 identification candidate area extraction unit, 51 image processing unit, 52 three-dimensional information processing unit, 53 identification candidate area ID assignment unit, 54 Identification candidate area information management unit, 54_n Identification candidate area information 55 Identification candidate area, 6 Identification device information acquisition unit, 64 Identification device, 65 Identification device information, 66 Template, 7a, 7b Image conversion method determination unit, 8 Image conversion unit , 82 viewpoints, 86 conversion images, 87 rectangular images, 9 identification units, 91 classifier recording units, 92 identification processing execution units, 93 identification result output units.

Claims (15)

An object detection device that determines whether or not a detection target exists within the measurement range.
A 3D information acquisition unit that acquires 3D information within the measurement range based on the input from the image pickup device, and
An identification candidate area extraction unit that extracts an identification candidate area in which the detection target may exist,
The classifier used to detect the detection target and
The classifier information acquisition unit that acquires the information of the classifier, and
An image conversion method determination unit that determines parameters for virtually performing viewpoint conversion processing on three-dimensional information in the identification candidate region,
An image conversion execution unit that generates a converted image based on the three-dimensional information in the identification candidate region that has been virtually subjected to viewpoint conversion processing.
An identification unit that detects the detection target using the classifier based on the converted image, and
An image acquisition unit that acquires image information within the measurement range based on the input from the image pickup device, and an image acquisition unit.
Equipped with a,
The image conversion method determining unit, by using the information of the identifier the image information and the three-dimensional information, a feature that you determine the parameters to generate optimal the converted image as an input of the discriminator Object detection device. In the object detection device according to claim 1 ,
The identification candidate region extraction unit
An identification candidate area ID assigning unit that assigns IDs to a plurality of the identification candidate areas,
An identification candidate area information management unit that collectively manages the ID of the identification candidate area, the position in the image information, and the position in the three-dimensional information.
An object detection device comprising. An object detection device that determines whether or not a detection target exists within the measurement range.
A 3D information acquisition unit that acquires 3D information within the measurement range based on the input from the image pickup device, and
An identification candidate area extraction unit that extracts an identification candidate area in which the detection target may exist,
The classifier used to detect the detection target and
The classifier information acquisition unit that acquires the information of the classifier, and
An image conversion method determination unit that determines parameters for virtually performing viewpoint conversion processing on three-dimensional information in the identification candidate region,
An image conversion execution unit that generates a converted image based on the three-dimensional information in the identification candidate region that has been virtually subjected to viewpoint conversion processing.
An identification unit that detects the detection target using the classifier based on the converted image, and
Equipped with a,
The identifier information acquiring unit, the identifier ID and object detection apparatus characterized that you get the identifier information representing the input signal exhibit particularly high identification capability. In the object detection device according to any one of claims 1 to 3 ,
The image conversion method determination unit
The optimization process is performed on the result of the virtual viewpoint conversion process, and the image conversion to the most suitable image in the discriminator for determining whether or not the detection target is included in the identification candidate region is realized. An object detection device characterized in determining parameters. In the object detection device according to claim 3 ,
An object detection device characterized in that the classifier information is any one of a template, color information, luminance information, contour, and gradient information. In the object detection device according to claim 5 ,
When the classifier information is a template
The image conversion method determining unit calculates the similarity between the image acquired by performing the viewpoint conversion process and the template, and selects the classifier having the maximum similarity. .. In the object detection device according to any one of claims 1 to 6 .
The image conversion method determining unit is an object detection device having a function of determining the parameters by using camera parameters expressing the installation state of the imaging device. In the object detection device according to any one of claims 1 to 7 .
The identification unit includes a classifier recording unit that records at least one identification unit having discriminating ability with respect to the detection target.
An identification processing execution unit that performs identification processing for identifying whether or not the detection target is included in the converted image using the identification device.
An identification result output unit that outputs a result when it is determined that the converted image includes the detection target,
An object detection device comprising. In the object detection device according to any one of claims 1 to 8 .
The image conversion method determining unit is an object detection device characterized in that the parameters are determined based on the three-dimensional shape of the detection target. In the object detection device according to claim 9 ,
The image conversion method determining unit acquires a straight line indicating a general posture direction of a detection target passing through the identification candidate region, and obtains a straight line.
A function of acquiring the parameter that realizes a virtual viewpoint conversion such that the straight line is parallel to the Y axis of the camera coordinate system of the imaging device.
An object detection device comprising. In the object detection device according to claim 10 ,
The image conversion method determination unit
After the straight line is converted into a state parallel to the Y axis of the camera coordinate system, a virtual viewpoint conversion is performed from the front surface, the side surface, and the upper surface to the viewpoint for observing the identification candidate region, and the conversion is performed at each viewpoint. An object detection device characterized by acquiring an image. In the object detection device according to claim 10 or 11 .
The image conversion method determination unit
An object detection device characterized in that the straight line is determined by connecting two points having the maximum Euclidean distance between each point of the three-dimensional point cloud included in the identification candidate region. In the object detection device according to claim 10 or 11 .
The image conversion method determination unit
Principal component analysis was performed on the three-dimensional point cloud included in the identification candidate region.
An object detection device characterized in that the straight line is determined by taking the direction of the first component. In the object detection device according to claim 10 ,
The image conversion method determination unit
Estimate the floor surface in the measurement range and
A specific site to be detected is detected in the identification candidate region,
An object detection device, characterized in that a straight line passing through one point corresponding to the portion and extending in a direction orthogonal to the floor surface is determined as the straight line. It is an object detection method that determines whether or not a detection target exists within the measurement range.
Based on the input from the image pickup device, the three-dimensional information within the measurement range is acquired, and
An identification candidate region in which the detection target may exist is extracted.
Obtain the information of the classifier used to detect the detection target, and
Determine the parameters for virtual viewpoint conversion processing of the three-dimensional information in the identification candidate area.
A converted image is generated based on the three-dimensional information in the identification candidate region that has been virtually subjected to viewpoint conversion processing.
The detection target is detected by using the classifier based on the converted image .
Based on the input from the imaging device, the image information within the measurement range is acquired, and
A method for detecting an object , which comprises using the image information, the three-dimensional information, and the information of the classifier to determine a parameter for generating the converted image that is optimal as an input of the classifier .

Images