JP2009237773A - Image processor and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor for detecting highly precisely a person by simple processing. <P>SOLUTION: Pedestrian candidate images are extracted from image data, by comparing the image data from an imaging device for imaging the surroundings of a vehicle, with a reference pattern that is a model pattern of a pedestrian. The presence of regularity is determined in at least one of a shape and an arrangement of each pedestrian candidate image. A displaying image data is generated based on the residual pedestrian candidate image excluding the pedestrian candidate images determined to have the regularity. An image is displayed based on the display image data. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing program, and more specifically to image processing used in a person recognition system that detects a person based on an image of a vehicle-mounted camera or the like.

  Conventionally, a front monitoring system is known in which a far infrared video camera is mounted on a vehicle such as an automobile, and recognizes a pedestrian in front of the vehicle or a person on a bicycle when driving at night to alert the driver. ing. As one of such systems, for example, a human recognition system is known in which a reference pattern representing a human luminance distribution is stored in advance and a pedestrian is detected by comparing camera image data with the reference pattern. (For example, patent document 1). There is also known an apparatus for detecting a pedestrian by performing pattern matching between an input image captured by an infrared camera and a whole body model image and a head model image stored in advance in a model image storage unit (for example, Patent Document 2).

Also, an input feature vector including the size of the object, the upper shape of the object, and the side shape of the object as elements is generated based on the image of the object, and whether or not the object is a pedestrian based on this An image processing apparatus that discriminates the image is also known (for example, Patent Document 3). In this apparatus, a vector y mapped to the discriminant space is generated by performing an operation for analysis based on the input feature vector. Then, it is determined whether or not the object is a pedestrian depending on whether or not the vector y is within a predetermined region in the determination space.
JP 2005-285011 A JP 2005-196590 A JP 2007-65924 A

  However, the devices disclosed in Patent Documents 1 and 2 as described above have the following problems. That is, when pattern matching is performed by the method as described in Patent Documents 1 and 2, for example, there is a problem that a guardrail pole is detected as a pedestrian.

  In addition, the technique of Patent Document 3 requires a complicated vector operation, and has a high processing load. In addition, expensive hardware is required to realize such processing, which increases costs. It was.

  Therefore, an object of the present invention is to provide an image processing apparatus capable of detecting a person with high accuracy with simpler processing.

  The present invention employs the following configuration in order to solve the above problems. The reference numerals in parentheses, supplementary explanations, and the like are examples of the correspondence with the embodiments described later in order to help understanding of the present invention, and do not limit the present invention.

  A first invention is an image processing device that recognizes the presence of a pedestrian based on image data from an imaging device that captures the outside of a vehicle, and compares the image data with a reference pattern that is a pedestrian model pattern. Whether or not at least one of the shape or arrangement of the pedestrian candidate image extracted by the pedestrian candidate extraction unit and the pedestrian candidate extraction unit that extracts the pedestrian candidate image from the image data is regular. Regularity analysis means (14) for determining whether or not the pedestrian candidate image determined to be regular by the regularity analysis means, and generating image data for display based on the remaining pedestrian candidate image Means (14, 15) and display means (17) for displaying an image based on the display image data generated by the exclusion means.

  According to the first invention, regular pedestrian candidate images are excluded from pedestrian candidates. Thereby, for example, an object having regularity in arrangement and shape such as a pole of a guardrail is prevented from being erroneously detected as a pedestrian, and more accurate pedestrian detection is possible.

  In a second aspect based on the first aspect, the regularity analyzing means determines whether or not the end portions of the plurality of candidate pedestrian images are aligned on a straight line, and the pedestrians aligned on the straight line It is determined that the candidate image has regularity.

  In a third aspect based on the second aspect, the regularity analyzing means calculates a straight line connecting each end of the plurality of pedestrian candidate images and a vanishing point on the image indicated by the image data. (14) and extraction means (14) for extracting a straight line in which the number of pedestrian candidate images whose ends intersect each other is a predetermined number or more. Then, the regularity analysis unit determines that the pedestrian candidate image whose end portion intersects with the straight line extracted by the extraction unit has regularity.

  According to the second to third inventions, pedestrian candidate images arranged in a straight line can be excluded from pedestrian candidates. Thereby, highly accurate pedestrian detection becomes possible.

  According to a fourth invention, in any one of the first to third inventions, the regularity analyzing means is configured such that the object corresponding to the pedestrian candidate image is based on the number of pixels between the plurality of pedestrian candidate images. Includes an equidistant arrangement determining means for determining whether or not they are arranged at equal intervals. Then, the regularity analysis unit determines that the plurality of pedestrian candidate images determined to be arranged at equal intervals by the equal interval arrangement determination unit have regularity.

  According to the fourth invention, pedestrian candidate images arranged at equal intervals are excluded from pedestrian candidates. Thereby, for example, it is possible to prevent erroneous detection as an pedestrian of an object arranged at equal intervals, such as a pole of a guardrail, and to further improve the accuracy of pedestrian detection.

  In a fifth aspect based on the first aspect, the regularity analysis means includes shape calculation means for calculating the shape of the pedestrian candidate based on the luminance of the plurality of pedestrian candidate images. Then, the regularity analysis means determines that there is regularity when the shapes of the pedestrian candidates are similar.

  In a sixth aspect based on the first aspect, the image processing apparatus associates the matching pattern information indicating the reference pattern used when the pedestrian candidate extracting unit extracts the pedestrian candidate image with each pedestrian candidate image. And a matching pattern information storage means for storing. Then, the regularity analysis unit determines that the pedestrian candidate images extracted based on the same reference pattern have regularity based on the matching pattern information.

  According to the fifth to sixth inventions, pedestrian candidate images having similar shapes can be excluded from pedestrian candidates, and the accuracy of pedestrian detection can be further increased.

  A seventh aspect of the invention is an image processing program for causing a computer of an image processing apparatus to recognize the presence of a pedestrian based on image data from an imaging apparatus that captures the outside of a vehicle. A pedestrian candidate extraction means (S2) for extracting a pedestrian candidate image from the image data by comparing with a reference pattern that is a model pattern of the pedestrian, and a shape or arrangement of pedestrian candidate images extracted by the pedestrian candidate extraction means Regularity analysis means (S3 to S8) for determining whether or not at least one of them has regularity, and pedestrian candidate images determined by the regularity analysis means to be regularity are excluded, and the remaining pedestrians Exclusion means (S7, S9) for generating display image data based on the candidate image, and display means (S10) for displaying an image based on the display image data generated by the exclusion means To function Te.

  According to the seventh aspect, the same effect as the first aspect can be obtained.

  According to the present invention, more accurate pedestrian detection can be realized.

  Embodiments of the present invention will be described below with reference to the drawings. In addition, this invention is not limited by this Example.

  FIG. 1 is a diagram showing a configuration of an image processing apparatus 10 according to an embodiment of the present invention. In FIG. 1, the image processing apparatus 10 includes a night view camera 11, a person detection unit 12, and a display unit 17.

  The night view camera 11 is, for example, a far-infrared video camera that is mounted on a vehicle and recognizes a pedestrian at night and a person on a bicycle. The night view camera 11 is connected to the person detection unit 12 via an in-vehicle LAN communication line compliant with IEEE1394. The image acquired by the night view camera 11 is output to the person detection unit 12.

  The person detection unit 12 detects a pedestrian based on the image acquired by the night view camera 11. Then, an image in which a caution frame is superimposed on the position of the pedestrian on the camera image is generated. The person detection unit 12 includes a primary extraction unit 13, a secondary extraction unit 14, and an image generation unit 15.

  The primary extraction unit 13 acquires the camera image output from the night view camera 11. Then, based on the luminance value of each pixel of the image, an area including the human candidate image (hereinafter, human candidate area) is extracted. Then, an image indicating the human candidate region (hereinafter referred to as a primary extraction image) is output to the secondary extraction unit 14.

  The secondary extraction unit 14 analyzes the regularity of the person candidate region by performing processing as described later based on the primary extracted image. Further, a process for excluding the candidate person group with regularity is performed. Then, an image indicating the extraction result (hereinafter referred to as a secondary extracted image) is output to the image generation unit 15.

  The image generation unit 15 generates a caution frame corresponding to the size of the human candidate region based on the secondary extracted image. The image generation unit 15 generates an image obtained by superimposing the attention frame on the camera image as a display image. Further, the image generation unit 15 outputs the display image to the display unit 17.

  The storage unit 16 stores various data used in processing as described later. Specifically, a reference pattern indicating a luminance pattern of a human model, an FOE coordinate 161 indicating a position of a vanishing point (FOE; Focus of Expansion) in the camera image, and the like are stored. The FOE coordinates are uniquely determined based on the installation position and angle of the night view camera 11, and the FOE position in the screen coordinate system of the display unit 17 is stored in advance as the FOE coordinates 161.

  The display unit 17 is a display device having a resolution corresponding to the number of pixels of the night view camera 11. For example, if the night view camera 11 is a camera having 450,000 pixels, the display unit 17 has a resolution of 770 × 578 dots. The display unit 17 displays the image output from the image generation unit 15 as a person detection result.

  Next, an outline of person detection processing in the present embodiment will be described with reference to FIGS. In the present embodiment, the position of the person candidate area and the regularity of the person candidate area detected by the conventional method are analyzed, and the person candidate area is, for example, a non-human person such as a guardrail pole. A process for determining whether or not it is an object is executed.

  FIG. 2 is a diagram schematically showing an example of a camera image taken by the night view camera 11. In FIG. 2, pedestrians 101a to 101c, guardrails 102a and 102b, and a sign 103 are displayed. The guard rail 102a includes poles 104a to 104d, and the guard rail 102b includes poles 104e to 104h.

  In the present embodiment, a person detection process is first performed on the camera image as shown in FIG. The person detection process here may be a conventionally known technique, but as an example, in the present embodiment, a person detection process based on luminance (hereinafter referred to as a primary extraction process) is performed. More specifically, an area composed of pixels having a luminance equal to or higher than a predetermined value is extracted for the luminance of each pixel of the camera image. Then, pattern matching with a reference pattern indicating a human luminance pattern stored in advance in the storage unit 16 is performed, and the matched one is set as a human candidate region. FIG. 3 is an example of an image showing the result of the primary extraction process (hereinafter referred to as a primary extracted image). The size (number of pixels) of the image is the same size as the camera image. In FIG. 3, areas where each pedestrian 101 and each pole 104 are surrounded by dotted lines are displayed as person candidate areas 110a to 110k. That is, FIG. 3 shows that the pole 104 is also erroneously recognized as a pedestrian in the primary extraction process.

  In the present embodiment, person detection processing is further performed on the primary extraction image (hereinafter referred to as secondary extraction processing). In the secondary extraction process, paying attention to the regularity that the poles 104 have the same height in real space and are aligned in a straight line, the human candidate area having such regularity is a pedestrian. The process of excluding it is not executed. Specifically, the following processing is executed.

  First, in the primary extracted image as shown in FIG. 3, the coordinates of the upper end of each person candidate area 110 (more precisely, the center coordinates of the upper side of the rectangle indicated by the dotted line, but in the following, simply referred to as the upper end coordinates). ) And coordinates corresponding to FOE are calculated. FIG. 4 shows an example of a straight line 122 connecting the upper end coordinates of the person candidate area 110 a (corresponding to the pole 104 a) and the FOE 121. Such a straight line is calculated for each person candidate region. FIG. 5 is a diagram illustrating a state where the upper end coordinates of each person candidate region and the FOE 121 are connected by a straight line.

  Here, as shown in FIG. 4 and FIG. 5, the upper end portions of the person candidate regions 110a to 110d corresponding to the left poles 104a to 104d are overlapped with the straight line 122a. Similarly, the upper right part of the right person candidate areas 110e to 110h overlaps the straight line 122b. In this way, objects that are linearly arranged at the same height, such as the pole 104, overlap with the straight lines 122 a and 122 b toward the FOE 121. Therefore, it can be considered that the person candidate region 110 overlapping the straight lines 122a and 122b is not a person.

  Therefore, in this embodiment, the candidate human regions 110a to 110h overlapping the straight lines 122a and 122b are deleted from the primary extraction image to generate a secondary extraction image. FIG. 6 is an example of a secondary extracted image. FIG. 6 shows a state in which only the human candidate areas 110i to 110k corresponding to the pedestrian 101 remain. Then, based on the secondary extracted image, attention frames 141a to 141c as shown in FIG. 7 are generated.

  Finally, as shown in FIG. 8, an image in which the attention frames 141a to 141c are superimposed on the camera image is generated, and the image is displayed on the display unit 17 as a pedestrian detection result.

  As described above, in this embodiment, even if an object such as a guardrail pole is erroneously detected as a human by a conventional method, paying attention to the regularity of the object, these objects are not human. Can be detected. Thereby, it is possible to further improve the accuracy of the person detection process.

  Hereinafter, the detailed operation of the person detection process performed by the image processing apparatus 10 will be described with reference to FIG. The process shown in the flowchart of FIG. 9 is repeated every 1/10 second.

  In FIG. 9, first, the primary extraction unit 13 acquires a camera image from the night view camera 11 (step S1).

  Next, the primary extraction part 13 performs the primary extraction process based on the said camera image (step S2). The process in the present embodiment is a process for extracting the candidate person area 110 based on the luminance of each pixel as described above. Since the principle of the processing and the basic processing method are known to those skilled in the art, detailed description is omitted, but in the present embodiment, first, a pixel having a luminance of a predetermined value or more is extracted, A region composed of these pixels is set as appropriate. Then, this region (luminance pattern) is pattern-matched with a reference pattern indicating a human luminance pattern (the reference pattern includes a large number of luminance patterns obtained by observing a human from all angles). If it matches the reference pattern (any one of them), processing for treating the area as a person candidate area is executed. As a result of the processing, a primary extraction image as described above (the size of which corresponds to the resolution of the display unit 17) is generated, and the primary extraction unit 13 sends the secondary extraction unit 14 to the secondary extraction unit 14. A primary extracted image is output.

  Next, in the secondary extraction unit 14, the following process (secondary extraction process) is executed. First, the secondary extraction unit 14 reads the FOE coordinates 161 from the storage unit 16 (step S3). Subsequently, the secondary extraction unit 14 calculates a straight line connecting the upper end coordinates of each person candidate area 110 on the primary extraction image and the FOE coordinates 161 (corresponding to the FOE 121 in FIG. 5) (step S4).

  Next, the secondary extraction unit 14 extracts a straight line where a predetermined number or more of the human candidate areas 110 overlap, and performs a process of grouping the human candidate areas 110 overlapping the straight lines for each straight line (step) S5). The processing in step S5 will be described more specifically. First, the secondary extraction unit 14 calculates the number of person candidate regions 110 that overlap the straight line. Here, the determination part of whether or not they are overlapped is basically based on the upper end coordinates, but in order to prevent the detection condition from becoming too strict and the object to be detected not being detected, FIG. As shown in FIG. 5, a part of the overlap determination is given a slight width. For example, if the portion from the upper side of the person candidate area 110 to 3 dots overlaps with a straight line, it is determined that the candidate area overlaps. Next, only straight lines in which the number of candidate human areas 110 overlapping the straight line is equal to or greater than a predetermined value, for example, “4” or more, are extracted. Then, the person candidate areas 110 overlapping these straight lines are grouped for each straight line (see FIG. 11).

  Next, the following processing is performed for each group grouped as described above. First, the secondary extraction unit 14 determines whether or not the arrangement intervals of the human candidate regions 110 included in the group are equal intervals in the three-dimensional space (Step S6). The process of step S6 will be described more specifically. First, the secondary extraction unit 14 counts the number of pixels between the person candidate regions along the straight line. That is, the size of the interval between the candidate human regions on the straight line is acquired as the number of pixels. Next, it is determined whether or not the number of pixels is in a relationship of decreasing at a predetermined ratio. For example, even if the guardrail poles and the like are arranged at equal intervals in the three-dimensional space, when represented by a two-dimensional image (two-dimensional coordinate system), as shown in FIG. The interval is expressed as being small (narrow). However, for objects that are arranged at equal intervals in the three-dimensional space, it is possible to calculate in advance by a predetermined calculation the ratio at which the intervals become smaller in the two-dimensional coordinate system. Therefore, for example, if the change in the number of pixels is a ratio (interval pattern) of 10: 8: 6 from the bottom to the top of the screen, it is determined that the intervals are equal in the three-dimensional space. Note that the above processing method is merely an example, and the present invention is not limited to this, and any processing method can be used as long as it is possible to determine that the arrangement relationship of the person candidate regions 110 as described above is equally spaced in the three-dimensional space. May be.

  As a result of the determination, when it is determined that the intervals are equal, the secondary extraction unit 14 deletes the grouped person candidate regions 110 from the primary extraction image because the grouped candidate human regions 110 are considered to be objects such as guardrail poles ( Step S7). On the other hand, when it determines with it being not equal intervals, the secondary extraction part 14 advances a process to following step S8.

  Next, the secondary extraction unit 14 determines whether or not the processing in steps S6 to S7 has been performed for all groups (step S8). If not performed (NO in step S8), the secondary extraction unit 14 has not yet checked. The processes in steps S6 to S7 are repeated for the group.

  On the other hand, if all the groups are checked as a result of the determination in step S8 (YES in step S8), the image generation unit 15 uses the primary extraction image from which the person candidate area 110 has been deleted by the above processing as the secondary extraction image. Output to. The image generation unit 15 generates a caution frame 141 as shown in FIG. 7 according to the size of each human candidate region remaining on the secondary extracted image (step S9).

  Furthermore, the image generation unit 15 generates an image in which the generated attention frame 141 is superimposed on the camera image. Then, the image generation unit 15 outputs the image as a display image to the display unit 17 (step S10). As a result, an image as shown in FIG. 8 is displayed on the display unit 17. The person detection process according to the present embodiment is thus completed.

  As described above, in the present embodiment, after the person candidate area is once detected by the conventional method, the person candidate area further has regularity, specifically based on whether or not the person candidate area is arranged in a straight line. An object that is not a person is detected and excluded from the person candidate area. As a result, it is possible to prevent a guardrail pole or the like from being erroneously detected as a person, and to realize more accurate person detection.

  In the above-described embodiment, a straight line connecting the upper end coordinates of the candidate area and the FOE is calculated, and the overlap between the straight line and the upper end portion of each candidate area is determined. In addition, the candidate area You may make it use together the lower end side of. That is, a straight line connecting the lower end coordinates of the candidate area and the FOE 121 is calculated, and the overlap between the lower end part of each person candidate area (the same as the above upper end coordinates, the part for performing overlap determination has a width) and the straight line is also calculated. It may be determined to determine whether or not they are aligned on a straight line. Then, if either one of the upper end and the lower end overlaps a predetermined number or more, the person candidate area may be deleted as not being a person.

  Also, the method for determining whether or not a person is a person is not limited to the process using a straight line as described above. For example, the determination may be made based on the similarity of the luminance patterns of each person candidate area. . This is because a guardrail pole is considered to be similar in luminance pattern. Furthermore, the shape of the object may be calculated from the luminance pattern, and the determination may be made based on the similarity of the shape. For example, since the poles of the guardrail have the same shape, if there are a predetermined number or more of human candidate regions having a high degree of similarity in shape, it may be determined that they are not persons.

  Further, in the above-described embodiment, the matching between the reference pattern and the luminance pattern has been described as an example of the process of extracting the person candidate area in step S2. Focusing on this reference pattern, a plurality of candidate human regions that match the same reference pattern may be determined not to be a person. In other words, for a person, the clothes and appearance are different for each individual, and the reflectance is also different. Therefore, it is unlikely that the person candidate area corresponding to the person was extracted by matching the same reference pattern. . On the other hand, it is considered that the reflectance is almost the same when a plurality of the same objects such as guardrail poles are continuously arranged. Therefore, it is considered highly possible that the candidate pattern is extracted as a person candidate area by matching the same reference pattern. Therefore, in the process of step S2, information on the reference pattern used when extracted as the person candidate area, for example, the identification number of the reference pattern, etc., is temporarily stored in the storage unit 16 as matching pattern information. To do. At this time, each extracted person candidate area and the matching pattern information are stored in association with each other. Then, the secondary extraction unit 14 reads out the matching pattern information, and the person candidate area extracted by the reference pattern having the same identification number (that is, the person candidate area having the same matching pattern information) is not a person. You may make it determine with not.

  Further, regarding the FOE coordinates, in the above-described embodiment, the FOE coordinates 161 stored in advance in the storage unit 16 are read out and used. However, the present invention is not limited to this, and the night view camera is used every time in the process of step S3. Information such as 11 angles and focal lengths may be acquired and FOE coordinates may be calculated and used.

  The image processing apparatus and the image processing program according to the present invention can improve the accuracy of person detection based on a captured image of a camera, and are useful for a vehicle-mounted person detection system and the like.

1 is a functional block diagram illustrating an image processing apparatus according to an embodiment of the present invention. The figure which shows an example of a camera image typically Example of primary extracted image The figure which shows the example of the straight line which connected the person candidate area | region and FOE The figure which shows the example of the straight line which connected the person candidate area | region and FOE Example of secondary extracted image Figure showing an example of a caution frame An example of an image with a caution frame superimposed on a camera image The flowchart which shows the person detection process which concerns on embodiment of this invention The figure for demonstrating the overlap judgment area | region of a person candidate area | region and a straight line The figure which shows an example of grouping of a person candidate area | region

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 11 Night view camera 12 Person detection part 13 Primary extraction part 14 Secondary extraction part 15 Image generation part 16 Storage part 17 Display part

Claims (7)

An image processing device for recognizing the presence of a pedestrian based on image data from an imaging device that images the outside of the vehicle,
A pedestrian candidate extraction means for comparing the image data with a reference pattern which is a model pattern of a pedestrian and extracting a pedestrian candidate image from the image data;
Regularity analysis means for determining whether at least one of the shape or arrangement of the pedestrian candidate images extracted by the pedestrian candidate extraction means has regularity;
Excluding the pedestrian candidate image determined by the regularity analysis means to be regular, and generating the display image data based on the remaining pedestrian candidate image,
An image processing apparatus comprising: display means for displaying an image based on the display image data generated by the exclusion means.   The regularity analysis means determines whether or not the end portions of the plurality of pedestrian candidate images are aligned on a straight line, and determines that the pedestrian candidate images aligned on the straight line are regular. The image processing apparatus according to claim 1. The regularity analysis means includes:
Straight line calculating means for calculating a straight line connecting each end portion of the plurality of pedestrian candidate images and a vanishing point on the image indicated by the image data;
Extracting means for extracting the straight line in which the number of pedestrian candidate images intersecting at the ends is a predetermined number or more,
The image processing apparatus according to claim 2, wherein the pedestrian candidate image whose end portion intersects with the straight line extracted by the extraction unit is determined to have regularity. The regularity analysis means includes:
Based on the number of pixels between the plurality of pedestrian candidate images, including equidistant arrangement determination means for determining whether or not the objects corresponding to the pedestrian candidate images are arranged at equal intervals in the outside world of the vehicle,
The image processing apparatus according to claim 1, wherein the plurality of pedestrian candidate images determined to be arranged at equal intervals by the equal interval arrangement determination unit are determined to have regularity. The regularity analysis means includes:
Including a shape calculating means for calculating the shape of the pedestrian candidate based on the luminance of the plurality of pedestrian candidate images,
The image processing apparatus according to claim 1, wherein when the shape of the pedestrian candidate is similar, it is determined that there is regularity. The image processing device includes matching pattern information storage means for storing matching pattern information indicating a reference pattern used when the pedestrian candidate extracting means extracts the pedestrian candidate image in association with each pedestrian candidate image. In addition,
The image processing apparatus according to claim 1, wherein the regularity analysis unit determines that pedestrian candidate images extracted based on the same reference pattern have regularity based on the match pattern information. An image processing program to be executed by a computer of an image processing device that recognizes the presence of a pedestrian based on image data from an imaging device that captures an external environment of a vehicle,
The computer,
A pedestrian candidate extraction means for comparing the image data with a reference pattern which is a model pattern of a pedestrian and extracting a pedestrian candidate image from the image data;
Regularity analysis means for determining whether at least one of the shape or arrangement of the pedestrian candidate images extracted by the pedestrian candidate extraction means has regularity;
Excluding the pedestrian candidate image determined by the regularity analysis means to be regular, and generating the display image data based on the remaining pedestrian candidate image,
An image processing program that functions as a display unit that displays an image based on display image data generated by the exclusion unit.

Images