JP5617819B2 - Pedestrian recognition device - Google Patents

Description

  The present invention relates to a pedestrian recognition device that recognizes a pedestrian.

  Conventionally, a technique for recognizing a pedestrian in a captured image is known. For example, in Patent Document 1 and Non-Patent Document 1, a captured image is obtained by using an overall classifier for the pedestrian's whole body and a plurality of partial classifiers for partial areas such as the pedestrian's head and lower body in two stages. A technique for recognizing a pedestrian inside is disclosed. Patent Document 1 and Non-Patent Document 1 use a partial identifier that identifies a walking pattern (gait pattern) that is characteristic of a pedestrian and has an inverted V-shaped contour composed of left and right legs. Thus, it has been proposed to further improve the accuracy of pedestrian recognition.

  As in the techniques disclosed in Patent Document 1 and Non-Patent Document 1, a partial discriminator that recognizes a walking pattern (hereinafter referred to as an inverted V-shaped walking pattern) having an inverted V-shaped contour composed of left and right legs. By using it, it is considered that a pedestrian in a situation where the upper body cannot be confirmed on the image can be recognized. The situation in which the upper body cannot be confirmed on the image includes, for example, the case where the upper body is hidden by an obstacle or the case where the upper body is not shown in the image due to light adjustment.

US Patent Application Publication No. 2007/0230792

Shashua, A and two others, “Pedestrian detection for driving assistance systems: single-frame classification and system level performance”, Intelligent Vehicles Symposium, 2004 IEEE, 14-17 June 2004, p.1-6

  However, even the techniques disclosed in Patent Literature 1 and Non-Patent Literature 1 have a problem in that the accuracy of pedestrian recognition cannot be sufficiently improved in a situation where the upper body cannot be confirmed on the image. Details are as follows. In the techniques disclosed in Patent Literature 1 and Non-Patent Literature 1, by identifying a reverse V-shaped walking pattern, an attempt is made to recognize a pedestrian in a situation where the upper body cannot be confirmed on the image. There are many cases in which the contour formed by the left and right leg portions of the pedestrian does not have an inverted V shape.

  For example, when the pedestrian's upper body to the position below the crotch is obstructed by an obstacle, the contour formed by the left and right legs of the pedestrian does not have an inverted V shape. As an example of an obstacle, the umbrella which a pedestrian is pointing to, the baggage which a pedestrian has, the skirt which a pedestrian wears, etc. can be considered. In addition, even when only the position below the pedestrian's inseam is shown due to light adjustment, the contour formed by the left and right leg portions of the pedestrian does not have an inverted V shape.

  In this way, when the contour formed by the left and right legs of the pedestrian does not have an inverted V shape, the technique disclosed in Patent Literature 1 and Non-Patent Literature 1 is not identified by the partial classifier and cannot recognize the pedestrian. . Therefore, when the contour formed by the left and right legs of the pedestrian does not have an inverted V shape, the pedestrian cannot be recognized in a situation where the upper body cannot be confirmed on the image.

  The present invention has been made in view of the above-described conventional problems, and the object thereof is pedestrian recognition that makes it possible to further improve the accuracy of pedestrian recognition in situations where the upper body cannot be confirmed on the image. To provide an apparatus.

  In the pedestrian recognition apparatus according to claim 1, the first contour line tilted rightward in the captured image, the second contour line tilted diagonally left, and the upper end of the first contour line and the upper end of the second contour line Pedestrian recognition by determining whether or not a pedestrian candidate detected from a captured image is a pedestrian based on a trapezoidal contour shape (hereinafter referred to as a trapezoidal contour shape) consisting of a third contour line connecting I do. This trapezoidal contour shape, such as when the upper part of the pedestrian to the lower part of the crotch is blocked by an obstacle, or when only the lower part of the pedestrian's inseam is reflected by the light, Detected when the left and right legs of the pedestrian are only visible from a position below the crotch. Specifically, the contour lines of the left and right legs of the pedestrian are extracted as the first contour line and the second contour line, and the light and dark boundary caused by light adjustment and the contour line of the obstacle are extracted as the third contour line. The

  Therefore, based on this trapezoidal outline shape, the left and right legs of the pedestrian are only visible from the position below the crotch, and the outline of the left and right legs of the pedestrian is reversed in the captured image. Even when the V shape is not obtained, it is possible to accurately determine whether or not the pedestrian candidate is a pedestrian and recognize the pedestrian. Therefore, according to the configuration of claim 1, even when the contour formed by the left and right legs of the pedestrian in the captured image does not have an inverted V shape, the pedestrian is recognized in a situation where the upper body cannot be confirmed on the image. It becomes possible to do. As a result, it is possible to further improve the accuracy of pedestrian recognition in situations where the upper body cannot be confirmed on the image.

  The lower end of the first contour line and the lower end of the second contour line in the trapezoidal contour shape based on each captured image in which the trapezoidal contour shape can be detected by the contour shape detecting means as in claim 2. It is good also as a mode which calculates the value which shows the time change of the lower end width | variety which is an space | interval between. According to this, the value indicating the temporal change in the lower end width can be used for pedestrian recognition. The value indicating the temporal change in the lower end width is, for example, the amount of change in the lower end width per time or the value of the period of the lower end width change.

  In the case of the second aspect, the pedestrian candidate is a pedestrian based on the value indicating the temporal change in the lower end width calculated by the change amount calculating means as in the third aspect. It is preferable that the determination is performed. As described above, since the contour lines of the left and right legs of the pedestrian are extracted as the first contour line and the second contour line, when the pedestrian candidate is actually a pedestrian, the lower end width is the pedestrian. The change in the lower end width corresponds to the change in the pedestrian's stride. In addition, when the pedestrian candidate is actually a pedestrian, since the change in the lower end width corresponds to the change in the pedestrian, the pedestrian candidate is determined based on the value indicating the temporal change in the lower end width. It becomes possible to determine that the person is a pedestrian. Details are as follows.

  When there is no change in the lower end width, there is a high possibility that the pedestrian has no change in the stride (that is, a stopped pedestrian) or a non-pedestrian. On the other hand, when there is a change in the lower end width, there is a high possibility that the pedestrian has a change in the stride (that is, a pedestrian who is walking or running). Therefore, it becomes possible to determine more accurately whether or not the pedestrian candidate is a pedestrian from the value indicating the temporal change in the lower end width. Therefore, according to the structure of Claim 3, it becomes possible to determine more accurately whether a pedestrian candidate is a pedestrian based on the value which shows the temporal change of a lower end width | variety.

  In the case of the third aspect, as in the fourth aspect, the movement state of the pedestrian is estimated based on the value indicating the temporal change in the lower end width calculated by the change amount calculating unit. It is preferable to do. When the pedestrian candidate is actually a pedestrian, the change in the lower end width corresponds to the change in the pedestrian's stride, so the pedestrian's motion state is determined from the value indicating the temporal change in the lower end width. Can be estimated. For example, it is highly likely that the walking time is small when the stride time change is small, running when the stride is large, and stopping when there is no change. In addition, it can be estimated whether the object determined to be a pedestrian is walking, running, or stopped. Moreover, since the motion state of a pedestrian can be estimated, the information about the motion state of a user's pedestrian can be notified.

  The inferiority between the first contour line and the third contour line of the trapezoidal contour shape based on each captured image in which the trapezoidal contour shape can be detected by the contour shape detecting means. It is good also as an aspect which calculates the value which shows the time change of the object angle which is an angle of at least any one of the corners and the minor angle which the 2nd outline and the 3rd outline make. According to this, the value indicating the temporal change in the target angle can be used for pedestrian recognition. The minor angle indicates a smaller angle among the angles sharing the two sides with the vertex.

  In the case of the fifth aspect, the pedestrian candidate is a pedestrian based on the value indicating the temporal change in the target angle calculated by the change amount calculating unit as in the sixth aspect. It is preferable that the determination is performed. As described above, since the contour lines of the left and right legs of the pedestrian are extracted as the first contour line and the second contour line, when the pedestrian candidate is actually a pedestrian, The angle of inferiority formed by the third contour line and the angle of inferiority formed by the second contour line and the third wheel angle line (hereinafter referred to as the target angle) will change according to the change in the pedestrian's stride. Specifically, the target angle increases as the stride increases, and the target angle decreases as the stride decreases. In addition, when the pedestrian candidate is actually a pedestrian, the change in the target angle corresponds to the change in the stride of the pedestrian, so the pedestrian is based on the value indicating the temporal change in the target angle. It becomes possible to determine that the candidate is a pedestrian. Details are as follows.

  When there is no change in the target angle, there is a high possibility that the pedestrian has no change in stride (that is, a pedestrian who is stopped) or a non-pedestrian. On the other hand, when there is a change in the target angle, there is a high possibility that the pedestrian has a change in the stride (that is, a pedestrian who is walking or running). Therefore, it becomes possible to determine more accurately whether or not the pedestrian candidate is a pedestrian from the value indicating the temporal change in the target angle. Therefore, according to the structure of Claim 6, it becomes possible to determine more accurately whether a pedestrian candidate is a pedestrian based on the value which shows the time change of an object angle.

  In the case of carrying out like Claim 6, the aspect which estimates the movement state of a pedestrian based on the value which shows the time change of the object angle computed by the variation | change_quantity calculation means like Claim 7 It is preferable to do. If the pedestrian candidate is actually a pedestrian, the change in the target angle corresponds to the change in the pedestrian's stride, so the pedestrian's motion state can be determined from the value indicating the temporal change in the target angle. Can be estimated. For example, it is highly likely that the stride is changing when the time change is small, if it is large, running, and if there is no change, it is likely that it is stopping. In addition, it can be estimated whether the object determined to be a pedestrian is walking, running, or stopped. Moreover, since the motion state of a pedestrian can be estimated, the information about the motion state of a user's pedestrian can be notified.

  It is preferable that the pedestrian candidate is determined not to be a pedestrian when the lower end width calculated by the lower end width calculating means is not less than a predetermined threshold. Since the lower end width corresponds to the pedestrian's stride as described above, if the lower end width is a value that is too large as the pedestrian's stride, there is a high possibility that the pedestrian candidate is not a pedestrian. On the other hand, according to the configuration of claim 9, when the lower end width is not equal to or less than the predetermined threshold value, it is determined that the pedestrian candidate is not a pedestrian, so the lower end width is too large as the pedestrian's step length. When it is, it becomes possible to determine that a pedestrian candidate is not a pedestrian. Therefore, according to the structure of Claim 8, it becomes possible to determine more accurately that a pedestrian candidate is not a pedestrian.

  It is preferable that the pedestrian candidate is determined not to be a pedestrian when the target angle calculated by the target angle calculation means is not less than a predetermined threshold. Since the size of the target angle corresponds to the pedestrian's stride as described above, if the target angle is too large to correspond to the stride value that is too large for the pedestrian's stride, the pedestrian candidate is Most likely not a pedestrian. On the other hand, according to the configuration of claim 10, when the target angle is not equal to or less than the predetermined threshold, it is determined that the pedestrian candidate is not a pedestrian, so when the target angle is too large, It becomes possible to determine that the pedestrian candidate is not a pedestrian. Therefore, according to the structure of Claim 9, it becomes possible to determine more accurately that a pedestrian candidate is not a pedestrian.

  As in claim 10, the ratio of the lengths of the first contour line, the second contour line, and the third contour line in the trapezoidal contour shape detected by the contour shape detecting means is not within a predetermined value. In this case, it is preferable that the pedestrian candidate is determined not to be a pedestrian. The range of motion of the left and right legs and the ratio of the lengths of the left and right legs are determined to some extent in the human body. Therefore, when the pedestrian candidate is actually a pedestrian, the ratio of the lengths of the first contour line, the second contour line, and the third contour line is within a certain value. Therefore, accurately determining that the pedestrian candidate is not a pedestrian based on the ratio of the lengths of the first contour line, the second contour line, and the third contour line not being within a predetermined value. Is possible.

1 is a block diagram illustrating a schematic configuration of a pedestrian recognition system 100. FIG. (A) is the schematic diagram which looked at the own vehicle from the left direction, (b) is the schematic diagram which looked at the own vehicle from the top. It is a flowchart which shows the operation | movement flow of the pedestrian recognition apparatus 2. FIG. (A) is a figure which shows an example of the image of a pedestrian candidate, (b) is a figure which shows an example of the result of having performed edge extraction with respect to the image shown to (a), (c) is It is a figure which shows an example of a feature shape. It is a flowchart which shows an example of the flow of a feature shape detection process. It is a flowchart which shows an example of the flow of a time series process. It is a schematic diagram for demonstrating the cross-section width and the cross-section angle. It is a flowchart which shows an example of the flow of a pedestrian determination process. (A)-(c) is a schematic diagram which shows the example in case a pedestrian's right and left leg part is reflected only from the position below the crotch. (A) And (b) is a schematic diagram which shows the example in case a pedestrian's right and left leg part is reflected only from the position of the lower part rather than a crotch.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of a pedestrian recognition system 100 to which the present invention is applied. A pedestrian recognition system 100 shown in FIG. 1 is mounted on a vehicle, and includes a camera 1, a pedestrian recognition device 2, a display 3, a speaker 4, and an indicator 5. Hereinafter, a vehicle equipped with the pedestrian recognition system 100 is referred to as a host vehicle.

  As shown in FIGS. 2A and 2B, the camera 1 is attached, for example, in the vicinity of a room mirror in the host vehicle, and sequentially captures a predetermined range in front of the host vehicle. Therefore, the camera 1 corresponds to the imaging device of the claims. Note that the camera 1 captures images every 33 msec to 100 msec, for example. FIG. 2A is a schematic diagram of the host vehicle viewed from the left direction, and FIG. 2B is a schematic diagram of the host vehicle viewed from above. In addition, the camera 1 may be provided on the front of the vehicle. In the present embodiment, the following description will be made assuming that the camera 1 is, for example, a color camera.

  The camera 1 includes an optical system including a lens 11 and an image sensor 12. In the camera 1, optical image information input to the image sensor 12 through an optical system including the lens 11 is converted into captured image data (RGB signals) to obtain a color image, and the color image is output to the pedestrian recognition device 2. To do. Hereinafter, this color image is referred to as a captured image. The imaging element 12 is an element that converts an optical image to be imaged into image data, and a CCD (Charge Coupled Device) or the like is used. In the present embodiment, all pixels of the captured image are represented by RGB values in which each of the R value (red degree), the G value (green degree), and the B value (blue degree) is a digital value of 0 to 255. It will be described as a thing.

  In addition, in this embodiment, although the camera 1 showed the structure which images the predetermined range ahead of the own vehicle, it does not necessarily restrict to this. The camera 1 may be configured to capture the surroundings of the host vehicle other than the front, such as capturing a predetermined range in the rear. In the present embodiment, a configuration in which the camera 1 is a color camera will be described.

  The pedestrian recognition device 2 is composed mainly of a microcomputer composed of a CPU, ROM, RAM, EEPROM, etc., and executes various control programs stored in the ROM based on captured image data input from the camera 1. And the process which recognizes a pedestrian by performing image recognition is performed. As shown in FIG. 1, the pedestrian recognition device 2 includes, as functional blocks, an image acquisition unit 21, a pre-recognition 22, an edge image creation unit 23, a feature shape detection unit 24, a time series processing unit 25, a pedestrian determination unit 26, And a notification processing unit 27.

  Note that image recognition is a technique for recognizing the shape of an image by analyzing the image content of the image data. In image recognition, an outline that is an object is extracted from image data, separated from the background, and then the object is analyzed. A pedestrian means a person who moves on a road by a method that does not depend on a vehicle, and includes both a moving person and a stopped person.

  The display 3 is a small vehicle-mounted display, acquires image data output from the pedestrian recognition device 2, and displays an image indicated by the image data. With regard to image display, for example, a configuration using a display used in an in-vehicle navigation device may be used, or an in-vehicle head-up display may be used.

  The speaker 4 is a small vehicle-mounted speaker, acquires the audio signal output from the pedestrian recognition device 2, and outputs the sound indicated by the audio signal. With regard to audio output, for example, a configuration using a speaker used in an in-vehicle navigation device may be used, or an in-vehicle audio speaker may be used. The indicator 5 is, for example, an LED, and lights up when a signal output from the pedestrian recognition device 2 is acquired. In addition, it is good also as a structure which blinks LED instead of lighting of LED.

  Then, the operation | movement flow of the pedestrian recognition apparatus 2 is demonstrated using FIG. FIG. 3 is a flowchart showing an operation flow of the pedestrian recognition device 2. This flow is started when captured image data is input from the camera 1. Moreover, this flow shall be complete | finished, for example, when the power supply of the pedestrian recognition apparatus 2 is turned off, or when the ignition power supply of the own vehicle is turned off.

  First, in step S1, the image acquisition unit 21 acquires a captured image input from the camera 1, stores it in a temporary storage memory such as a RAM, for example, and proceeds to step S2. Therefore, the image acquisition unit 21 corresponds to the image acquisition unit in the claims.

  In step S2, the pre-recognition unit 22 performs pre-recognition processing, and proceeds to step S3. In the pre-recognition process, pedestrian candidates are detected by roughly identifying whether or not a pedestrian is present in the captured image. In the pre-recognition process, most pedestrians are detected as pedestrian candidates, and pedestrian candidates are detected by known pattern matching using a coarse classifier that is set to remove images that are clearly not pedestrians. Shall.

  Specifically, an example of an image of a pedestrian of any posture (that is, an example of a correct image) and an example of an image of a structure or the like that is clearly not a pedestrian (ie, an example of an incorrect image) Stored in a non-volatile memory or the like. Then, an image similar to the non-correct image example stored in the nonvolatile memory is removed, while an image similar to the correct image example is detected as a pedestrian candidate. Various specific algorithms for pattern matching can be employed. For example, the pattern matching may be performed using a pattern constructed by machine learning for a correct image example and a non-correct image example in advance. Further, the subsequent processing is performed by extracting an image of an appropriate rectangular area including the pedestrian candidate from the captured image.

  In step S3, if a pedestrian candidate is detected (YES in step S3), the process proceeds to step S4. If a pedestrian candidate is not detected (NO in step S3), the process returns to step S1. Repeat the flow.

  In step S4, the edge image creation unit 23 performs edge image creation processing, and proceeds to step S5. In the edge image creation process, a known edge extraction is performed on the pedestrian candidate image (see FIG. 4A) to extract edges (that is, contour lines) in the pedestrian candidate image (see FIG. 4). 4 (b)). Therefore, the edge image creation unit 23 corresponds to a contour line extraction unit. FIG. 4A is a diagram illustrating an example of a pedestrian candidate image, and FIG. 4B is an example of a result of edge extraction performed on the image illustrated in FIG. FIG.

  In addition to edge extraction, for example, image processing such as mirror image conversion for converting to a mirror image and distortion correction for correcting distortion generated in the peripheral portion of the image due to lens characteristics may be performed. Further, in a case where image processing such as edge image creation processing is performed in the pre-recognition processing, the edge image creation processing may not be performed in step S4, and the result of edge extraction in the pre-recognition processing may be used.

  In step S5, the feature shape detection unit 24 performs feature shape detection processing, and proceeds to step S6. Here, the outline of the feature shape detection process will be described with reference to the flowchart of FIG. FIG. 5 is a flowchart illustrating an example of the flow of the feature shape detection process.

  First, in step S51, a first feature edge detection process is performed. In the first feature edge detection process, an edge in the diagonally right direction (that is, an edge tilted diagonally to the right) that is the length within the threshold range is detected from the edge extracted in the edge image creation process. Hereinafter, the diagonally rightward edge having a length within the threshold range is referred to as a first feature edge. The threshold value here is a value that can be arbitrarily set, for example, a value that excludes an edge having a length that deviates from the length of the leg of the pedestrian in the captured image. What is necessary is just composition.

  If the first feature edge can be detected (YES in step S51), the process proceeds to step S52. If the first feature edge cannot be detected (NO in step S51), it is determined that the feature shape has not been detected, and the process proceeds to step S6. The first feature edge corresponds to the first contour line in the claims.

  In step S52, a second feature edge detection process is performed. In the second feature edge detection process, an edge in the diagonally left direction (that is, an edge tilted diagonally to the left) that is the length within the threshold range is detected from the edge extracted in the edge image creation process. Hereinafter, the diagonally left edge that is the length within the threshold range is referred to as a second feature edge. The threshold value here is a value that can be arbitrarily set, for example, a value that excludes an edge having a length that deviates from the length of the leg of the pedestrian in the captured image. What is necessary is just composition.

  If the second feature edge can be detected (YES in step S52), the process proceeds to step S53. If the second feature edge cannot be detected (NO in step S52), it is determined that the feature shape has not been detected, and the process proceeds to step S6. The second feature edge corresponds to the second contour line in the claims.

  In step S53, a third feature edge detection process is performed. In the third feature edge detection process, within the threshold range connecting the upper end of the first feature edge detected in step S51 and the upper end of the second feature edge detected in step S52 from the edge extracted in the edge image creation process. The edge of the length of is detected. Hereinafter, this edge is referred to as a third feature edge. The threshold here is a value that can be arbitrarily set, and for example, a value that excludes an edge having a length about the pedestrian's step length in the captured image may be set.

  If the third feature edge can be detected (YES in step S53), the process proceeds to step S54. If the third feature edge cannot be detected (NO in step S53), it is determined that the feature shape has not been detected, and the process proceeds to step S6. The third feature edge corresponds to the third contour line in the claims.

  In step S54, the ratio between the lengths of the line segments of the first feature edge detected in step S51, the second feature edge detected in step S52, and the third feature edge detected in step S53 is set threshold range. It is determined whether it is in. Specifically, the ratio between the first feature edge and the second feature edge, the ratio between the second feature edge and the third feature edge, and the ratio between the third feature edge and the first feature edge are all within the set threshold range. It is determined whether or not there is.

  The setting threshold range here is a value that can be arbitrarily set, and for example, a ratio that deviates from the ratio between the line segments of each feature edge detected from an actual pedestrian in the captured image. What is necessary is just to set it as the value which excludes. In addition, the setting threshold value is different for each of the ratio between the first feature edge and the second feature edge, the ratio between the second feature edge and the third feature edge, and the ratio between the third feature edge and the first feature edge. It is good also as a structure to use.

  If it is determined that the length ratio between the line segments of each feature edge is within the set threshold range (YES in step S54), the process moves to step S55, assuming that the feature shape has been detected. If it is determined that the length ratio between the line segments of each feature edge is not within the set threshold range (NO in step S54), it is determined that the feature shape has not been detected, and the process proceeds to step S6.

  Here, an example of the feature shape is shown using FIG. FIG. 4C is a diagram illustrating an example of a feature shape. In FIG. 4C, A indicates the first feature edge, B indicates the second feature edge, C indicates the third feature edge, and white circles indicate the end portions of the feature edges. The feature shape has a trapezoidal shape composed of a first feature edge and a second feature edge having a C-shape and a third feature edge which is an edge in the left-right direction. In the example of FIG. 4C, there is almost no inclination of the third edge, but the third feature edge constituting the feature shape may be inclined in the left oblique direction or the right oblique direction. Therefore, the feature shape detection unit 24 corresponds to the contour shape detection means in the claims. The characteristic shape corresponds to the trapezoidal outline shape of the claims.

  In step S55, line segment / color information storage processing is performed, and the flow proceeds to step S6. In the line segment / color information storage process, the line segment information of the first feature edge, the second feature edge, and the third feature edge in the detected feature shape is stored in a temporary storage memory such as a RAM. For example, pixel coordinates of the upper end and lower end of the first edge are stored as line segment information of the first feature edge, and pixel coordinates of the upper end and lower end of the second edge are stored as line segment information of the second feature edge. And the pixel coordinates of the left end and the right end of the third edge may be stored as the line segment information of the third feature edge. The pixel coordinates are represented by two axes, the left-right direction (hereinafter, x-axis direction) and the vertical direction (hereinafter, y-axis direction), and the pixel coordinates of the upper left corner of the captured image are (0, 0).

  In the line segment / color information storage process, the color information about the line segment of the first feature edge, the second feature edge, and the third feature edge, or the first feature edge, the second feature edge, and the third feature. Color information about the edge of the edge is calculated and stored in a temporary storage memory such as a RAM. In the present embodiment, RGB values of a plurality of pixels in the vicinity of an end portion where the first feature edge and the third feature edge intersect (that is, the upper end portion of the first feature edge or the left end portion of the third feature edge), Color information is the average value of the RGB values of a plurality of pixels near the end where the second feature edge and the third feature edge intersect (that is, the upper end of the second feature edge or the left end of the third feature edge). It shall be calculated and saved as

  It is assumed that the plurality of pixels near the end where the first feature edge and the third feature edge intersect are several pixels on the dominant angle side formed by the first feature edge and the third feature edge. Further, it is assumed that the plurality of pixels near the end where the second feature edge and the third feature edge intersect are several pixels on the dominant angle side formed by the second feature edge and the third feature edge. The dominant angle indicates the larger angle among the angles sharing the two sides with the vertex. It also includes the pixels of the edge itself.

  In the example of the present embodiment, a configuration using color information about an end portion where the first feature edge and the third feature edge intersect and an end portion where the second feature edge and the third feature edge intersect is shown. However, this is not necessarily the case. For example, an average value including RGB values of a plurality of pixels near the other end may be used as color information, or an average value of RGB values of a plurality of pixels near the line segment of each feature edge is used as the color information. It is good also as a structure.

  Returning to FIG. 3, in step S6, when the feature shape can be detected by the feature shape detection process (YES in step S6), the process proceeds to step S7. If the feature shape cannot be detected by the feature shape detection process (NO in step S6), the process returns to step S1 and the flow is repeated.

  In step S7, the time series processing unit 25 performs time series processing, and proceeds to step S8. Here, an outline of the time series processing will be described with reference to the flowchart of FIG. FIG. 6 is a flowchart illustrating an example of a flow of time series processing.

  First, in step S71, a first shape information storage process is performed, and the process proceeds to step S72. In the first shape information storage process, shape information about the feature shape detected in the feature shape detection process is calculated and stored in a temporary storage memory such as a RAM, for example. The shape information is the shape width and the shape angle of the characteristic shape.

  Here, the width of the C and the angle of the C will be described with reference to FIG. FIG. 7 is a schematic diagram for explaining a C-shaped width and a C-shaped angle. FIG. 7A shows the first feature edge of the feature shape, B shows the second feature edge, and C shows the third feature edge. In addition, D indicates a cross-section width, and E indicates a cross-section angle.

  As shown in FIG. 7, the C-shaped width is the distance between the lower end of the first feature edge and the lower end of the second feature edge in the feature shape (that is, the lower end of the first feature edge and the second feature). (Interval in the x-axis direction of the lower end portion of the edge). For example, the C-shaped width may be calculated by subtracting the x coordinate of the lower end portion of the first feature edge from the x coordinate of the lower end portion of the second feature edge. Note that the C-shaped width corresponds to the lower end width of the claims, and the time-series processing unit 25 corresponds to the lower end width calculating means of the claims.

  Further, as shown in FIG. 7, the C-shaped angle is a minor angle formed by the first feature edge and the third feature edge in the feature shape, and a minor angle formed by the second feature edge and the third feature edge. Is the angle. The minor angle indicates the smaller one of the angles sharing the two sides with the vertex. For example, the C-shaped angle is an inferior angle formed by the first feature edge and the third feature edge based on the line segment information of each feature edge stored in the line segment / color information storing process, and the second feature edge And the third feature edge may be calculated by calculating the subordinate angle. The square angle corresponds to the target angle of the claims, and the time-series processing unit 25 corresponds to the target angle calculation means of the claims.

  In the present embodiment, a configuration is adopted in which both the angle of the minor angle formed by the first feature edge and the third feature edge and the angle of the minor angle formed by the second feature edge and the third feature edge are formed into a C-shaped angle. Although shown, it is not necessarily limited to this. For example, it is good also as a structure which makes only one of the above-mentioned a square-shaped angle, and good also as a structure which uses both average values.

  In step S72, it is determined whether or not shape information is stored in the temporary storage memory for the captured images that are equal to or greater than the set number of frames. Note that the number of set frames is a value that can be arbitrarily set, and may be a value that can calculate the period of change in the C-shaped width or C-shaped angle. If it is determined that shape information is stored for the number of captured images equal to or greater than the set number of frames (YES in step S72), the process proceeds to step S8. If it is not determined that shape information has been stored for the number of captured images equal to or greater than the set number of frames (NO in step S72), the process proceeds to step S73.

  In step S73, the captured image of the next frame of the captured image that has been processed is acquired from the captured images sequentially acquired from the camera 1 by the image acquisition unit 21, and the process proceeds to step S74. Specifically, when processing in steps S74 to S75 described later is not performed, a captured image of the next frame of the captured image acquired in step S1 is acquired. On the other hand, if the processing of steps S74 to S75 described later has already been performed before the processing of step S73, the captured image of the next frame of the latest captured image that has been subjected to the color similar image creation processing described later is obtained. get.

  In step S74, a color similar image creation process is performed, and the process proceeds to step S75. In the color-similar image creation process, based on the captured image acquired in step S73, an image representing the color similarity (hereinafter referred to as a color-similar image) using the color information stored in the line segment / color information storage process as a reference value. create. The creation of the color-similar image is performed for a region (hereinafter, a target region) corresponding to a region where a pedestrian candidate is detected in the captured image. The target area is an area where a pedestrian candidate is estimated to be captured, and an area indicated by the same coordinates as the area where the pedestrian candidate is detected in the pre-recognition process (hereinafter, the same area) may be the target area. Good. In addition, a region that is wider in the vertical and horizontal directions by a predetermined width than the same region may be the target region, and a region that is shifted from the same region in the vertical and horizontal directions may be the target region.

  A color-similar image is created by calculating the absolute value of the difference between the RGB value of each pixel and the RGB value as the reference value for each pixel in the target area, and using the calculated absolute value as a new RGB value. . Note that the absolute value of the difference between the RGB value of each pixel and the RGB value as the reference value is calculated for each of R, G, and B. In the color similar image, a portion corresponding to the feature shape is displayed in black, and a portion corresponding to the background is displayed in white. This is due to the following reason.

  When the RGB value is R = 255, G = 255, and B = 255, it is displayed in white, and when the RGB value is R = 0, G = 0, and B = 0, it is displayed in black. A color area similar to the color information stored in the information storage process is displayed in black, and a color area not similar to the color information is displayed in white. Since the color information stored in the line segment / color information storage process is the average of the RGB values of a plurality of pixels in the vicinity of the feature shape, the feature shape is displayed in black in the color similar image, and the background other than the feature shape is displayed in white. It will be.

  In step S75, a second shape information storage process is performed, and the flow returns to step S72 to repeat the flow. In the second shape information storage process, shape information in a portion corresponding to the feature shape is calculated from the color similar image created in the color similar image creation process, and stored in a temporary storage memory such as a RAM. In addition, it is good also as a structure which complete | finishes a flow, when the part corresponded to a feature shape cannot be detected from a color similar image.

  In the color-similar image, since the portion corresponding to the feature shape is displayed in black as described above, the feature shape is detected using this, and shape information such as the C-shape width and the C-shape angle is calculated. For example, a feature shape is detected by performing edge extraction on a color-similar image, and pixel coordinates of locations corresponding to the first feature edge, the second feature edge, and the third feature edge of the feature shape are obtained. The configuration may be such that the shape information is calculated based on the coordinates in the same manner as described above.

  Returning to FIG. 3, in step S8, the pedestrian determination unit 26 performs a pedestrian determination process, and proceeds to step S9. Here, an outline of the pedestrian determination process will be described using the flowchart of FIG. FIG. 8 is a flowchart illustrating an example of a flow of pedestrian determination processing. Moreover, the pedestrian determination part 26 is corresponded to the variation | change_quantity calculation means of a claim.

  First, in step S81, a change presence / absence determination process is performed, and the process proceeds to step S82. In the change presence / absence determination process, changes in the C-shape width and the C-shape angle of the feature shape (that is, the C-shape angle) based on the plurality of shape information stored in the temporary storage memory for the captured images of the set number of frames or more in the time series processing (that is, Whether there is a change in the value of the shape information). For example, the change amount of the C-shaped width and the change amount of the C-shaped angle between two consecutive frames are calculated, and when this change amount is within the threshold range, it is determined that there is no change, When the amount of change is not within the threshold range, it is determined that there is a change. The amount of change corresponds to a value indicating a temporal change in the lower end width of the claims and a value indicating a temporal change in the target angle.

  The threshold value here is a value that can be arbitrarily set, and for example, a value that is about the calculation error of the shape information may be set. The threshold value may be configured such that different values are set for the C-shaped width and the C-shaped angle, or the same value may be set. Further, in the present embodiment, the configuration for calculating the change amount of the C-shaped width and the change amount of the C-shaped angle between two consecutive frames is shown, but the present invention is not necessarily limited thereto. For example, the change amount of the C-shaped width and the change amount of the C-shaped angle between two frames other than between consecutive frames may be calculated. Moreover, it is good also as a structure which calculates and uses the average value of the variation | change_quantity of the C-shaped width | variety between three or more continuous frames, and the average value of the variation | change_quantity of a C-shaped angle.

  In step S82, if it is determined in the change presence / absence determination process that there is a change in the C-shaped width or C-shaped angle (YES in step S82), the process proceeds to step S83. If it is determined in the change presence / absence determination process that there is no change in the C-shaped width or C-shaped angle (NO in step S82), the process proceeds to step S87.

  In step S83, a change period calculation process is performed, and the process proceeds to step S84. In the change period calculation process, the C-shaped width and the C-shaped angle change from decreasing to increasing based on a plurality of pieces of shape information stored in the temporary storage memory for captured images that are equal to or larger than the set number of frames in the time series processing. By finding the point (the point at the bottom of the valley below) and the point where the C-shaped width and the C-shaped angle change from increasing to decreasing (the point at the top of the mountain) along the time series, And the value of the half cycle (1/2 cycle) of the change in the C-shaped angle is calculated. The value of the half cycle is the time interval from the bottom point of the valley to the top point of the mountain, and the top point of the valley to the bottom of the valley for the plurality of shape information stored in the temporary storage memory. It is calculated by taking the average of the time interval to the point. The half cycle value is calculated for each of the C-shaped width and the C-shaped angle. In addition, the value of this half cycle is equivalent to the value which shows the time change of the lower end width | variety of a claim, and the value which shows the time change of an object angle.

  In step S84, it is determined whether or not the half cycle value calculated in the change cycle calculation process is greater than or equal to a predetermined value. If the half cycle value is equal to or greater than the predetermined value (YES in step S84), the process proceeds to step S86. If the half cycle value is not equal to or greater than the predetermined value (NO in step S84), the process proceeds to step S85.

  The predetermined value here is a value that can be arbitrarily set, and it is only necessary to set a half-width value of a C-shaped width or a C-shaped angle calculated for a pedestrian who is actually running. . In addition, what is necessary is just to set it as the structure by which a different value is set about each of a predetermined value about a square shape width and a square shape angle. If both the half-width value of the C-shaped width and the half-cycle value of the C-shaped angle are greater than or equal to a predetermined value, the process proceeds to step S86. What is necessary is just to set it as the structure which moves to step S85.

  In step S85, the pedestrian candidate is determined to be a pedestrian, and the pedestrian is estimated to be walking, and the process proceeds to step S9. In step S86, the pedestrian candidate is determined to be a pedestrian, and the pedestrian is estimated to be traveling, and the process proceeds to step S9.

  In the present embodiment, the configuration in which the pedestrian's motion state (whether walking or running) is estimated based on whether or not the half-cycle value is equal to or greater than a predetermined value is not necessarily limited thereto. For example, instead of a half cycle value, another cycle value such as one cycle or two cycles may be used. Moreover, it is good also as a structure which estimates the pedestrian's exercise | movement state (it is walking or is drive | working) according to whether the variation | change_quantity per hour of a C-shaped width and a C-shaped angle is more than predetermined value.

  In step S87, it is determined whether or not the value of the shape information is within the threshold range. For example, it is determined whether or not the C-shaped width and the C-shaped angle stored in the primary storage memory in the first shape information storing process are within the threshold range. And when it determines with it being in the threshold value range (it is YES at step S87), it moves to step S88. If it is determined that the value is not within the threshold range (NO in step S87), the process proceeds to step S89.

  What is necessary is just to set it as the structure by which a different value is set for each of a threshold value width and a letter-shaped angle. The threshold value for the C-shaped width is a value that can be arbitrarily set, and for example, a value that is large enough to deviate from the length of the pedestrian's step in the captured image may be set. According to this, it is possible to determine that a pedestrian candidate is not a pedestrian when the C-shaped width is too large as the pedestrian's step length. Further, the threshold value for the C-shaped angle is a value that can be arbitrarily set, and for example, sets the value of the C-shaped angle when it is large enough to deviate from the length of the pedestrian's step in the captured image. What is necessary is just composition. According to this, it is possible to determine that a pedestrian candidate is not a pedestrian when the cross-section angle is a value that is too large as the pedestrian's cross-section angle.

  In step S88, it is determined that the pedestrian candidate is a stopped pedestrian, and the process proceeds to step S9. In step S89, it is determined that the pedestrian candidate is a non-pedestrian, and the process proceeds to step S9.

  In step S9, when the pedestrian candidate is determined to be a pedestrian in the pedestrian determination process (YES in step S9), the process proceeds to step S10. If the pedestrian candidate is determined to be a non-pedestrian in the pedestrian determination process (NO in step S9), the flow returns to step S1 to end the flow.

  In step S10, the notification processing unit 27 performs notification processing and ends the flow. In the notification process, a process of notifying the occupant of the host vehicle of the presence of a pedestrian is performed. For example, an icon image or text indicating the presence of a pedestrian may be displayed on the display 3. Moreover, it is good also as a structure which performs highlighted display, such as surrounding a pedestrian in a captured image with a frame while displaying a captured image on the display 3. FIG. In addition, voice guidance indicating the presence of a pedestrian may be output from the speaker 4, or an indicator 5 indicating the presence of a pedestrian may be lit or blinked.

  Furthermore, it is good also as a structure which performs a various process according to the motion state of the pedestrian estimated by the pedestrian determination process. For example, an icon image or text indicating the type of the estimated motion state (that is, stopping, walking, and running) may be displayed on the display 3, and voice guidance indicating the estimated motion state is displayed from the speaker 4. It may be output, or the indicator 5 corresponding to the estimated type of exercise state may be turned on or blinked. Moreover, while displaying a captured image on the display 3, it is good also as a structure which changes highlighting, such as surrounding a pedestrian in a captured image with a frame, according to the kind of the estimated exercise state. For example, it is good also as a structure which performs the emphasis display which urges the passenger | crew of the own vehicle to call attention more strongly in order of driving | running | working, walking, and stopping.

  When a plurality of pedestrian candidates are detected in one frame, the processing shown in the flow of FIG. 3 may be performed for each pedestrian candidate to recognize the pedestrian.

  In the pedestrian recognition device 2 of the present embodiment, as described above, the first feature edge tilted to the right in the captured image, the second feature edge tilted to the left, and the upper end of the first feature edge and the first feature edge Based on the trapezoidal feature shape consisting of the third feature edge connecting the upper ends of the two feature edges, it is determined whether or not the pedestrian candidate detected from the captured image is a pedestrian and recognizes the pedestrian. .

  As shown in FIGS. 9 (a) to 9 (c), 10 (a) and 10 (b), this characteristic shape is blocked by an obstacle from the upper body of the pedestrian to the position below the crotch. Detected when the left and right legs of the pedestrian are visible only from the position below the inseam, such as when the subject is in the lower part of the leg and below the inseam due to light adjustment The Specifically, the contour lines of the left and right legs of the pedestrian are extracted as the first feature edge and the second feature edge, and the light / dark boundary and the contour line of the obstacle caused by light adjustment are extracted as the third feature edge. .

  9A is an example blocked by an umbrella, FIG. 9B is an example blocked by baggage, and FIG. 9C is an example blocked by a guardrail. It is. FIG. 10A is an example where the skirt blocks the skirt, and FIG. 10B is an example where only the position below the pedestrian's inseam is illuminated by the headlight at night. 9A to 9C, 10A, and 10B, A represents the first feature edge, B represents the second feature edge, and C represents the third feature edge. Yes.

  Therefore, based on the feature shape, the left and right legs of the pedestrian are only seen from the position below the crotch, and the contours of the left and right legs of the pedestrian in the captured image are inverted V. Even when the character does not have a letter shape, it is possible to accurately determine whether or not the pedestrian candidate is a pedestrian and recognize the pedestrian. Therefore, according to the configuration of the present embodiment, the pedestrian is recognized in a situation in which the upper body cannot be confirmed on the image even when the contour formed by the left and right legs of the pedestrian in the captured image does not have an inverted V shape. can do. As a result, it is possible to further improve the accuracy of pedestrian recognition in situations where the upper body cannot be confirmed on the image.

  If there is no change in the above-mentioned c-shaped width or c-shaped angle in the pedestrian candidate in the captured image, it is considered that the pedestrian candidate is not moving, so the pedestrian candidate is a stopped pedestrian There is a high possibility of being a non-pedestrian such as a structure. On the other hand, when there is a change in the C-shaped width or the C-shaped angle, it is considered that the pedestrian candidate is exercising, so there is a high possibility that the pedestrian is walking or running. According to the configuration of the present embodiment, when it is determined that there is a change in the C-shaped width or the C-shaped angle, it is determined that the pedestrian candidate is a pedestrian. Therefore, it can be determined more accurately that the pedestrian candidate is a pedestrian based on the presence or absence of changes in the C-shaped width or the C-shaped angle.

  Further, according to the configuration of the present embodiment, a pedestrian is based on a value indicating a temporal change, such as a half-width value of a C-shaped width or a C-shaped angle change in a pedestrian candidate in a captured image. It is possible to estimate an exercise state such as whether the object determined to be walking, running, or stopped. Moreover, since the motion state of a pedestrian can be estimated, it is also possible to notify information about the motion state of the user's pedestrian.

  Furthermore, according to the configuration of the present embodiment, the ratio between the line segments of each feature edge detected in the feature shape detection process is the ratio between the line segments of each feature edge detected from the actual pedestrian in the captured image. If a set threshold range that excludes a ratio that deviates from the above is exceeded, the pedestrian candidate is not determined to be a pedestrian as a feature shape cannot be detected. Therefore, it becomes possible to accurately determine pedestrian candidates that are not pedestrians based on the fact that the ratio between the line segments of each feature edge is not within the set threshold range.

  Further, according to the configuration of the present embodiment, when a feature shape is detected from a certain captured image, the color information stored in the line segment / color information storage process is used as the reference value for the captured image of the frame following the captured image. The feature shape is detected by creating a color-similar image. Therefore, once the feature shape is detected from the captured image, it is not necessary to perform pre-recognition processing and feature shape detection processing for the captured image of the subsequent frame every time, and the processing load on the pedestrian recognition device 2 can be reduced. it can.

  In this embodiment, when a feature shape is detected from a certain captured image, a color-similar image is created using the color information stored in the line segment / color information storage processing as a reference value for the captured image of the frame following the captured image. However, the present invention is not limited to this. For example, when a feature shape is detected from a certain captured image, the feature shape is detected by performing pre-recognition processing, edge image creation processing, and feature shape detection processing each time for a captured image of a frame following the captured image. It is good also as a structure.

  In the present embodiment, the configuration for performing the pre-recognition process is shown, but the present invention is not necessarily limited thereto. For example, a configuration may be adopted in which edge image creation processing is performed on the entire captured image without performing pre-recognition processing to detect a feature shape. In this case, a configuration may be adopted in which a region in a predetermined range including a region where the feature shape is detected is handled as a region of a pedestrian candidate image.

  In the present embodiment, the configuration using both the C-shaped width and the C-shaped angle is shown, but the present invention is not necessarily limited thereto. For example, only one of the C-shaped width and the C-shaped angle may be used.

  In this embodiment, although the structure which estimates the pedestrian's motion state was shown, it does not necessarily restrict to this. For example, it is good also as a structure which determines a pedestrian candidate as a pedestrian, when there is a change of the above-mentioned C-shaped width | variety or C-shaped angle in the pedestrian candidate in a captured image, without estimating a motion state. In this case, when there is no change in the C-shaped width or C-shaped angle and the C-shaped width or C-shaped angle is within the threshold range, the pedestrian candidate is determined to be a pedestrian. The pedestrian candidate may be determined to be a non-pedestrian when the C-shaped width or the C-shaped angle is not within the threshold range.

  In addition, the pedestrian recognition device 2 includes an identifier for identifying a pattern other than a feature shape, such as an identifier for identifying a pattern in which the contour of the left and right legs of the pedestrian in the captured image is an inverted V shape. It is preferable to use a combined configuration. According to this, it becomes possible to recognize a pedestrian image in which a feature shape is not detected as a pedestrian, and the recognition performance of the pedestrian is further improved.

  Further, in the present embodiment, an example in which the RGB color model is used as the color expression method has been described. However, the present invention is not limited to this, and the present invention is also applied to a case where a color expression method other than the RGB color model is used. It is good also as a structure. Further, in the present embodiment, the case where a color camera is used as the camera 1 has been described as an example. However, another type of camera such as a monochrome camera or an infrared camera may be used as the camera 1.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the technical means disclosed in different embodiments can be appropriately combined. Such embodiments are also included in the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Camera (imaging device), 2 Pedestrian recognition device, 3 Display, 4 Speaker, 5 Indicator, 11 Lens, 12 Image sensor, 21 Image acquisition part (image acquisition means), 22 Pre-recognition part, 23 Edge image creation part ( (Contour line extraction means), 24 feature shape detection section (contour shape detection means), 25 time series processing section (lower end width calculation means, target angle calculation means), 26 pedestrian determination section (change amount calculation means), 27 notification processing , 100 Pedestrian recognition system

Claims (10)

A pedestrian recognition device that recognizes a pedestrian by detecting a pedestrian candidate by image recognition from a captured image obtained by imaging the periphery of the vehicle with an imaging device mounted on a vehicle,
Image acquisition means for acquiring a captured image captured by the imaging device;
Contour extraction means for extracting a contour line in the image from the captured image acquired by the image acquisition means;
Based on the contour line extracted by the contour line extraction means, a first contour line that is a slanted right contour line, a second contour line that is a slanted left contour line, and the first contour line A contour shape detecting means for detecting a trapezoidal contour shape which is a trapezoidal contour shape composed of a third contour line which is a contour line connecting the upper end and the upper end of the second contour line;
When the trapezoidal contour shape can be detected by the contour shape detecting means, based on the trapezoidal contour shape, it is determined whether or not the pedestrian candidate in the captured image is a pedestrian. A pedestrian recognition apparatus characterized by recognizing a pedestrian. In claim 1,
The image acquisition means sequentially acquires captured images captured by the imaging device,
From the trapezoidal contour shape detected by the contour shape detection unit, a lower end width calculation unit that calculates a lower end width that is an interval between the lower end of the first contour line and the lower end of the second contour line in the trapezoidal contour shape. When,
Based on the lower end width calculated by the lower end width calculating unit for each captured image that was able to detect the trapezoidal outline shape by the contour shape detecting unit among the captured images sequentially acquired by the image acquiring unit, A pedestrian recognition device comprising: a change amount calculating means for calculating a value indicating a temporal change in the lower end width. In claim 2,
A pedestrian recognition apparatus, wherein the pedestrian candidate is determined to be a pedestrian based on a value indicating a temporal change in the lower end width calculated by the change amount calculation means. In claim 3,
A pedestrian recognition apparatus characterized by estimating a motion state of the pedestrian based on a value indicating a temporal change in the lower end width calculated by the change amount calculation means. In claim 1,
The image acquisition means sequentially acquires captured images captured by the imaging device,
An inferior angle formed by the first contour line and the third contour line of the trapezoidal contour shape from the trapezoidal contour shape detected by the contour shape detecting means, and the second contour line and the third contour line. A target angle calculation means for calculating a target angle that is an angle of at least one of the minor angles formed by
Based on the target angle calculated by the target angle calculation unit for each captured image that was able to detect the trapezoidal contour shape by the contour shape detection unit among the captured images sequentially acquired by the image acquisition unit, A pedestrian recognition apparatus comprising: a change amount calculation unit that calculates a value indicating a temporal change in the target angle. In claim 5,
A pedestrian recognition apparatus, wherein the pedestrian candidate is determined to be a pedestrian based on a value indicating a temporal change in the target angle calculated by the change amount calculation means. In claim 6,
The pedestrian recognition apparatus characterized by estimating the movement state of the said pedestrian based on the value which shows the temporal change of the said target angle computed by the said variation | change_quantity calculation means. In any one of Claims 2-4,
The pedestrian recognition apparatus characterized by determining that the pedestrian candidate is not a pedestrian when the lower end width calculated by the lower end width calculating means is not less than a predetermined threshold. In any one of Claims 5-7,
The pedestrian recognition apparatus characterized by determining that the pedestrian candidate is not a pedestrian when the target angle calculated by the target angle calculation means is not less than a predetermined threshold. In any one of Claims 1-9,
When the ratio of the lengths of the first contour line, the second contour line, and the third contour line in the trapezoidal contour shape detected by the contour shape detecting means is not within a predetermined threshold. And determining that the pedestrian candidate is not a pedestrian.