JP5699497B2 - Image processing apparatus and image processing program - Google Patents

Description

  The technology disclosed in the present application relates to an image processing apparatus and an image processing program.

  Conventionally, for example, for a face in an image, a technique for detecting both end positions and center positions of the face by extracting edges on the image and obtaining the face orientation from the detected both end positions and center position of the face is known. is there.

  Further, as information for obtaining the face orientation, there is a technique for detecting the position of a face part that is a part constituting the face such as a nose or eyes. For example, an edge image is generated by extracting an edge from the face image, and the position of the face part on the image is detected by analyzing the shape of a histogram obtained by projecting the edge image in a predetermined direction. In addition to the technique using image edge information, there is a technique for detecting the position of a facial part on a facial image based on luminance-based feature data such as Haar feature data.

JP 2007-72628 A

  However, the technology for extracting edges from images captures changes in luminance values as edges, so it is easily affected by changes in the light applied to the subject, and detection of facial parts due to image contrast degradation and image blurring. There is a problem that accuracy may be lowered.

  For example, if the light emitted to the face that is the subject is not uniform, not only the edge formed in the face image by the facial parts, but also the boundary between the light and shadow formed in the face image as an edge It will be extracted. For this reason, the detection accuracy of face parts may be reduced.

  In addition, there may be a case where the contrast of the face image changes with changes in the intensity of light applied to the face that is the subject or the irradiation range. For example, when an edge is extracted from a face image, an arbitrary threshold value is set to determine whether or not the luminance value has changed. However, it is practically difficult to set a threshold value for accurately extracting only the edges formed in the face image by the face parts, assuming that the contrast of the image is lowered in advance. In addition, there is a method of calculating a different threshold value for each part of an image, but it is necessary to set a parameter based on a luminance value change model in order to calculate the threshold value. For this reason, if the threshold value and the parameter are not changed according to the change of the external light, the detection accuracy of the face part may be lowered with the change of the light intensity and the irradiation range.

  Further, depending on individual differences in focus performance of a photographing apparatus that photographs a face that is a subject, how a lens is attached, and the like, a case where a face image with blurred image is photographed may be considered. However, assuming that image blur caused by individual differences in focus performance of the imaging device is assumed in advance, setting a threshold value for accurately extracting only the edges formed on the face image by the facial parts is a threshold value for each imaging device. It is practically difficult to correct this. For this reason, if the threshold value is not changed, the detection accuracy of the face part may decrease due to individual differences in the focusing performance of the photographing apparatus, the degree of lens attachment, and the like.

  As described above, in the technique of detecting a face part from a face image by extracting an edge, the detection accuracy is lowered due to a change in light irradiated to a subject, a decrease in contrast, an image blur, or the like.

  Further, the technique using the Haar feature amount described above is also a technique for handling the luminance value of the face image. For this reason, the detection accuracy of face parts is easily affected by changes in the light applied to the face, similar to the technique for extracting the edge, and is caused by changes in the light applied to the subject, a decrease in contrast, image blurring, etc. In some cases, the detection accuracy of face parts may be reduced. The Haar feature amount is a technique for determining whether or not the image includes a detection target based on whether or not the light / dark pattern of the image matches the template. The light and dark pattern appearing on the image is easily affected by changes in external light. Therefore, since the appearance of the light / dark pattern changes even for the same subject, it may be difficult to evaluate whether or not it matches the template, and the detection accuracy may be reduced.

  The disclosed technology has been made in view of the above, and can prevent a reduction in detection accuracy when detecting a feature point region such as a facial part from an image even under the influence of external light. An object of the present invention is to provide an image processing apparatus and an image processing program.

  In one aspect, an image processing apparatus disclosed in the present application includes an acquisition unit and a detection unit. The acquisition unit obtains luminance values of the first pixel included in the image and a plurality of pixels located at a distance of one pixel or more from the first pixel, and the first pixel is included in the image. Acquire each while scanning. The detection unit detects a feature point in the image based on the luminance values of the first pixel and the plurality of pixels acquired while scanning.

  According to one aspect of the technology disclosed in the present application, it is possible to prevent a decrease in detection accuracy when a feature point region is detected from an image even under the influence of external light.

FIG. 1 is a diagram used for explaining the image processing apparatus according to the first embodiment. FIG. 2 is a functional block diagram illustrating the configuration of the image processing apparatus according to the first embodiment. FIG. 3 is a diagram illustrating an example of information stored in the feature amount storage unit according to the first embodiment. FIG. 4 is a diagram used for explaining the operation of the detection unit according to the first embodiment. FIG. 5 is a supplementary diagram used to describe the operation of the detection unit according to the first embodiment. FIG. 6 is a supplementary diagram used to describe the operation of the detection unit according to the first embodiment. FIG. 7 is a supplementary diagram used to describe the operation of the detection unit according to the first embodiment. FIG. 8 is a diagram illustrating an example of the feature amount image according to the first embodiment. FIG. 9 is a diagram illustrating an overall flow of processing by the image processing apparatus according to the first embodiment. FIG. 10 is a diagram illustrating a flow of feature point detection processing by the image processing apparatus according to the first embodiment. FIG. 11 is a diagram for explaining the processing result of the image processing apparatus according to the first embodiment. FIG. 12 is a diagram for explaining a processing result by the image processing apparatus for an image with reduced contrast according to the first embodiment. FIG. 13 is a diagram for explaining a processing result by the image processing apparatus for an image in which the intensity of irradiation light according to the first embodiment is changed. FIG. 14 is a diagram illustrating an example of an electronic device that executes an image processing program.

  Hereinafter, an embodiment of an image processing apparatus and an image processing program disclosed in the present application will be described in detail with reference to the drawings. The technology disclosed in the present application is not limited by the examples described later as an embodiment of the image processing program and the image processing apparatus disclosed in the present application. In the following embodiments, an image in which a human face is reflected in an image is called a face image. An embodiment in which a feature point indicating a region corresponding to, for example, a nostril is detected from a face image will be described. In addition, the techniques disclosed in the following embodiments can be appropriately combined within a range that does not cause a contradiction in processing contents.

  FIG. 1 is a diagram used for explaining the image processing apparatus according to the first embodiment. 1A shown in FIG. 1 represents a face image. 1B shown in FIG. 1 represents a face image photographed when light having a different intensity or irradiation range from the light irradiated on the face when the face image 1A was photographed. 1C shown in FIG. 1 is a graph showing a change in luminance value corresponding to a certain partial area in the face image 1A. The partial region shown in 1C includes a region R1 in which the luminance value of each pixel increases from the inside to the outside of the region. 1D shown in FIG. 1 is a graph showing a change in luminance value corresponding to a certain partial area in the face image 1B. Similar to the partial region shown in 1C, the partial region shown in 1D includes a region R2 in which the luminance value of each pixel increases from the inside to the outside of the region. Here, if the xy coordinate axes of the image are the x-axis and y-axis, respectively, and the luminance value of the pixel at the position indicated by the xy coordinates is taken on the z-axis, the change state of the luminance value is regarded as a three-dimensional curved surface. be able to. In such a luminance curved surface, the region R1 or the region R2 is represented as a hole having a bottom recessed in the middle of the region.

  When a person's face is photographed from the front, the area corresponding to the nostril shown in the face image has a deeper nose than the surrounding area. As long as there is no shadow. Accordingly, the region corresponding to the nostril increases like the region R1 and the region R2 as the luminance value of each pixel increases from the inside to the outside of the region. In other words, the luminance value increases from the outside to the inside of the region. The inventor noticed that it became smaller as it went. Furthermore, the inventor has realized that the position of the facial part on the image can be specified without extracting an edge from the image by using the feature of the luminance value change on the image. The image processing apparatus according to the first embodiment is a partial area in the face image, such as the area R1 and the area R2, in which the luminance value of each pixel gradually increases from the inside to the outside of the area. It is noted that the feature of the brightness value change is retained without being affected by the change in the light applied to the face. That is, the region R1 and the region R2 shown in FIG. 1 retain the characteristics of the change in luminance value not only when the light irradiated to the face changes, but also when the contrast of the entire face image is reduced, The same applies to any case where the image is blurred.

  Therefore, in the image processing apparatus according to the first embodiment, when detecting the position of the facial part on the face image, the luminance value of each pixel increases from the inside to the outside of the region, like a nostril. By using the pixel position within the area, the reduction in detection accuracy of the face part is prevented.

[Configuration of Image Processing Apparatus (Example 1)]
FIG. 2 is a functional block diagram illustrating the configuration of the image processing apparatus according to the first embodiment. As illustrated in FIG. 2, the image processing apparatus 300 according to the first embodiment is mounted on the vehicle 1, for example. The image processing device 300 is connected to one or a plurality of imaging devices 100 and an output device 200 mounted on the vehicle 1. In FIG. 2, a plurality of photographing apparatuses 100 are illustrated, but the number of photographing apparatuses 100 may be one.

  The imaging device 100 corresponds to, for example, a monocular camera or a stereo camera. The photographing apparatus 100 is equipped with a steering column or instrument panel in the vehicle interior for photographing a face image obtained by photographing the driver's face from the front or slightly below, or a vehicle for photographing the surroundings of the vehicle 1. Some are installed in place. The output device 200 corresponds to, for example, a monitor or a display provided in the passenger compartment, and displays an image photographed by the photographing device 100, for example.

  In addition, as illustrated in FIG. 2, the image processing apparatus 300 includes a storage unit 310 and a control unit 320.

  As illustrated in FIG. 2, the storage unit 310 includes an image storage unit 311 and a feature amount storage unit 312. The storage unit 310 is a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory.

  The image storage unit 311 stores, for example, a face image obtained by photographing the driver's face from the front or slightly below. The driver's face image is acquired by the photographing apparatus 100 described above. The feature amount storage unit 312 stores, for example, the coordinates on the face image and a predetermined feature amount in association with each other as a result of the feature point detection processing executed by the detection unit 322 described later. FIG. 3 is a diagram illustrating an example of information stored in the feature amount storage unit according to the first embodiment. As shown in FIG. 3, the feature amount storage unit 312 has coordinates (X1, Y1) and feature amounts (V1), coordinates (X2, Y2), feature amounts (V2), coordinates (X3, Y3), and feature amounts ( V3) and the like are stored in association with each other. Note that the coordinates and feature quantities stored in the feature quantity storage unit 312 will be clarified in the description of the detection unit 322 described later.

  Returning to FIG. 2, the control unit 320 includes an acquisition unit 321, a detection unit 322, a determination unit 323, and a notification unit 324. Note that the control unit 320 corresponds to, for example, an electronic circuit or an integrated circuit. Examples of the electronic circuit include a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). Examples of integrated circuits include ASIC (Application Specific Integrated Circuit) and FPGA (Field Programmable Gate Array).

  The acquisition unit 321 acquires face image data from the image storage unit 311 and acquires the luminance value of each pixel included in the face image.

  Based on the luminance value of each pixel included in the face image acquired by the acquisition unit 321, the detection unit 322 executes a feature point detection process that detects a feature point such as a nostril from the face image. . As shown in FIG. 2, the detection unit 322 includes an extraction unit 322A, a calculation unit 322B, a storage unit 322C, and a determination unit 322D.

  Based on the luminance value of each pixel included in the face image acquired by the acquisition unit 321, the extraction unit 322 </ b> A extracts a region where the luminance value of each pixel may increase as it goes from the inside to the outside of the region. Pixels to be identified are extracted. Hereinafter, the extraction unit 322A will be described with reference to FIG. FIG. 4 is a diagram used for explaining the operation of the detection unit according to the first embodiment.

  P1 shown in FIG. 4 represents a target pixel. 4A shown in FIG. 4 represents a partial area of the face image. A region 4B shown in FIG. 4 is hatched to make the image of the processing performed by the calculation unit 322B easy to understand when P1 is the target pixel. P2 and P3 shown in FIG. 4 respectively represent two neighboring pixels located at a position 5 pixels away from each other in the left-right direction around the target pixel P1, that is, in the X-axis direction. P4 and P5 shown in FIG. 4 respectively represent two neighboring pixels located at a position separated by 5 pixels in the vertical direction around the target pixel P1, that is, in the Y-axis direction. C1 shown in FIG. 4 represents an approximate quadratic function calculated from the luminance values of three pixels arranged in the X-axis direction. C2 shown in FIG. 4 represents an approximate quadratic function calculated from the luminance values of three pixels arranged in the Y-axis direction. B1 shown in FIG. 4 represents a line segment indicating the Y coordinate corresponding to the bottom of the quadratic function C1, that is, the position having the minimum value. B2 shown in FIG. 4 represents a line segment indicating the X coordinate corresponding to the bottom of the quadratic function C2, that is, the position having the minimum value. 4C shown in FIG. 4 represents the position of the pixel where the line segment B1 and the line segment B2 intersect. Here, 4C shown in FIG. 4 corresponds to a pixel whose luminance value is estimated to be the smallest among the pixels included in the region 4B based on the approximate expression. Hereinafter, the pixel 4C corresponding to the target pixel P1 is referred to as a luminance minimum point.

  As illustrated in FIG. 4, the extraction unit 322A selects one of the pixels included in the face image 4A as the target pixel P1, and neighboring pixels P2 and P3 that are located at five pixels on both sides in the X-axis direction from the target pixel P1. To get. Subsequently, the extraction unit 322A acquires neighboring pixels P4 and P5 located at positions of 5 pixels on both sides in the Y-axis direction from the target pixel P1. Subsequently, the extraction unit 322A compares the luminance value of the target pixel P1 with the luminance values of the neighboring pixels P2 and P3, and compares the luminance value of the target pixel P1 with the luminance values of the neighboring pixels P4 and P5. . As a result of the comparison, if the luminance value of the target pixel P1 is not greater than the luminance values of the neighboring pixels P2 to P5, the target pixel P1 and the four neighboring pixels are set as processing targets of the calculation unit 322B. If the target pixel P1 has a luminance value larger than any of the four neighboring pixels, the target pixel P1 can be regarded as a position where the luminance value is not darker than the surroundings, such as a nostril. Thus, when it is obvious that the target pixel P1 is outside the region corresponding to the nostril, the calculation amount can be reduced by not making the P1 a processing target of the calculation unit 322B. Note that the neighboring pixels need not be limited to the four pixels above, below, left, and right of the target pixel, and a plurality of pixels at positions surrounding the target pixel can be used.

  Note that the above-described distance (size) from the target pixel to the neighboring pixels is an example. How many pixels away from the target pixel are used as the neighboring pixels, in other words, how much the distance (size) between the left and right or upper and lower neighboring pixels is not limited to 5 pixels. You can set a size like For example, the general value of the width of the nostril area projected in the face image when the image of the driver's face in the driver's seat is photographed by the photographing apparatus 100 is determined in advance. Then, the above-mentioned size is set to be smaller than the width of the region of the nostril in the face image and larger than half of the width. For example, a size corresponding to about 70% of the vertical or horizontal width of the nostril area projected in the face image is appropriate. Of course, the technique disclosed in the present embodiment can detect any area other than the nostrils as long as the luminance value of each pixel increases from the inside to the outside of the area. Therefore, a size corresponding to about 70% of the vertical or horizontal width of the general value of the region to be detected may be used according to the size of the region to be detected on the image.

  The calculation unit 322B calculates a quadratic function that approximates the variation in the luminance value of the pixel of interest pixel P1 and four neighboring pixels acquired by the extraction unit 322A. Hereinafter, the operation of the calculation unit 322B will be described with reference to FIG.

  For example, the calculation unit 322B acquires the luminance value of the target pixel P1 and the luminance values of two neighboring pixels P2 and P3 arranged in the X-axis direction including the target pixel P1. Then, the calculation unit 322B calculates a quadratic function C1 using the luminance value of the target pixel P1, the luminance value of the neighboring pixel P2, and the luminance value of P3. The quadratic function C1 can be regarded as approximating the state of change in the luminance value of each pixel between the pixels P2 and P3 having the same Y coordinate as the pixel P1. Similarly, the calculation unit 322B acquires the brightness values of two neighboring pixels P4 and P5 arranged in the Y-axis direction including the target pixel P1. Then, the calculation unit 322B obtains a quadratic function C2 using the luminance value of the target pixel P1, the luminance value of the neighboring pixel P4, and the luminance value of P5. The quadratic function C2 can be regarded as approximating the state of change in luminance value of each pixel between the pixels P4 and P5 having the same X coordinate as the pixel P1.

  The calculation unit 322B is not limited to approximating the luminance change state using a quadratic function, and any function can be used as long as the luminance change state can be evaluated. Good. For example, a trigonometric function can be used.

  The storage unit 322C stores a predetermined feature amount in association with the coordinates of the position specified by the bases of the two quadratic functions obtained by the calculation unit 322B. Hereinafter, the storage unit 322C will be described with reference to FIG.

  For example, the storage unit 322C obtains the bottom of the quadratic function C1 and the quadratic function C2 obtained by the calculation unit 322B, that is, the X coordinate and the Y coordinate of the position having the minimum value. Subsequently, as illustrated in 4C of FIG. 4, the storage unit 322C determines the position where the line segment B1 obtained by extending the position where the quadratic function C1 takes the minimum value in the Y-axis direction and the position where the quadratic function C2 takes the minimum value as X A position 4C where the line segment B2 extending in the axial direction intersects is detected. That is, the storage unit 322C acquires the X coordinate of the position where the quadratic function C1 takes the minimum value and the Y coordinate of the position where the quadratic function C2 takes the minimum value.

  Subsequently, the storage unit 322C determines the coordinates of the position indicated by 4C in FIG. 4 as the minimum luminance point in the process focusing on the target pixel P1. Subsequently, the storage unit 322C compares the coefficient value of the quadratic term of the quadratic function C1 with the coefficient value of the quadratic term of the quadratic function C2, and acquires the smaller coefficient value as a feature amount. Subsequently, the storage unit 322C stores the coefficient value acquired as the feature amount in the feature amount storage unit 312 in association with the determined coordinates of the minimum luminance point.

  When the extraction unit 322A and the calculation unit 322B complete the process on a certain pixel of interest P1, the position of the pixel of interest P1 is shifted by one in the face image, and the above-described extraction unit 322A and Processing by the calculation unit 322B is performed. That is, the face image is scanned while shifting the target pixel P1. Here, for simplification of processing, it is also conceivable to process by thinning out the face image. In the case of performing processing by thinning out, scanning may be performed by shifting the pixel of interest P1 every predetermined number instead of shifting by one pixel at a time. For simplification of processing, if a range where a face is likely to be reflected can be identified from the face image, it is conceivable to scan the target pixel P1 only for such a range. For example, if the position of the photographing apparatus 100 is fixed with respect to a detection target such as a face or a nostril, an area where the detection target appears on the image can be roughly specified. Further, for example, it is conceivable to specify a skin color region and scan the pixel of interest P1 within the skin color region.

  Here, for example, the luminance minimum point is calculated between the result calculated by the calculation unit 322B for the target pixel P1 processed immediately before and the result calculated by the calculation unit 322B for the target pixel P1 processed next. The coordinates may be the same. That is, the position of the minimum luminance point obtained for each different pixel of interest may overlap. In particular, when the pixel of interest P1 is in an area where the luminance value of each pixel increases from the inside toward the outside in a partial area in the face image, the minimum luminance point may overlap. As described above, when the coordinates of the minimum luminance point are the same, the storage unit 322C correlates with the same coordinates and stores the feature amount stored for a certain target pixel with respect to other target pixels. The calculated feature amount is added, and the feature amount stored in the feature amount storage unit 312 is updated.

  Note that the storage unit 322C uses the smaller coefficient value of the quadratic term coefficient value of the quadratic function as the feature value in order to keep the feature value related to the region different from the region to be finally specified low. It is. For example, in a region such as region R1 or region R2 shown in FIG. 1 corresponding to a nostril in a face image, the curves of the quadratic function approximating the change in luminance in the X-axis direction and the Y-axis direction are both the same. Therefore, it is expected that there is no significant difference in the coefficient values of the quadratic terms of the respective quadratic functions. Here, unlike a region such as the region R1 or the region R2, in a region where the curve of the quadratic function in either the X-axis direction or the Y-axis direction is sharper than the other, the quadratic function of the approximate quadratic function is approximated. The coefficient value of one of the terms becomes larger. As described above, the region where the curve of the quadratic function in either the X-axis direction or the Y-axis direction is sharp does not correspond to the region to be finally detected, for example, the region R1 or the region R2 shown in FIG. It is convenient in processing that the feature amount does not increase in such a region. Therefore, by storing the smaller coefficient value of the quadratic function as a feature amount, the stored feature amount is suppressed as much as possible as a result of the processing for the region not corresponding to the region to be finally detected.

  Here, the feature point detection process executed by the detection unit 322 described above will be supplementarily described with reference to FIGS. 5 to 7 are supplementary diagrams used for explaining the operation of the detection unit according to the first embodiment.

  5A shown in FIG. 5 is a three-dimensional model representing a part of a face image area having a closed area (α) in which the luminance value of each pixel increases from the inside to the outside of the area. The X axis and the Y axis indicate the positions of the face image in the X axis direction and the Y axis direction, and the Z axis direction indicates the luminance value of the pixel at the position specified by the XY coordinates. For example, an area corresponding to a nostril in the face image is in a state like a closed area (α). When the luminance variation of a certain pixel column in the X-axis direction included in the closed region (α) is approximated, for example, a quadratic function C1 shown in FIG. 4 described above is obtained. Similarly, when the luminance variation of a certain pixel column in the Y-axis direction included in the closed region (α) is approximated, for example, a quadratic function C2 shown in FIG. 4 described above is obtained. 5B shown in FIG. 5 is stored in the feature amount storage unit 312 when each pixel in the face image in the range indicated by 5A is scanned as the target pixel P1 by the extraction unit 322A, the calculation unit 322B, and the storage unit 322C. It is a three-dimensional model that represents stored feature values in a three-dimensional manner. The X axis and the Y axis correspond to the X axis and the Y axis in FIG. 5A, and the Z axis direction indicates the feature amount of the pixel at the position specified by the XY coordinates. As shown in 5B, as in the closed region (α) shown in 5A, if the curve has a quadratic function that is similar in both the X-axis direction and the Y-axis direction, the pixel having the minimum luminance point among a plurality of target pixels is displayed. The feature amount with respect to the closed region (α) appears to be largely concentrated at the position.

  Subsequently, 6A shown in FIG. 6 is a three-dimensional model representing a part of a face image area having a groove-like area (β) in which the luminance value of each pixel increases from the inside to the outside of the area. is there. The X axis and the Y axis indicate the positions of the face image in the X axis direction and the Y axis direction, and the Z axis direction indicates the luminance value of the pixel at the position specified by the XY coordinates. For example, a region corresponding to a facial wrinkle or a linear depression in a face image is in a state like a closed region (β). 6B shown in FIG. 6 is stored in the feature amount storage unit 312 when each pixel in the face image in the range shown in 6A is scanned as the target pixel P1 by the extraction unit 322A, the calculation unit 322B, and the storage unit 322C. It is a three-dimensional model that represents stored feature values in a three-dimensional manner. The X axis and the Y axis correspond to the X axis and the Y axis in FIG. 6A, and the Z axis direction indicates the feature amount of the pixel at the position specified by the XY coordinates. As shown in 6B, as in the groove-like region (β) shown in 6A, if the curves of the quadratic function in the X-axis direction and the Y-axis direction are not similar, that is, the quadratic function quadratic function If the coefficient values of the terms are significantly different, the minimum luminance points are dispersed among the plurality of target pixels at the positions of the plurality of pixels, and the feature amount of the groove-like region (β) appears.

  7A shown in FIG. 7 is a face having a local closed region (γ) in which the luminance value of each pixel increases from the inside to the outside of the region, and a region in which the luminance changes stepwise. It is a three-dimensional model representing a part of an image area. For example, when a shadow is generated at a position on the closed region (γ), which is a region corresponding to the nostril, in the face image, the state shown in FIG. 7A is obtained. The X axis and the Y axis indicate the positions of the face image in the X axis direction and the Y axis direction, and the Z axis direction indicates the luminance value of the pixel at the position specified by the XY coordinates. 7B is a feature amount storage unit 312 when each pixel in the face image in the range indicated by 7A is scanned as the attention pixel P1 by the extraction unit 322A, the calculation unit 322B, and the storage unit 322C described above. This is a three-dimensional model that three-dimensionally represents the feature amount stored in. The X axis and the Y axis correspond to the X axis and the Y axis in FIG. 7A, and the Z axis direction indicates the feature amount of the pixel at the position specified by the XY coordinates. As shown in 7B, even when there is a region where the luminance changes stepwise, that is, even when the face is not irradiated with light uniformly and a shadow is generated, the local closure shown in 7A A large amount of feature appears concentrated at the position of a pixel in the region (γ).

  As shown in FIGS. 5 and 7, in the local closed region where the luminance value of each pixel increases from the inside to the outside of the region, the minimum luminance point is obtained while the target pixel exists in the closed region. In many cases, the positions of the centers are concentrated or overlapped. Therefore, the minimum luminance points are concentrated on a specific pixel or its surroundings, and as a result, a large amount of feature appears. The reason for this will be explained. First, the size shown in 4B of FIG. 4 is set to a size that is smaller than the area of the nostril in the face image and always includes the center position of the nostril in the face image. Then, when the target pixel and the neighboring pixel exist in the nostril region, the possibility that the minimum luminance point overlaps can be increased. Therefore, as a result of adding the feature quantity to the coordinates of the specific pixel in the nostril area, for example, as shown in FIGS. 5 and 7, the feature quantity appears concentrating near the specific pixel.

  On the other hand, as shown in FIG. 6, in the groove-shaped region, even when the target pixel and neighboring pixels exist in the region, if the position of the target pixel changes, the bottom of the approximate quadratic function does not overlap, It can be seen that feature quantities appear dispersed at the pixel positions. In the groove-shaped region, pixels having a lower luminance value than the surrounding region continue on the line without being solidified within a certain region. Therefore, the position of the minimum luminance point changes as the position of the pixel of interest changes, and is difficult to overlap. For this reason, as a result of storing the feature amounts in a distributed manner, for example, as shown in FIG. 6, the feature amounts appear at a plurality of pixel positions. In the present invention, utilizing the fact that the luminance value near the center of the nostril region is lower than the peripheral pixels, pixels having a large feature amount are detected as candidates for detecting the nostril. Therefore, it can be considered that specifying the position of a pixel having a large feature amount is synonymous with specifying the position corresponding to the nostril on the face image.

  As described above, the feature point detection process executed by the detection unit 322 described above associates with a specific pixel in a region that is highly likely to be a face part to be detected in the face image, for example, a nostril. The feature quantity to be increased. Therefore, the determination unit 322D described later can determine the position of the nostril from the face image based on the feature amount.

  The determination unit 322D determines the position of the nose of the face image based on the feature amount stored in the feature amount storage unit 312. The feature amount acquired from the feature amount storage unit 312 is compared with a predetermined threshold value, and the coordinates of a pixel whose feature amount is larger than the threshold value are determined as feature point candidates. Then, the determination unit 322D generates a feature amount image in which pixels corresponding to the feature point candidates are drawn in “white” and pixels other than the feature point candidates are drawn in “black”. FIG. 8 is a diagram illustrating an example of the feature amount image according to the first embodiment. 8A shown in FIG. 8 represents a face image, and 8B shown in FIG. 8 represents a feature amount image. The determination unit 322D generates a feature amount image as illustrated in 8B of FIG. 8 for the face image as illustrated in 8A of FIG.

  Subsequently, the determination unit 322D acquires a feature point candidate having the largest feature amount from among a plurality of feature point candidates included in the feature amount image. Here, a single pixel may be treated as a feature point candidate, grouping is performed when pixels having feature amounts are in contact with each other, and a group of pixels after grouping is treated as a feature point candidate. May be. In the face image, there is often no area other than the nostril that has a lower brightness than the surrounding area, so the feature point candidate with the largest feature is one of the nostrils. It can be regarded as corresponding to one position. Subsequently, when the feature point candidate having the largest feature amount is set to one of the nostrils, the determination unit 322D includes a plurality of feature point candidates that can be the other nostril in the feature amount image. The feature point candidates are acquired from the feature point candidates, and combinations of feature point candidates are determined. For example, the determination unit 322D determines a combination of feature point candidates by using an average width of nostrils, an interval between holes, a luminance pattern of nostrils, and the like that are stored in advance. . Then, the determination unit 322D determines each feature point candidate included in the determined combination of feature point candidates as a feature point, and determines the position of the feature point as the position of the nostril.

  Alternatively, the determination unit 322D may determine the position of the nose as follows. For example, the determination unit 322D acquires coordinates associated with the largest feature amount from the feature amounts stored in the feature amount storage unit 312. Subsequently, the determining unit 322D stores, in the feature amount storage unit 312, the feature amount associated with the coordinate that can be the position of the other nostril when the acquired coordinate is the position of one nostril. One or a plurality of feature amounts are acquired, and the largest feature amount among the acquired feature amounts is specified. In this way, the determination unit 322D determines the combination of feature points that become the nostrils, and determines the coordinates of each feature point as the position of the nostril.

  The determination unit 323 detects the orientation of the face based on the position of the nostril determined by the determination unit 322D, and determines whether or not the orientation of the face during driving is appropriate. For example, the determination unit 323 uses the average eye position with respect to the position of the nostril determined by the determination unit 322D and the position of the nostril stored in advance to approximate the position of the eye in the face image. To get. Then, in the vicinity of the approximate eye position, the eye position is specified by eye shape pattern detection or edge detection stored in advance. Here, a threshold value for edge detection for detecting the eyes may be set based on the luminance average in the vicinity of the position of the approximate eye specified based on the position of the nostril.

  Subsequently, the determination unit 323 captures the position of the nostril and the eye position in the face image continuously captured by the image capturing apparatus 100, and acquires the movement amount of the nostril position and the eye position. To do. Subsequently, the face shape is restored from the movement amount of the nostril position and the eye position, and the face orientation is detected from the nostril position, the eye position, and the face shape. Subsequently, the determination unit 323 starts tracking the face direction, and executes a process for determining whether or not the face direction during driving is appropriate. For example, it may be determined that the face is not appropriate when the face deviates from the front by a predetermined angle or more. Here, the determination unit 323 captures only the position of the nostril or only the position of the nostril and the position of the eyes over time, and when the position changes significantly or deviates from a predetermined range on the image. In such a case, it can also be determined that a look has been made, that is, the face orientation is not appropriate. As a result of the determination, if it is appropriate, the determination unit 323 sends a notification stop instruction to the notification unit 324. As a result, when the state where the notification is made by the notification unit 324 continues, the notification is stopped. If the notification unit 324 is not notified, the state where the notification is not performed is continued. On the other hand, if the determination result is inappropriate, the determination unit 323 sends a warning notification instruction to the notification unit 324 that the face orientation is inappropriate.

  The notification unit 324 performs warning notification and warning stop according to the instruction from the determination unit 323 described above. When there is a warning notification instruction from the determination unit 323, the notification unit 324 repeatedly reports a warning that warns about the observation at regular intervals. In addition, when there is a warning stop instruction from the determination unit 323, the notification unit 324 stops the warning notification.

[Processing by Image Processing Apparatus (Example 1)]
Subsequently, the flow of processing by the image processing apparatus according to the first embodiment will be described with reference to FIGS. 9 and 10. FIG. 9 is a diagram illustrating an overall flow of processing by the image processing apparatus according to the first embodiment. FIG. 10 is a diagram illustrating a flow of feature point detection processing by the image processing apparatus according to the first embodiment.

  First, the overall flow of processing by the image processing apparatus 300 will be described with reference to FIG. Note that the image processing device 300 starts the processing of FIG. 9 in accordance with, for example, the operation of the ignition device of the vehicle 1. If it is determined by the control unit 320 or an instruction signal from another device that the image processing apparatus 300 starts processing (YES in step S101), the detection unit 322 performs feature point detection processing. Is executed (step S102). Note that when the ignition device of the vehicle 1 is not operating, the image processing apparatus 300 repeats the determination in step S101 with the determination result in step S101 as No. Alternatively, when the ignition device is not operating, the image processing device 3 may not be activated. Further, not only the ignition device is operating, but also when the control unit 320 detects that the traveling speed of the vehicle 1 has reached a predetermined value or more, it may be determined as YES in S101.

  Subsequently, the determination unit 322D generates a feature amount image based on the detection result of the feature point detection process in step S102 (step S103), and based on the generated feature amount image, determines the position of the nostril of the face image. It detects (step S104). The determination unit 322D may specify the position of the feature point corresponding to the position of the nostril in step S104 without performing step S103 as described above. The determination unit 323 detects the driver's face orientation from the position of the nostril (step S105), starts tracking the face orientation (step S106), and determines whether the driver's face orientation is inappropriate. The determination process of NO is started (step S107). If the result of determination is that the driver's face orientation is inappropriate (step S107, YES), the determination unit 323 sends a warning notification instruction to the notification unit 324 (step S108). Subsequently, the control unit 320 determines whether or not the next processing frame has been input (step S109). As a result of the determination, when the next processing frame is input (step S109, YES), the control unit 320 performs a process end determination (step S110). As a result of the determination, if the process is to be ended (step S110, YES), the process is ended. On the other hand, when the process is not terminated (step S110, NO), the process returns to the above-described step S102 to execute the feature point extraction process.

  Here, the description returns to step S107. If the result of determination is that the driver's face orientation is not inappropriate (step S107, NO), the determination unit 323 determines whether or not a warning notification instruction has been sent (step S111). As a result of the determination, if the transmission has been completed (step S111, YES), the determination unit 323 transmits a warning notification stop instruction to the notification unit 324 (step S112). Subsequently, the control unit 320 returns to step S109 described above and waits for input of the next image frame. Here, the description returns to step S109. If the result of determination is that the next processing frame has not been input (step S109, NO), the control unit 320 waits for input of the next processing frame. Control unit 320 repeats the processing from step S101 to step S112 described above for the input processing frame until it is determined in step S110 that the processing is to be terminated.

  Next, a flow of feature point detection processing by the image processing apparatus 300 will be described with reference to FIG. As illustrated in FIG. 10, the extraction unit 322A acquires a face image from the image storage unit 311 and selects one pixel included in the acquired face image as a target pixel (step S201). Subsequently, the extraction unit 322A determines whether the luminance value of the target pixel is equal to or lower than the luminance value of the neighboring pixels (Step S202). As a result of the determination, when the luminance value of the target pixel is equal to or lower than the luminance value of the neighboring pixel (step S202, Yes), the extraction unit 322A extracts the luminance value and the coordinate position of the target pixel and the neighboring pixel from the face image. (Step S203).

  Subsequently, the calculation unit 322B obtains an approximate quadratic function in the X-axis direction based on the target pixel and neighboring pixels extracted by the extraction unit 322A (step S204). Subsequently, the calculation unit 322B obtains an approximate quadratic function in the Y-axis direction based on the target pixel and neighboring pixels extracted by the extraction unit 322A (step S205). Note that either step S204 or S205 may be executed first.

  Subsequently, the storage unit 322C acquires the coordinates specified by the position at which each quadratic function takes the minimum value (step S206), compares the quadratic coefficient values of the quadratic function, and determines the smaller coefficient value. Obtained as a feature amount (step S207). Subsequently, the storage unit 322C stores the coefficient value acquired as the feature amount in step S207 in association with the coordinates acquired in step S206 (step S208). Subsequently, the storage unit 322C determines whether or not all the pixels to be processed included in the face image have been selected as the target pixel (step S209). As a result of the determination, if all the pixels have not been selected (No at Step S209), the storage unit 322C updates the position of the pixel acquired as the target pixel and returns the process to Step S201 described above. On the other hand, as a result of the determination, if all the pixels have been selected (step S209, Yes), the storage unit 322C ends the feature point detection process. In step S208 described above, the storage unit 322C adds the current feature amount to the stored feature amount when the feature amount has been stored in association with the coordinate of the same minimum luminance point. Update.

  Here, the description returns to step S202. As a result of the determination, if the luminance value of the target pixel is larger than the luminance value of the neighboring pixel, the extraction unit 322A sets the determination result in step S202 to No, updates the position of the pixel acquired as the target pixel, The process returns to S201.

  Next, processing results of the image processing apparatus according to the first embodiment will be described with reference to FIGS. FIG. 11 is a diagram for explaining the processing result of the image processing apparatus according to the first embodiment. FIG. 12 is a diagram for explaining a processing result by the image processing apparatus for an image with reduced contrast according to the first embodiment. FIG. 13 is a diagram for explaining a processing result by the image processing apparatus for an image in which the intensity of irradiation light according to the first embodiment is changed. 11 to 13 show the relationship among the face image, the feature amount image, and the feature amount as a processing result by the image processing apparatus 300. FIG.

  11A shown in FIG. 11 represents a face image. 11B shown in FIG. 11 represents a feature amount image corresponding to 11A. 11C shown in FIG. 11 is a diagram in which the face image 11A, the feature amount image 11B, and the feature amount storage result are superimposed. 11D shown in FIG. 11 represents the storage result of the feature amount projected on the Y-axis, and the value in the X-axis direction is shown larger as the feature amount is larger. For convenience of explanation, since the feature amount is projected on the Y-axis, a state is shown in which feature amounts for a plurality of pixels having the same Y coordinate are added. As illustrated in 11C, when the feature amount image 11B is superimposed on the face image 11A, the position of the nostril projected on the face image 11A matches the position of the feature point drawn in the feature amount image 11B. To do. Also, as shown in 11C, when the feature value storage result 11D is superimposed on the face image 11A, a peak of feature points appears at a position corresponding to the position of the nostril. In 11B, for example, in addition to the nostrils, shadows of the nose wings, shadows of the eyes, and shadows of the eyebrows also appear. However, as shown in 11D, the features other than the nostrils can be distinguished from the features corresponding to the nostrils because the feature amount is not large.

  12A shown in FIG. 12 represents, for example, a face image with a lower contrast than 11A shown in FIG. For example, it corresponds to a case where a face image is taken in a dim environment. 12B shown in FIG. 12 represents a feature amount image corresponding to 12A. 12C shown in FIG. 12 represents a diagram in which a face image, a feature amount image, and a feature amount storage result are superimposed. 12D shown in FIG. 12 represents the storage result of the feature amount projected on the Y-axis, and the value in the X-axis direction is shown larger as the feature amount is larger. For convenience of explanation, since the feature amount is projected on the Y-axis, a state is shown in which feature amounts for a plurality of pixels having the same Y coordinate are added. 12C, when the feature amount image 12B is superimposed on the face image 12A, the position of the nostril projected on the face image 12A matches the position of the feature point drawn in the feature amount image 12B. To do. Further, as shown in 12C, when the feature amount storage result 12D is superimposed on the face image 12A, a peak of the feature point appears at a position corresponding to the position of the nostril. Even when the overall contrast of the face image is low, as shown in 12D, the feature corresponding to the nostril is associated with a large feature amount in comparison with the feature corresponding to the nostril. It can be distinguished from other features.

  13A shown in FIG. 13 represents, for example, a face image in which the intensity of irradiation light has changed from 11A shown in FIG. For example, it corresponds to a case where a picture is taken with a shadow on a part of the face. 13B shown in FIG. 13 represents a feature amount image corresponding to 13A. 13C shown in FIG. 13 represents a diagram in which a face image, a feature amount image, and a feature amount storage result are superimposed. 13D shown in FIG. 13 represents the storage result of the feature amount projected on the Y-axis, and the value in the X-axis direction is shown larger as the feature amount is larger. For convenience of explanation, since the feature amount is projected on the Y-axis, a state is shown in which feature amounts for a plurality of pixels having the same Y coordinate are added. As illustrated in 13C, when the feature amount image 13B is superimposed on the face image 13A, the position of the nostril projected on the face image 13A matches the position of the feature point drawn in the feature amount image 13B. To do. Further, as shown in 13C, when the storage result 13D of the feature amount is superimposed on the face image 13A, a peak of the feature point appears at a position corresponding to the position of the nostril. Even when a part of the face image has a region where the luminance is partially low due to the influence of a shadow or the like, the feature corresponding to the nostril is larger than the other feature as shown in 13D. A feature corresponding to a feature hole and a feature corresponding to a nostril can be distinguished from other features.

[Effects of Example 1]
As described above, in the image processing apparatus 300 according to the first embodiment, the luminance value of each pixel can be increased from the inside to the outside of the region based on the luminance value of each pixel included in the face image. A pixel of interest and its neighboring pixels that are located in a region having high characteristics are acquired. Subsequently, the image processing apparatus 300 stores a predetermined feature amount in association with the coordinates of the pixel having the lowest luminance within the range specified by the neighboring pixels. Then, the image processing apparatus 300 detects the coordinate associated with the highest feature amount as, for example, a pixel included in a nostril region projected in the face image. Therefore, according to the first embodiment, since the position is detected based on the state of luminance change, not the edge included in the image, the influence of the external light and the characteristics of the imaging device when detecting the detection target from the image It is possible to prevent a decrease in detection accuracy due to. In other words, as shown in FIGS. 11 to 13 described above, it is possible to realize detection of face parts that is robust against changes in the light applied to the face, a reduction in image contrast, image blur, and the like.

  In the first embodiment, the approximation function is calculated only for a pixel whose luminance value is not larger than that of neighboring pixels in order to increase the speed. For this reason, according to the first embodiment, it is possible to efficiently perform processing from an area in which the luminance value of each pixel may increase as it goes from the inside to the outside of the area.

  In the first embodiment, the size between the target pixel and the neighboring pixels is set to be smaller than the width of the nostril in the face image and larger than half of the width. If the target pixel and neighboring pixels are included in the nostril area projected in the face image, the minimum luminance point is in the vicinity of the center position of the nostril projected in the face image and the center position. Is considered to be easily determined. In this way, in the first embodiment, when the target pixel and neighboring pixels are included in the region to be detected, the possibility that the minimum luminance point overlaps is increased. For this reason, according to Example 1, the feature-value of the coordinate corresponding to the specific pixel in the nostril area | region can be enlarged.

  In the first embodiment, a quadratic function that approximates the luminance variation between the neighboring pixel and the target pixel is obtained in the X-axis direction and the Y-axis direction between the neighboring pixels, and each quadratic function has a minimum value. Based on the above, the pixel whose luminance value is estimated to be the smallest is obtained. Here, if only the target pixel having a lower luminance value than the neighboring pixel is set as the approximation function calculation target, the pixel estimated to have the smallest luminance value in the range of the X coordinate and the Y coordinate sandwiched between the neighboring pixels can be obtained. It will be. For this reason, according to the first embodiment, the minimum luminance point within the range of the X coordinate and the Y coordinate sandwiched between neighboring pixels can be easily obtained. Further, it is possible to obtain the minimum luminance point in the region without specifying the range of the region in which the luminance value of the pixel increases from the inside to the outside of the region, that is, without performing processing such as edge extraction. it can.

  In the first embodiment, the smaller coefficient value among the coefficient values of the quadratic term of the quadratic function is stored as a feature amount in association with the coordinates of the pixel corresponding to the minimum luminance point. For this reason, according to the first embodiment, for example, in the groove-shaped region in the face image shown in FIG. Note that the coefficient value is not limited to being stored as a feature amount, and a predetermined frequency may be stored for each minimum luminance point.

  Further, in the first embodiment, the processing based on the determined threshold is not performed as in the case of edge extraction, so that the feature point of the face part to be detected can be detected even when the image contrast is low, such as a nostril. There is no fear of being buried in the image, and the detection accuracy of the face part can be increased.

  In the first embodiment, the pixel position in the region where the luminance value of each pixel increases from the inside to the outside of the region is detected from the image. The case where a nostril can be detected is described. However, the present invention is not limited to this, and it is also possible to detect a facial part having a luminance curved surface such as a nostril, for example, an ear hole. Naturally, even if it is not a face part, it can be detected as long as it is a region exhibiting the same luminance curved surface on the image.

  In the first embodiment, for example, based on the position where the quadratic function C1 shown in FIG. 4 takes the minimum value and the position where the quadratic function C2 takes the minimum value, the position of the pixel having the minimum luminance value is determined. Although the estimation has been described, the present invention is not limited to this. For example, for the pixel of interest P1 shown in FIG. 4, the pixel having the minimum luminance value in the pixel column in the range of the neighboring pixels P2 to P3 is acquired, and the X coordinate of this pixel is acquired. Similarly, with respect to the target pixel P1, a pixel having the minimum luminance value in the pixel row in the range from the neighboring pixels P4 to P5 is acquired, and the Y coordinate of this pixel is acquired. Then, the pixel specified by the X coordinate and the Y coordinate may be a pixel whose luminance value may be minimized within the area specified by the four neighboring pixels.

  In the first embodiment, the method of detecting the pixel position in the region where the luminance value of each pixel increases from the inside to the outside of the region, in other words, in the region where the luminance curved surface is gently depressed. Of course, the image is not limited to the detection of a nostril in a face image, but is an image with a subject other than the face as long as the luminance value of each pixel increases from the inside to the outside of the region. Can also be applied. Further, in the same manner as in the first embodiment described above, for example, the pixel position in a region where the luminance value of each pixel gradually decreases as it goes from the inside to the outside of the region, in other words, in the region that rises on the luminance curved surface. Can also be detected. For example, it can be used for detecting the position of light of an LED (Light Emitting Diode) projected in a camera image. If the position of the LED light in each camera image photographed by each in-vehicle camera can be detected, for example, the in-vehicle camera can be calibrated using this position.

  In the first embodiment, the pixel position in the recessed area on the luminance curved surface may be detected, and the pixel position in the region raised on the luminance curved surface may be detected together. For example, the head of the nose projected in the face image is likely to be a region where the luminance gradually decreases toward the outside centering on a certain pixel. Therefore, for example, if a plurality of combinations that are candidates for the position of the nostril are detected from the feature points in the feature amount image, it is assumed that one of the areas where the brightness has risen is the head of the nose, In some cases, the position of the nostril can be identified.

  Hereinafter, other embodiments of the image processing apparatus and the image processing program disclosed in the present application will be described.

(1) Apparatus Configuration, etc. For example, the functional block configuration of the image processing apparatus 300 shown in FIG. 2 is conceptual and does not necessarily need to be physically configured as illustrated. For example, the extraction unit 322A, the calculation unit 322B, and the storage unit 322C of the detection unit 322 illustrated in FIG. 2 may be integrated functionally or physically. As described above, all or a part of the functional blocks of the image processing apparatus 300 can be configured to be functionally or physically distributed / integrated in arbitrary units according to various loads or usage conditions.

(2) Image processing program Various processes executed by the image processing apparatus 300 described in the first embodiment are, for example, a microcomputer mounted on an ECU (Electronic Control Unit) mounted on the vehicle 1 This can also be realized by executing a predetermined program on the electronic device. In the following, an example of an electronic apparatus that executes an image processing program that realizes the same function as the process executed by the image processing apparatus 300 described in the first embodiment will be described with reference to FIG. FIG. 14 is a diagram illustrating an example of an electronic device that executes an image processing program.

  As illustrated in FIG. 14, the electronic apparatus 400 that implements various processes executed by the image processing apparatus 300 includes a CPU (Central Processing Unit) 410 that executes various arithmetic processes. As shown in FIG. 14, the electronic apparatus 400 includes a camera interface 420 for acquiring camera images and a display interface 430 for exchanging various data with the display. As illustrated in FIG. 14, the electronic device 400 includes a graphic engine 440 that functions as a hardware accelerator.

  As shown in FIG. 14, the electronic apparatus 400 includes a hard disk device 450 that stores programs and data for realizing various processes by the CPU 410, and a memory such as a RAM (Random Access Memory) that temporarily stores various information. 460. Each device 410 to 460 is connected to a bus 470.

  Instead of the CPU 410, for example, an electronic circuit such as an MPU (Micro Processing Unit) or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array) can be used. Further, instead of the memory 460, a semiconductor memory device such as a flash memory can be used.

  The hard disk device 450 stores an image processing program 451 and image processing data 452 that exhibit functions similar to those of the image processing device 300. Note that the image processing program 451 may be appropriately distributed and stored in a storage unit of another computer that is communicably connected via a network.

  Then, the CPU 410 reads the image processing program 451 from the hard disk device 450 and develops it in the RAM 460, whereby the image processing program 451 functions as an image processing process 461 as shown in FIG. The image processing process 461 expands various data such as the image processing data 452 read from the hard disk device 450 in an area allocated to itself on the memory 460 as appropriate, and executes various processes based on the expanded data. .

  Note that the image processing process 461 includes, for example, processing executed by the control unit 320 of the image processing apparatus 300 shown in FIG. 2, for example, processing shown in FIG.

  Note that the image processing program 451 is not necessarily stored in the hard disk device 450 from the beginning. For example, each program is stored in a “portable physical medium” such as a flexible disk (FD), a CD-ROM, a DVD disk, a magneto-optical disk, and an IC card that can connect a corresponding drive to the ECU on which the electronic device 400 is mounted. Let me. Then, the electronic apparatus 400 may read and execute each program from these.

  Furthermore, each program is stored in “another computer (or server)” connected to the ECU on which the electronic device 400 is mounted via a public line, the Internet, a LAN, a WAN, or the like. Then, the electronic apparatus 400 may read and execute each program from these.

  The following supplementary notes are further disclosed with respect to the embodiments including the above examples.

(Supplementary Note 1) Luminance values of a first pixel included in an image and a plurality of pixels located at a distance of one pixel or more from the first pixel, and the first pixel in the image An acquisition unit for acquiring each while scanning;
An image processing apparatus comprising: a detection unit configured to detect a feature point in the image based on luminance values of the first pixel and the plurality of pixels acquired while scanning.

(Appendix 2) The detection unit
An extraction unit that extracts a plurality of pixels that are the first pixels and whose luminance values are not larger than the plurality of pixels;
For each of the plurality of first pixels extracted by the extraction unit, the first pixel and the plurality of pixels are located at the pixel positions specified based on the luminance values of the first pixel and the plurality of pixels. A storage unit that associates a value calculated based on a luminance value with a pixel and stores the value in a storage unit;
The image processing apparatus according to claim 1, further comprising: a determination unit that determines the feature point based on a value stored by the storage unit.

(Supplementary Note 3) Each of the plurality of pixels is a pixel located in a first direction existing at a position sandwiching the first pixel, among pixels located at an equal number of pixels away from the first pixel. And two pixels located in a position sandwiching the first pixel and located in a second direction orthogonal to the first direction,
The detection unit further includes a first approximation function obtained from luminance values of two pixels located in the first direction and the first pixel, and two pixels located in the second direction. And a second approximation function calculated from the luminance values of the first pixel and the second approximation function, respectively,
The storage unit includes a position between the position of the two pixels positioned in the first direction and the position of the two pixels positioned in the second direction. And the second approximation function, the coefficient value of the term of the maximum degree of the first approximation function and the second approximation function in association with the position of the pixel specified based on the position where the second approximation function takes an extreme value. The smaller coefficient value of the coefficient values of the maximum degree term is stored in the storage unit,
The image processing according to appendix 2, wherein the determination unit determines the feature point based on the position of the pixel associated with the largest value among the coefficient values stored in the storage unit. apparatus.

(Appendix 4) The image is an image including a face,
A determination unit that detects the orientation of the face based on the feature points and determines whether the detected orientation of the face is within an appropriate range;
The image processing apparatus according to any one of supplementary notes 1 to 3, further comprising: a notification unit that notifies a warning according to the determination result.

(Appendix 5)
Luminance values of a first pixel included in an image and a plurality of pixels located at a distance of one pixel or more from the first pixel while scanning the first pixel in the image, respectively Acquired,
An image processing program for executing a process of detecting a feature point in the image based on luminance values of the first pixel and the plurality of pixels acquired while scanning.

(Appendix 6)
Extracting a plurality of pixels that are the first pixels and whose luminance values are not larger than those of the plurality of pixels;
For each of the plurality of extracted first pixels, the first pixel and the plurality of pixels are located at pixel positions specified based on the luminance values of the first pixel and the plurality of pixels. A value calculated based on the luminance value is associated and stored in the storage unit,
The image processing program according to appendix 5, further comprising executing a process of determining the feature point based on a value stored by the storage unit.

(Supplementary Note 7) Each of the plurality of pixels is two pixels located in a first direction existing at a position sandwiching the first pixel among pixels located at an equal number of pixels away from the first pixel. And two pixels located in a position sandwiching the first pixel and located in a second direction orthogonal to the first direction,
In the computer,
A first approximation function obtained from luminance values of two pixels located in the first direction and the first pixel; two pixels located in the second direction; and the first pixel. And further executing a process of calculating each of the second approximate functions obtained from the luminance values of
The position where the first approximate function takes an extreme value between the positions of two pixels located on the first direction and the position between the positions of the two pixels located on the second direction The coefficient value of the term of the maximum order of the first approximation function and the term of the maximum order of the second approximation function are associated with the position of the pixel specified based on the position where the approximation function takes an extreme value. The smaller coefficient value of the coefficient values of is stored in the storage unit,
The image processing program according to appendix 6, wherein a process for determining the feature point is executed based on the position of the pixel associated with the largest value among the coefficient values stored in the storage unit. .

DESCRIPTION OF SYMBOLS 1 Vehicle 100 Imaging device 200 Output device 300 Image processing device 310 Storage unit 311 Image storage unit 312 Feature amount storage unit 320 Control unit 321 Acquisition unit 322 Detection unit 322A Extraction unit 322B Calculation unit 322C Storage unit 322D Determination unit 323 Determination unit 324 Notification Part 400 electronic device 410 CPU
420 Camera Interface 430 Display Interface 440 Graphic Engine 450 Hard Disk Device 451 Image Processing Program 452 Image Processing Data 460 RAM
461 Image processing process

Claims (4)

The luminance value of the first pixel included in the image, and the luminance values of a plurality of pixels located at least one pixel apart from the first pixel across the first pixel in the first direction And luminance values of a plurality of pixels located at least one pixel apart from the first pixel across the first pixel in a second direction orthogonal to the first direction, An acquisition unit that acquires each pixel while scanning one pixel in the image;
An extraction unit that extracts a plurality of pixels in the first direction and a plurality of first pixels having a luminance value smaller than the plurality of pixels in the second direction while scanning the image;
A position where a first approximation function calculated by the first pixel extracted by the extraction unit and a plurality of pixels in the first direction takes a minimum value among the plurality of pixels; The second approximate function calculated by the pixel and the plurality of pixels in the second direction is associated with the position of the minimum luminance point specified based on the position where the minimum value is taken between the plurality of pixels. A storage unit that stores the coefficient value of the first approximate function or the coefficient value of the second approximate function in the storage unit as a feature amount;
The position of the minimum luminance point of the other first pixel when the position of the minimum luminance point of the other first pixel extracted by the extraction unit is the same as the position of the luminance point of the first pixel. An update unit that updates the feature value by adding the coefficient value of the first approximate function or the coefficient value of the second approximate function corresponding to the feature value stored in the storage unit;
And a specifying unit that specifies a feature point or a combination of feature points to be a facial part based on a feature amount corresponding to each luminance minimum point stored in the storage unit. apparatus. When the position of the luminance minimum point of the other first pixel extracted by the extracting unit is the same as the position of the luminance point of the first pixel, the update unit The smaller coefficient value of the coefficient value of the maximum order term of the first approximate function or the coefficient value of the maximum order term of the second approximate function corresponding to the position of the minimum luminance point is stored in the storage unit. To the feature amount
The image according to claim 1, wherein the specifying unit determines the feature point based on a position of the pixel associated with the largest value among the feature amounts stored in the storage unit. Processing equipment. The image is an image including a face,
A determination unit that detects the orientation of the face based on the feature points and determines whether the detected orientation of the face is within an appropriate range;
The image processing apparatus according to claim 1, further comprising: a notification unit that notifies a warning according to the determination result. On the computer,
The luminance value of the first pixel included in the image, and the luminance values of a plurality of pixels located at least one pixel apart from the first pixel across the first pixel in the first direction And luminance values of a plurality of pixels located at least one pixel apart from the first pixel across the first pixel in a second direction orthogonal to the first direction, Each pixel is acquired while scanning in the image,
Extracting a plurality of pixels in the first direction and a plurality of first pixels having a luminance value smaller than the plurality of pixels in the second direction while scanning the inside of the image,
First approximation function and position of taking a minimum value among the plurality of pixels calculated by extraction with the first pixel issued and a plurality of pixels on the first direction, and said first pixel The second approximation function calculated by the plurality of pixels in the second direction is associated with the position of the minimum luminance point specified based on the position where the minimum value is taken between the plurality of pixels, The coefficient value of the first approximate function or the coefficient value of the second approximate function is stored as a feature quantity in the storage unit,
Position of the extracted issued the other brightness minimum point of the first pixel, if the common with the position of the luminance point of the first pixel, corresponding to the positions of the luminance minimum point of the other of the first pixel Updating the feature value by adding the coefficient value of the first approximate function or the coefficient value of the second approximate function to the feature value stored in the storage unit;
An image processing program for executing a process for specifying a feature point or a combination of feature points to be a facial part based on a feature amount corresponding to each minimum luminance point stored in the storage unit .

Images