JP2020140478A - Abnormality detection apparatus and abnormality detection method - Google Patents

Abstract

To provide an abnormality detection apparatus capable of stably detecting an abnormality of a camera by removing feature points that are difficult to extract stably in time.SOLUTION: An abnormality detection apparatus 100 comprises: an acquisition unit 110 for acquiring an image from a camera mounted on a moving object; a candidate extraction unit 120 for extracting feature point candidates from the image; a discrimination pixel identification unit 130 for identifying a first discriminant pixel to an N-th discriminant pixel (N is a natural number of 2 or more) that are in the predetermined first to N-th positional relationship with the feature point candidate; a luminance difference calculation unit 140 for calculating the luminance difference between the feature point candidate and a k-th (k is a natural number 1 or more and N or less) discrimination pixel; a comparison unit 150 for comparing the k-th luminance difference with a k-th threshold set according to the k-th positional relationship; a feature point extraction unit 160 for extracting the feature point from the feature point candidates based on the results of comparison by the comparison unit; and an abnormality detection unit 170 for detecting whether or not the camera has the abnormality based on the temporal position change of the feature point and the moving amount of the moving object.SELECTED DRAWING: Figure 1

Description

The present invention relates to an abnormality detection device and an abnormality detection method.

Conventionally, vehicles such as automobiles have sometimes been equipped with cameras for use in parking assistance and the like. With such a camera, image recognition cannot be performed normally if an abnormality in the mounting state, particularly a deviation in the mounting direction or mounting position occurs. Therefore, for example, as in Patent Document 1, a vehicle imaging device that extracts a feature point from a photographed image of the outside world of the vehicle, calculates an optical flow from the time change of the feature point, and detects an abnormality in the mounting state is known. There is.

Japanese Unexamined Patent Publication No. 2011-055342

By the way, the feature points that can be extracted from an image include those that can be continuously extracted stably in time and those that are difficult to be extracted in a stable manner in time. The former is, for example, a feature point obtained from the corner of the white line marking. The latter is obtained, for example, from the unevenness of the road surface. When the latter feature points are used, the vehicle imaging device cannot obtain the continuity of the feature points between frames, cannot calculate a stable optical flow, and can stably detect anomalies. It will be difficult.

An object of the present invention is to provide an anomaly detection device capable of stably detecting anomalies, such as the latter.

The abnormality detection device according to the present invention has an acquisition unit that acquires an image from a camera mounted on a moving body, a candidate extraction unit that extracts feature point candidates from the image acquired by the acquisition unit, and a candidate extraction unit. The discriminant pixel identification unit that identifies the first to Nth discriminant pixels having a predetermined positional relationship between the feature point candidate and the predetermined first to Nth (N is a natural number of 2 or more), and the extracted feature point candidate. A brightness difference calculation unit that calculates the brightness difference from the k-th (natural number k is 1 or more and N or less) discrimination pixel specified by the discrimination pixel identification unit, and the k-th brightness difference according to the k-th positional relationship. Based on the result of the comparison unit comparing with the set kth threshold value and the comparison unit with the first to Nth brightness differences with the first to Nth threshold values, the feature points are selected from the feature point candidates. An abnormality detection unit that detects the presence or absence of an abnormality in the camera based on the feature point extraction unit to be extracted, the time change of the position of the feature point extracted by the feature point extraction unit, and the movement amount of the moving body. To be equipped.

The present invention can stably detect anomalies.

It is a functional block diagram which shows the structure of the abnormality detection apparatus by embodiment of this invention. It is a figure for demonstrating the method of extracting a feature point candidate. It is a figure for demonstrating the relationship between a feature point candidate and a discrimination pixel. It is a figure which shows the specific example of the relationship between a positional relationship and a threshold value. It is a figure which shows an example of the luminance distribution in the vicinity of a feature point candidate. It is a figure for demonstrating the method of finding an optical flow. It is a figure for demonstrating the coordinate conversion process. It is a flowchart which shows an example of the detection process of a camera displacement. It is a flowchart which shows an example of the feature point extraction process. It is a figure which shows an example of the case where the positional relationship is set parallel to the recording direction of a pixel. It is a figure which shows an example of the correction of the positional relationship of the discriminant pixel for the feature point candidate near the center of a frame image, and the discriminant pixel for a feature point candidate near the outer edge of a frame image.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The same or corresponding parts in the drawings are designated by the same reference numerals and the description thereof will not be repeated. In the following, a vehicle will be described as an example of a moving body, but the moving body to which the present invention is applied is not limited to the vehicle. The present invention may be applied to a moving body to which a camera is attached, such as a robot. Vehicles include a wide range of vehicles with wheels, such as automobiles, trains, and automatic guided vehicles.

Further, in the following description, for convenience, the direction related to the moving body is defined as follows based on the direction of gravity, the direction of travel, and the left and right. The direction in which gravity acts is "downward", and the opposite direction is "upward". The traveling direction of the moving body is defined as "forward direction", and the opposite direction is defined as "rearward direction". The direction in which the forward direction is rotated 90 degrees clockwise around the direction in which gravity acts is defined as the "right direction", and the opposite direction is defined as the "left direction". That is, the directions used in the following description correspond to the generally used front-back, left-right, up-down directions when the person sitting in the driver's seat is facing the traveling direction through the windshield.

<1. Mobile control system ＞
FIG. 1 is a functional block diagram showing a configuration of a mobile body control system according to an embodiment of the present invention. The moving body control system SYS1 includes an abnormality detection system SYS2, an automatic operation control device 500, and a display device 600. The moving body control system SYS1 controls the moving body by using the image taken by the photographing unit 200 included in the abnormality detection system SYS2. The photographing unit 200 photographs an image around the moving body for use in the automatic driving control device 500 and the like.

The abnormality detection system SYS2 detects the presence or absence of an abnormality in the camera mounted on the moving body. Details will be described later. The automatic operation control device 500 controls the automatic operation of the moving body. The automatic operation control device 500 is mounted on each moving body. Specifically, the automatic operation control device 500 includes information from the photographing unit 200 and the sensor unit 300 mounted on the moving body, and a driving unit of the moving body (not shown) based on a preset purpose of automatic control. It is an ECU (Electronic Control Unit) that controls the braking unit and the steering unit.

The purpose of the preset automatic control is, for example, automatic parking or following the vehicle in front. The drive unit includes an engine and a motor. The braking unit is equipped with a brake. The steering unit includes a steering wheel (handle).

When the control by the automatic driving control device 500 is started, the driving unit, the braking unit, and the steering unit are automatically controlled regardless of the driver's operation. Taking automatic parking as an example, the automatic driving control device 500 recognizes an empty space based on an image acquired from the photographing unit 200, and sets the drive unit, the braking unit, and the steering unit so that the moving body moves to the empty space. Control.

The control operation by the automatic operation control device 500 as described above is configured to be able to be switched on / off according to the abnormality detection status in the abnormality detection system SYS2. In other words, when the abnormality detection system SYS2 detects that there is an abnormality in the photographing unit 200, the control operation of the automatic operation control device 500 can be turned off. As a result, control based on information from the photographing unit 200 in an abnormal state can be suppressed, and automatic driving control can be performed more safely.

The display device 600 is mounted on each moving body and displays various information to the driver. Specifically, the display device 600 is mounted in the interior of the vehicle at a position where the driver can see it. The display device 600 includes a display unit (not shown) and an operation unit.

The display unit is composed of a liquid crystal panel, an OLED (Organic Light Emitting Diode) panel, a fluorescent display tube, and the like. The display unit displays an image of the surroundings of the moving body taken by the camera 210, a guide line for movement control in automatic driving control, and a state of presence or absence of abnormality in the camera 210 detected by the abnormality detection system SYS2.

The operation unit includes buttons, a touch panel, and the like. The operation unit is an input device when the driver operates the automatic operation control device 500.

<2. Anomaly detection system>
The abnormality detection system SYS2 detects the presence or absence of an abnormality in the photographing unit 200. Specifically, the presence or absence of an abnormality in the camera 210 included in the photographing unit 200 is detected.

In the present embodiment, the abnormality refers to a state in which the camera 210 mounted on the moving body is misaligned (hereinafter referred to as “camera misalignment”). Specifically, the camera deviation is a system that detects a deviation from the mounting state of the camera 210, which is a reference in advance at the time of shipment from the factory. The camera deviation indicates a change from the mounted state of the camera 210 at a reference time such as when the factory is picked up. The camera deviation includes a change in the mounting position of the camera 210 and a change in the orientation of the camera 210. The change in orientation is a change in the tilt angle, pan angle, and roll angle of the camera 210.

The abnormality detection system SYS2 includes an abnormality detection device 100, a photographing unit 200, a sensor unit 300, and a communication bus 400.

The abnormality detection device 100 detects whether or not there is a camera shift based on the image around the moving body taken by the photographing unit 200. Details will be described later.

As described above, the photographing unit 200 captures an image of the periphery of the moving body for use in the automatic driving control device 500 or the like. In the abnormality detection system SYS2, as another function, an image captured by the abnormality detection device 100 is output. The photographing unit 200 includes a plurality of cameras 210. Each of the plurality of cameras 210 captures an image around the moving body. Since each of the plurality of cameras 210 captures a wide area around the moving body with one camera, it is preferable to have an optical system having a wide angle of view such as a fisheye lens.

Each of the plurality of cameras 210 is connected to the abnormality detection device 100 by wire or wirelessly, and outputs the captured image to the abnormality detection device 100.

The front camera is a camera that captures the front of a moving body. The back camera is a camera that captures the rear of a moving body. The left side camera is a camera that captures the left side of a moving body. The right side camera is a camera that captures the right side of a moving object.

The cameras are arranged so that their shooting ranges overlap each other. This makes it possible to photograph the entire circumference of the moving body in the horizontal direction. However, the number of cameras 210 included in the photographing unit 200 may be a single number, or may be a plurality of cameras other than four.

As described above, the sensor unit 300 measures various states of the vehicle for use in the automatic driving control device 500 and the like. In the abnormality detection system SYS2, the speed of the moving body is detected as another function. The sensor unit 300 includes at least a vehicle speed sensor 310.

The vehicle speed sensor 310 detects the speed of the moving body. The detected speed is output as an electric signal according to the detected value. The output electric signal is, for example, an analog voltage or a coded digital signal. In addition to the speed sensor 310, the sensor unit 300 may include position sensors such as a radar, a steering angle sensor, and a GPS (Global Positioning System). The sensor unit 300 is connected to the abnormality detection device 100 via the communication bus 400. The sensor data detected by the sensor unit 300, such as the above-mentioned electric signal based on the vehicle speed, is output to the abnormality detection device 100 via the communication bus 400. The communication bus 400 is, for example, a CAN (Controller Area Network) bus.

<3. Anomaly detection device>
<3.1. Configuration of anomaly detection device>
The configuration of the abnormality detection device 100 will be described with reference to FIG. As shown in FIG. 1, the abnormality detection device 100 includes an acquisition unit 110, a candidate extraction unit 120, a discrimination pixel identification unit 130, a brightness difference calculation unit 140, a comparison unit 150, and a feature point extraction unit 160. An abnormality detection unit 170 and a storage unit 180 are provided.

The abnormality detection device 100 includes, for example, a microcomputer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), an input / output port, and various circuits.

The CPU of the microcomputer, for example, realizes the functions of each functional block such as the acquisition unit 110 and the candidate extraction unit 120 included in the abnormality detection device 100 by reading and executing the program stored in the ROM. It should be noted that at least one or all of each functional block may be configured by hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The acquisition unit 110 acquires a captured image (frame image) from the camera 210 mounted on the moving body. The acquisition unit 110 outputs the acquired frame image to the candidate extraction unit 120.

The acquisition unit 110 is connected to the camera 210 by wire or wirelessly via an input / output port of a microcomputer that realizes the function.

The format of the frame image acquired by the acquisition unit 110 may be either analog or digital. When analog is used as the format of the frame image, the acquisition unit 110 may include an AD converter (Analog to Digital converter).

The candidate extraction unit 120 extracts feature point candidates from the frame image acquired by the acquisition unit 110. The candidate extraction unit 120 outputs the extracted feature point candidates to the discrimination pixel identification unit 130.

The discrimination pixel identification unit 130 identifies the first to Nth discrimination pixels having a predetermined first to Nth positional relationship from the feature point candidates extracted by the candidate extraction unit 120 (N is a natural number of 2 or more). .. The discrimination pixel identification unit 130 outputs the feature point candidate and the specified first to Nth discrimination pixels to the luminance difference calculation unit 140.

The luminance difference calculation unit 140 is the luminance difference between the feature point candidate extracted by the candidate extraction unit 120 and the kth (k is a natural number of 1 or more and N or less) discrimination pixel specified by the discrimination pixel identification unit 130. k The brightness difference is calculated. The luminance difference calculation unit 140 outputs the calculated k-th luminance difference to the comparison unit 150.
The comparison unit 150 compares the k-th luminance difference calculated by the luminance difference calculation unit 140 with the k-th threshold value set according to the k-th positional relationship. The comparison unit 150 outputs the comparison result to the feature point extraction unit 160. The feature point extraction unit 160 uses the feature point candidates extracted by the candidate extraction unit 120 based on the result of the comparison performed by the comparison unit 150. Is extracted. The feature point extraction unit 160 outputs the extracted feature points to the abnormality detection unit 170.

The abnormality detection unit 170 detects the camera deviation based on the time change of the position of the feature point extracted by the feature point extraction unit 160 and the movement amount of the moving body.

The abnormality detection unit 170 is connected to the communication bus 400 via an input / output port of a microcomputer that realizes the function. The abnormality detection unit 170 is connected to the sensor unit via the communication bus 400.

The storage unit 180 stores a program executed by the microcomputer, various settings, and the like. Specifically, the storage unit 180 is composed of, for example, a semiconductor storage medium such as a ROM or a flash memory, or a magnetic storage medium such as an HDD (Hard Disk Drive).

<3.2. Operation of anomaly detection device>
Next, the operation of the abnormality detection device 100 will be described in detail with reference to FIG.

The acquisition unit 110 acquires a captured image (frame image) from the camera 210 mounted on the moving body. The acquisition unit 110 communicates with the camera 210 via an input / output port of a microcomputer or the like to acquire a captured image.

The acquisition unit 110 periodically acquires the frame image at a predetermined cycle, for example, a 1/30 second cycle.

In the present embodiment, since the photographing unit 200 has a plurality of cameras 210, the acquisition unit 110 acquires a frame image from each camera 210.

The acquisition unit 110 performs predetermined image processing on the acquired frame image, and outputs the processed frame image to the candidate extraction unit 120.

When the frame image to be acquired is analog, the acquisition unit 110 may convert the analog frame image into a digital frame image (A / D conversion) by using an AD converter (not shown).

The candidate extraction unit 120 extracts feature point candidates from the frame image acquired by the acquisition unit 110. The candidate extraction unit 120 outputs the extracted feature point candidates to the discrimination pixel identification unit 130.

The feature point candidate is a feature point candidate extracted by the feature point extraction unit 160 described later. The feature point is a point that can be conspicuously detected in the frame image, such as an intersection of edges in the frame image.

Feature point candidates are detected, for example, from the corners of road markings drawn with white lines, shadows of cracks and irregularities on the road surface, stains on the road surface, gravel on the road surface, and the like. The feature point candidates can be extracted using, for example, a known feature point extraction method such as a Harris operator.

The extraction of feature point candidates will be described with reference to FIG. FIG. 2 is an example of a situation in which the vehicle body VB, the road surface RS, and the white line marking RM of the own vehicle are shown in the frame image PIC.

ROI (Region of Interest) 121, which is a range for extracting feature points, is set in the frame image PIC. Feature point candidates are extracted from within ROI 121. ROI 121 is set in a range in which the road surface RS is captured. Further, it is preferable that the ROI 121 is set in a wide range including the center of the frame image PIC. As a result, the candidate extraction unit 120 can suitably extract the feature point candidates even when the feature point candidates are not uniformly generated with respect to the ROI 121 and are unevenly distributed in a biased region. In the figure, ROI 121 is illustrated as a rectangular area, but this is not the case.

In the present embodiment, the candidate extraction unit 120 calculates a feature amount for each pixel in the ROI 121 set in the frame image PIC, and extracts feature point candidates based on the feature amount. The feature amount is an index showing how characteristic a pixel is with respect to surrounding pixels. For example, it is the degree of corner that shows the uniqueness of a corner. In the present embodiment, the feature amount is calculated by using the KLT (Kanade-Lucas-Tomasi tracker) method.

Prior to the calculation of the feature amount, the candidate extraction unit 120 performs various preprocessing on the frame image PIC. For example, the candidate extraction unit 120 converts a color image into a grayscale image and resamples the grayscale image. The size of the grayscale image is changed by resampling. After the resample, the candidate extraction unit 120 performs various filter processing and blurring processing for noise removal on the resampled graceful image.

The candidate extraction unit 120 converts the preprocessed frame image PIC into a differential image. The differential image is an image based on the amount of change in brightness with respect to the position of the pixel. In the differential image, the luminance differential value corresponding to each pixel of the frame image PIC is recorded. The luminance differential value includes an x-direction luminance differential value and a y-direction luminance derivative value. The x-direction luminance differential value is the amount of change in luminance in the x-direction, which is the horizontal direction of the frame image PIC. The y-direction luminance differential value is the amount of change in luminance in the y-direction, which is the vertical direction of the frame image PIC. A known differential filter such as a Sobel filter is used to generate a differential image.

For pixels located at coordinates (x, y) in the differential image, when the luminance differential value in the x direction is dx (x, y) and the luminance differential value in the y direction is dy (x, y), G11, G12 , G22 is defined as follows.

As shown in the equations (1) to (3), G11 is the square of the luminance derivative value in the x direction, G22 is the square of the luminance derivative value in the y direction, and G12 is the luminance derivative value in the x direction. It is the product of the value and the brightness derivative value in the y direction. Using G11, G12, and G22, a matrix M can be obtained from the following equation (4) for each pixel of the frame image PIC.

Next, the eigenvalue λ of the matrix M is obtained. Assuming that I is an identity matrix, it is calculated by the following equation (5).

Equation (5) can be solved as a quadratic equation of λ as two eigenvalues λ1 and λ2 as in the following equations (6) and (7).

In the present embodiment, the candidate extraction unit 120 selects the smaller of λ1 and λ2 as the feature amount. When the feature amount is larger than the predetermined threshold value T, the candidate extraction unit 120 extracts the corresponding pixel as the feature point candidate FPC. The extracted feature point candidate FPC is output to the discrimination pixel identification unit 130.

In the example shown in FIG. 2, three feature point candidates FPC1, FPC2 and FPC3 are extracted from the frame image PIC. The feature point candidates FPC1 and FPC2 correspond to the corners of the white line marking RM. The feature point candidate FPC3 corresponds to the unevenness of the road surface.

The number of feature point candidates extracted varies depending on the content of the frame image. For example, when a pedestrian crossing is shown, many corners of white line markings are shown, so more than three feature point candidates may be extracted. On the other hand, in the weather where shadows are hard to form such as cloudy weather, when only the road surface is reflected, the feature point candidates may not be extracted at all.

The discrimination pixel identification unit 130 identifies the first to Nth discrimination pixels having a predetermined first to Nth positional relationship from the feature point candidates extracted by the candidate extraction unit 120 (N is a natural number of 2 or more). .. The discrimination pixel identification unit 130 outputs the feature point candidate and the specified first to Nth discrimination pixels to the luminance difference calculation unit 140.

In the following description, the case where N = 2, that is, the case where there are two types of positional relationships will be described, but this is not the case.

FIG. 3 is a diagram for explaining the relationship between the feature point candidate and the discriminant pixel. As shown in FIG. 3, the first discrimination pixel DP1 is a pixel located in the first positional relationship from the feature point candidate FPC. The first positional relationship is a position of +4 pixels (hereinafter referred to as Δx = +4) in the horizontal direction and -2 pixels (hereinafter referred to as Δy = -2) in the vertical direction based on the feature point candidate FPC.

As shown in FIG. 3, the horizontal direction is positive from the left to the right of the image, and the vertical direction is positive from the top to the bottom of the image.

Further, the second discrimination pixel DP2 is a pixel located in a second positional relationship from the feature point candidate FPC. The second positional relationship is a position of -4 pixels (Δx = -4) in the horizontal direction and +2 pixels (Δy = +2) in the vertical direction with reference to the feature point candidate FPC.

The first to Nth positional relationships are preferably set so as to come on both sides of the feature point candidate FPC on the straight line L passing through the feature point candidate FPC. In FIG. 3, a first positional relationship is set on the upper right side of the feature point candidate FPC and a second positional relationship is set on the lower left side on the straight line L passing through the feature point candidate FPC.

The setting of the positional relationship will be described with reference to FIG. FIG. 4 shows the luminance distribution of 11 pixels in the horizontal vicinity centered on the feature point candidate FPC. Here, for the sake of simplicity, the discrimination pixel DP1 and the discrimination pixel DP2 are set at positions of Δx = ± 3 in the horizontal direction, unlike FIG.

In the present embodiment, the abnormality detection device 100 evaluates the brightness values of the feature point candidate FPC and the pixels in the vicinity of the feature point candidate FPC in order to detect a characteristic brightness distribution such as road surface unevenness. The luminance value of the pixel is evaluated by the luminance difference calculation unit 140, the comparison unit 150, and the feature point extraction unit 160, as will be described later.

When the feature point candidate FPC is extracted by the unevenness of the road surface or the shadow of the convex portion such as gravel, the brightness distribution in the vicinity of the feature point candidate FPC is a mountain having the feature point candidate FPC as the apex as shown in FIG. Often shows a type brightness distribution. The top of the mountain is a high-brightness part due to the reflection of light on the convex part, and the hem of the mountain is a low-brightness part due to the shadow of the convex part.

Therefore, by comparing the brightness of the pixel at the top of the mountain with the brightness of the pixel at the foot of the mountain, it is easy to determine whether or not the brightness distribution is a characteristic mountain-shaped distribution in the road surface unevenness. can do.

Therefore, in the present embodiment, the positional relationship corresponding to the foot of the mountain is defined in advance as the first to Nth positional relationships when the top of the mountain is used as a reference. As a result, when the position of the feature point candidate FPC is used as a reference, the pixels in the first to Nth positional relationships, that is, the discrimination pixels can be specified as the pixels at the foot of the mountain. Thereby, it is possible to easily evaluate whether or not the vicinity of the feature point candidate FPC has a chevron-shaped luminance distribution characteristic of the road surface unevenness.

Since one of the causes of the mountain-shaped luminance distribution as described above is the shadow, the positional relationship between the discriminant pixel and the feature point candidate FPC is preferably on a straight line L passing through the feature point candidate FPC. .. Further, the direction of the straight line L is more preferably directed to the direction of the light source, for example, the sun. As a result, the reflection and shadow of the convex portion can be appropriately captured, and the feature points can be extracted with higher accuracy.

The identified discriminant pixel is output to the luminance difference calculation unit 140 together with the feature point candidate.

The luminance difference calculation unit 140 is the luminance difference between the feature point candidate extracted by the candidate extraction unit 120 and the kth (k is a natural number of 1 or more and N or less) discrimination pixel specified by the discrimination pixel identification unit 130. k The brightness difference is calculated. The luminance difference calculation unit 140 outputs the calculated k-th luminance difference to the comparison unit 150.

Specifically, the luminance difference calculation unit 140 calculates the difference in luminance between the pixel of the feature point candidate pixel and the k-th discrimination pixel, and calculates the k-th luminance difference (k is a natural number of 1 to N). .. The luminance difference calculation unit 140 repeats the calculation of the luminance difference by changing k from 1 to N for one feature point candidate. As a result, the first to Nth luminance differences corresponding to the first to Nth discriminant pixels are calculated for one feature point candidate. The feature point candidate and the luminance difference are output to the comparison unit 150.

The comparison unit 150 compares the k-th luminance difference calculated by the luminance difference calculation unit 140 with the k-th threshold value set according to the k-th positional relationship. The comparison unit 150 outputs the comparison result to the feature point extraction unit 160. Specifically, the k-th luminance difference and the k-th threshold value are compared.

When the kth luminance difference is larger than the kth threshold value, the comparison result is true. When the k-th luminance difference is equal to or less than the k-th threshold value, the comparison result is false. For one feature point candidate, the comparison unit 150 repeats the comparison by changing k from 1 to N. As a result, N comparison results can be obtained for one feature point candidate. The comparison unit 150 performs a process of comparing the brightness difference with the threshold value for all the feature point candidates extracted by the candidate extraction unit 120. The feature point candidate and the comparison result are output to the feature point extraction unit 160.

Each discriminant pixel has a threshold value corresponding to the positional relationship. As shown in FIG. 3, the first discriminant pixel has a first threshold value corresponding to the first positional relationship. The second discriminant pixel has a second threshold value corresponding to the second positional relationship.

The setting of the threshold value corresponding to the positional relationship will be described with reference to FIG. As shown in FIG. 4, the luminance distribution in the vicinity of the feature point candidate FPC often takes a left-right asymmetric distribution. In FIG. 4, the brightness of the first discrimination pixel DP1 on the right side of the feature point candidate FPC is higher than the brightness of the second discrimination pixel DP2 on the left side of the feature point candidate FPC. Therefore, the first luminance difference LD1 between the first discriminant pixel DP1 and the feature point candidate FPC is larger than the second luminance difference LD2 between the second discriminant pixel DP2 and the feature point candidate FPC.

One of the causes of the asymmetry of the chevron-shaped brightness distribution of the road surface unevenness is the shadow of the convex portion as described above. Of the convex portions, the surface on the light source side reflects light and therefore tends to have high brightness. On the other hand, among the convex portions, the shaded surface tends to have low brightness because the shadow cast by itself may overlap.

Therefore, by setting different threshold values according to the positional relationship with the feature point candidate FPC, it is possible to extract feature points with higher accuracy.

For example, in FIG. 4, the threshold value corresponding to the discrimination pixel DP1 may be set smaller than the threshold value corresponding to the discrimination pixel DP2. That is, the first threshold value corresponding to the first positional relationship may be set smaller than the second threshold value corresponding to the second positional relationship.

FIG. 5 shows an example of the correspondence between the positional relationship and the threshold value. As shown in FIG. 5, the positional relationship, that is, the horizontal position Δx, the vertical position Δy, and the threshold value with respect to the feature point candidate are associated with each other with respect to the first to Nth numbers.

The correspondence between the positional relationship and the threshold value shown in FIG. 5 is specifically stored in the storage unit 180 in the form of a table. The discriminant pixel identification unit 130 described above appropriately refers to the table of the storage unit 180 and calls a predetermined positional relationship to specify the discriminant pixel. The comparison unit 150 appropriately refers to the table of the storage unit 180, calls the threshold value corresponding to the positional relationship, and compares the brightness difference with the called threshold value.

The feature point extraction unit 160 extracts feature points from the feature point candidates extracted by the candidate extraction unit 120 based on the result of the comparison performed by the comparison unit 150. The feature point extraction unit 160 outputs the extracted feature points to the abnormality detection unit 170.

Specifically, when at least one of the N comparison results obtained for one feature point candidate is false, the feature point extraction unit 160 extracts the feature point candidate as a feature point. In other words, if the N comparison results are all true, the feature point extraction unit 160 does not extract the feature point candidates as feature points. The extracted feature points are output to the abnormality detection unit 170.

The abnormality detection unit 170 detects the camera deviation based on the time change of the position of the feature point extracted by the feature point extraction unit 160 and the movement amount of the moving body. Specifically, the abnormality detection unit 170 performs optical flow calculation processing, movement vector calculation processing, movement amount estimation processing, and deviation determination processing.

The optical flow calculation process is a process of detecting an optical flow based on a time change of the position of an extracted feature point. Specifically, the anomaly detection unit 170 compares the feature points extracted based on the previously acquired frame image with the feature points extracted based on the frame image acquired this time, and compares the time of the position of the feature points. Calculate the change as an optical flow. There are various known methods for calculating the optical flow.

The optical flow calculation process will be described with reference to FIG. FIG. 6 shows a frame image PIC acquired at the next acquisition timing with respect to the frame image PIC shown in FIG. That is, the frame image PIC of FIG. 6 can be said to be an image after a predetermined time with respect to the frame image PIC of FIG. For example, it can be said to be an image around the moving body after 1/30 second, which is the image acquisition cycle. Similar to FIG. 2, the frame image PIC of FIG. 6 shows the vehicle body VB of the own vehicle, the road surface RS, and the white line marking RM.

In FIG. 6, the feature points extracted in the frame image PIC of FIG. 2 are superimposed and shown by white circles (◯). In FIG. 6, only the feature point candidates caused by the corners of the white line marking RM in FIG. 2 are extracted as feature points by the feature point extraction unit 160 and remain. The feature point candidates due to the road surface unevenness are not extracted by the feature point extraction unit 160.

The feature points extracted from the frame image PIC acquired this time are indicated by black circles (●). Two feature points, FP1 and feature point FP2, which are caused by the corners of the white line marking RM, are extracted.

The anomaly detection unit 170 associates the feature points of the frame image PIC this time with the feature points of the previous frame image in consideration of the pixel values in the vicinity thereof. The abnormality detection unit 170 calculates the optical flow based on the respective positions of the associated feature points. Specifically, of the two associated feature points, the vector when the feature point of the previous frame image is the start point and the feature point of the frame image this time is the end point is used as the optical flow for the associated feature points. ..

In FIG. 6, two are calculated: an optical flow OF1 with feature points corresponding to the lower left corner of the white line marking RM, and an optical flow OF2 with feature points corresponding to the lower right corner.

If the corresponding feature point is not found, the abnormality detection unit 170 does not calculate the optical flow. The calculated optical flow is used in the movement vector calculation process.

The movement vector calculation process is a process of converting the calculated optical flow into coordinates and calculating the movement vector of the feature points in the real space.

The optical flow calculated by the optical flow calculation process is a frame image taken by the camera 210, that is, the movement of feature points in a two-dimensional coordinate system. The anomaly detection unit 170 converts the optical flow into a movement vector, which is the movement in the three-dimensional coordinate system in which the actual moving body moves, in the movement vector calculation process.

The calculation of the movement vector will be described with reference to FIG. 7. For the sake of simplicity, FIG. 7 will be described based on the pinhole model.

In the pinhole model, the image taken by the camera is obtained by projecting the subject onto a predetermined shooting surface PPc along a straight line connecting the viewpoint VPc of the camera and the subject.

For example, as shown in FIG. 7, the start point of the movement vector OFt in the real space is projected to the point where the straight line connecting the start point of the movement vector OFt and the viewpoint VPc of the camera intersects the photographing surface PPc.

Similarly, the end point of the movement vector OFt is projected at the point where the straight line connecting the end point of the movement vector OFt and the viewpoint VPc of the camera intersects the photographing surface PPc. When the start point and the end point of the movement vector OFt projected on the imaging surface PPc are connected in this way, the optical flow OFc in the frame image is obtained.

Conversely, the subject in the real space exists on the straight line connecting the viewpoint VPc of the camera and the subject projected on the shooting surface PPc. That is, the start point of the movement vector OFt is on the straight line connecting the viewpoint VPc of the camera and the start point of the optical flow OF1. Similarly, the end point of the movement vector OFt is on the straight line connecting the viewpoint VPc of the camera and the end point of the optical flow OFc.

However, since the information in the depth direction is lost at the shooting stage, the movement vector OFt cannot be uniquely determined from the optical flow OFc. Therefore, in this embodiment, it is assumed that the movement vector OFt is on the virtual plane PPt. Specifically, the virtual plane PPt is a road surface. Based on this assumption, the abnormality detection unit 170 can uniquely obtain the movement vector OFt from the optical flow OFc. Specifically, the abnormality detection unit 170 obtains the movement vector OFt from the intersection of the straight line connecting the viewpoint VPc of the camera and the optical flow OFc and the virtual plane PPt.

In the movement vector calculation process, the abnormality detection unit 170 indicates in which direction the point of the coordinate is from the viewpoint of the camera with respect to the point of an arbitrary coordinate in the frame image from the shooting direction of the camera, the horizontal angle of view, and the vertical angle of view. Find the direction vector that represents. Next, the abnormality detection unit 170 sets a predetermined value as the viewpoint position of the camera. In the above, a straight line connecting an arbitrary pixel in the frame image and the viewpoint VPc of the camera is determined. The abnormality detection unit 170 finds the intersection of the straight line and the virtual plane PPt. This is done for the start and end points of the optical flow OFc, respectively. As a result, the movement vector OFt can be obtained. As the shooting direction and the viewpoint position of the camera, for example, the coordinates and orientation of the default mounting position at the time of shipment from the factory may be used.

In FIG. 7, the description is based on the pinhole model, but this is not the case. When a fisheye lens is used as in the present embodiment, it may be desirable to set the imaging surface PPc as a curved surface such as a hemisphere instead of a flat surface. It is also preferable that the distortion of the optical system is corrected in advance.

The movement amount estimation process is a process of calculating the estimated movement amount of the moving body based on the movement vector. As described above, the movement vector indicates the amount and direction of movement of the feature point on the road surface. Therefore, if the feature points are caused by a stationary object, the movement vector can be considered to relatively indicate the movement amount and the movement direction of the moving body. In this embodiment, it is assumed that the feature points are caused by a stationary object such as a white line mark. Therefore, based on the movement vector, the abnormality detection unit 170 can estimate the movement amount of the moving body. The estimated movement amount is, for example, a front-back estimated movement amount which is a front-back direction component of the movement vector and a left-right estimated movement amount which is a left-right direction component. The estimated movement amount is used in the deviation determination process.

When a plurality of movement vectors are detected, it is preferable that the abnormality detection unit 170 performs various statistical processing such as using an average or a median to calculate the estimated movement amount. In addition, it is more preferable to create a histogram and perform advanced statistical processing such as excluding outliers and selecting the maximum peak.

The deviation determination process is a process of comparing the estimated movement amount with the movement amount of the moving body calculated based on the vehicle speed acquired from the sensor unit 300, and determining the presence or absence of camera deviation. Specifically, the abnormality detection unit 170 determines that there is a camera deviation when the estimated movement amount estimated from the frame image and the movement amount of the moving body do not match.

The abnormality detection unit 170 calculates the amount of movement of the moving body in the deviation determination process based on the vehicle speed acquired from the sensor unit 300. Specifically, the movement amount of the moving body is calculated from the difference between the acquisition time of the current frame and the acquisition time of the previous frame, that is, the product of the frame acquisition interval and the vehicle speed acquired from the sensor unit 300.

When the moving body is in the straight-ahead state, the calculated movement amount of the moving body is in the front-rear direction. The amount of movement in the left-right direction is 0. Therefore, the abnormality detection unit 170 compares the estimated front-back movement amount with the movement amount of the moving body. Also, the estimated left-right movement amount is compared with 0.

As a result of the comparison, when it is determined that there is no difference between the front-back estimated movement amount and the left-right estimated movement amount, the abnormality detection unit 170 determines that there is no camera deviation. On the contrary, when it is determined that there is a difference between the estimated front-back movement amount and the left-right estimated movement amount, the abnormality detection unit 170 determines that there is a camera deviation.

In addition, "there is no difference" here means that the difference or ratio between the two values is within a predetermined range. For example, when the difference between the estimated front-rear movement amount and the movement amount of the moving body is smaller than a predetermined threshold value, it may be determined that there is no difference.

Next, the abnormality detection process according to the embodiment will be described with reference to FIG. FIG. 8 is a flowchart illustrating an abnormality detection process. This process is repeated in a predetermined cycle.

When the abnormality detection process starts, the abnormality detection device 100 extracts feature points (step S81). The feature point extraction process will be described later.

The abnormality detection device 100 determines whether or not the number of extracted feature points is equal to or greater than a predetermined value (step S82). When the number of feature points is less than the predetermined value (S82: No), the abnormality detection device 100 determines that the camera deviation cannot be determined (step S83), and ends the abnormality detection process.

When the number of the extracted feature points is equal to or more than a predetermined value (S83: Yes), the abnormality detection device 100 is based on the feature points extracted based on the image acquired this time and the feature points extracted based on the image acquired last time. The optical flow is calculated (step S84).

The abnormality detection device 100 performs coordinate conversion on the calculated optical flow and calculates a movement vector (step S85).

The abnormality detection device 100 estimates the estimated movement amount of the moving body based on the calculated movement vector (step S86).

The abnormality detection device 100 compares the estimated estimated movement amount with the movement amount of the moving body calculated based on the vehicle speed, determines whether or not the camera is displaced (step S87), and ends the abnormality detection process. To do.

Next, the feature point extraction process (step S81) according to the embodiment will be described in detail with reference to FIG. FIG. 9 is a flowchart illustrating the feature point extraction process. As an example, the number of discrimination pixels is two, but this is not the case.

When the feature point extraction process starts, the abnormality detection device 100 acquires an image from the camera (step S901).

The abnormality detection device 100 extracts feature point candidates from the acquired image (step S902).

The abnormality detection device 100 selects one from the extracted feature point candidates (step S903).

The abnormality detection device 100 identifies two discriminant pixels in two types of positional relationships with respect to one selected feature point candidate (step S904). Hereinafter, the two types of positional relationships will be referred to as a first positional relationship and a second positional relationship, respectively. Further, the two discrimination pixels are the first discrimination pixel and the second discrimination pixel, respectively. Here, the number of discrimination pixels is two as an example, but any number of N (natural number with N ≧ 2) may be used.

The abnormality detection device 100 calculates the first luminance difference, which is the luminance difference between the first discriminant pixel and the feature point candidate (step S905).

The abnormality detection device 100 compares the first luminance difference with the first threshold value corresponding to the first positional relationship (step S906). When the first luminance difference is equal to or less than the first threshold value (S906: No), the abnormality detection device 100 ends the process for one selected feature point candidate, and executes step S910 and subsequent steps.

When the first luminance difference is larger than the first threshold value (S906: Yes), the abnormality detection device 100 executes the process for the second discriminant pixel.

The abnormality detection device 100 calculates the second luminance difference, which is the luminance difference between the second determining pixel and the feature point candidate (step S907).

The abnormality detection device 100 compares the second luminance difference with the second threshold value corresponding to the second positional relationship (step S908). When the second luminance difference is equal to or less than the second threshold value (S908: No), the abnormality detection device 100 ends the processing for one selected feature point candidate, and executes step S910 and subsequent steps.

When the second luminance difference is larger than the second threshold value (S908: Yes), the abnormality detection device 100 extracts one selected feature point candidate as a feature point (step S909).

When arbitrarily setting N discriminant pixels, it is preferable to repeat the processes of S905 to S906 as one set N times. In the above description, it has been described as an example of N = 2.

The abnormality detection device 100 determines whether or not the above processing has been performed on all the feature point candidates (step S910). If it has not been executed (S910: No), step 903 and subsequent steps are executed again for the feature point candidates that have not been implemented yet.

If the abnormality detection device has performed processing on all the feature point candidates (S910: Yes), the feature point extraction process ends.

<3.3. Effect of anomaly detection device>
Next, the effect of the abnormality detection device according to the embodiment will be described.

The abnormality detection device 100 identifies the first to Nth discriminant pixels having a predetermined first to first Nth positional relationship (N is a natural number of 2 or more) with the feature point candidate.

The abnormality detection device 100 calculates the brightness difference between the extracted feature point candidate and the identified kth discrimination pixel (k is a natural number of 1 or more and N or less), and sets the kth brightness difference in the kth positional relationship. Compare with the kth threshold set accordingly.

As a result, the abnormality detection device 100 can detect the asymmetric luminance distribution caused by the road surface unevenness in the vicinity of the feature point candidate, and can accurately extract the feature points that can be stably extracted in time. ..

Further, the abnormality detection device 100 specifies a pixel located on a straight line passing through the feature point candidate as the first discriminating pixel, and the first discriminating pixel is located on the straight line and based on the extracted feature point candidate. A pixel located on the opposite side of the pixel is specified as a second discrimination pixel.

As a result, the anomaly detection device 100 can accurately capture the shadow of the road surface unevenness, which is one of the causes of the asymmetric luminance distribution in the vicinity of the feature point candidate, and can extract it stably in time. Feature points can be extracted with higher accuracy.

As described above, the present embodiment has been described. Here, a comparison with the prior art is made. In the prior art, it is known that feature points are extracted from feature point candidates by comparing feature quantities. For example, a feature point candidate obtained from a feature of an image with clear light and darkness such as a corner of a white line can obtain a high feature amount. On the other hand, the feature point candidates obtained from the relatively vague image features such as the unevenness of the road surface have a low feature amount near the threshold value T.

That is, the feature points that can be stably extracted in time have a relatively high feature amount. On the other hand, it can be said that feature points that are stable in time and difficult to extract have relatively low feature quantities. Taking advantage of this, conventionally, attempts have been made to extract feature points that can be stably extracted in a timely manner by appropriately setting a threshold value for the feature amount. However, it was difficult to draw an appropriate threshold value for the feature quantity. For this reason, the feature points may be excessively reduced, or conversely, the feature points may be left too much. As described above, the problem is that the accuracy of extracting the feature points from the feature point candidates is low.

One of the reasons for this is that the magnitude of the feature amount is mainly due to the magnitude of the change in brightness. That is, even if the luminance distribution is asymmetrical and characteristic, such as the unevenness of the road surface, the magnitude of the luminance value itself is often not reflected in the feature amount. Therefore, it is difficult to extract with high accuracy by the method of evaluating the feature quantity itself like the conventional method.

On the other hand, the present embodiment is characterized in that the feature point candidates are extracted based on the luminance distribution in the vicinity of the feature point candidates as described above. The anomaly detection device 100 can accurately discriminate an asymmetric characteristic brightness distribution such as unevenness of a road surface by setting a threshold value corresponding to a predetermined positional relationship based on a feature point candidate. it can. In addition, by setting the positional relationship on a straight line passing through the feature point candidates, it is possible to accurately capture the shadow of unevenness, which is one of the causes of the asymmetric luminance distribution. As a result, the abnormality detection device 100 can more accurately extract feature points that can be stably extracted in time.

Next, a modified example will be described.

The abnormality detection device 100 according to the modified example determines a straight line passing through the feature point candidate based on the recording direction of the pixels of the acquired image. FIG. 10 is a diagram showing an example in the case where a straight line is set parallel to the recording direction of the pixels. In FIG. 10, when the pixels are recorded in the horizontal direction of the image, the straight line L passing through the feature point candidate FPC is set in the horizontal direction. That is, the discriminating pixel is specified from a predetermined position located in the horizontal direction with respect to the feature point candidate. In FIG. 10, as an example, two discriminant pixels DP1 and discriminant pixel DP2 are specified at positions of Δx = ± 3.

In general, the pixels that make up a frame image are often recorded in memory with pixels for one horizontal line as one unit. Therefore, in the horizontal direction, that is, in the direction parallel to the recording direction of the pixels, the pixels are recorded at positions adjacent to or close to each other on the memory. On the other hand, in a direction that is not parallel to the recording direction of the pixel, such as a vertical direction or an oblique direction, the pixel is recorded at a distant position on the memory.

Generally, a memory look-ahead function such as a cache function mounted on a CPU has a high look-ahead effect for data at a position close to the memory and a low look-ahead effect for data far from the memory. Therefore, when the abnormality detection device 100 has the memory look-ahead function, the effect of the memory look-ahead function can be improved and the calculation time can be reduced by setting the recording direction of the pixels in the direction of the straight line L.

In the above description, the horizontal direction is taken as an example of the recording direction of the pixels, but this is not the case. Pixels may be recorded in the vertical direction or the diagonal direction.

Further, in the above modification, the direction of the straight line passing through the feature point candidate is fixed with respect to the light source direction that changes depending on the traveling direction of the moving body. Therefore, it may not be possible to accurately capture the shadows of the unevenness. Therefore, the image may be rotated in advance so that the recording direction of the pixels and the direction of the light source match.

For example, when the sun, that is, a light source, is directly in front of the camera, the direction of the light source is vertical in the image. On the other hand, the abnormality detection device according to the modified example identifies the discriminant pixel at the position in the recording direction of the pixel, that is, the horizontal direction with respect to the feature point candidate. Therefore, it may be difficult to detect the shadow of unevenness. In such a case, the acquired image is rotated 90 degrees in advance. Then, the direction of the light source becomes the horizontal direction in the image, and the recording direction of the pixels and the direction of the light source match. As a result, the shadows of the unevenness can be accurately captured, and the feature points that can be stably extracted in time can be extracted with higher accuracy.

Further, the abnormality detection device 100 according to the modified example may correct the positional relationship for specifying the discriminant pixel based on the position in the frame image of the feature point candidate.

The correction of the positional relationship will be described with reference to FIG. FIG. 11 is a diagram showing an example of correction of the positional relationship between the discrimination pixel for the feature point candidate near the center of the frame image PIC and the discrimination pixel for the feature point candidate near the outer edge of the frame image PIC.

As a premise, in the case of no correction, as an example, the discriminant pixel is specified in the positional relationship (Δx = ± 3) 3 pixels horizontally away from the feature point candidate.

First, the feature point candidate is the frame image PIC. It is assumed that it is located in the central region A1. In this case, the abnormality detection device 100 does not correct the positional relationship. Therefore, the discriminant pixel is specified in a positional relationship (Δx = ± 3) 3 pixels horizontally separated from the feature point candidate, which is the case of no correction.

Next, it is assumed that the feature point candidate is located in the slightly outer edge region A2 of the frame image PIC. In this case, the abnormality detection device 100 corrects the positional relationship. The abnormality detection device 100 corrects the positional relationship so that the closer the feature point candidate is to the outer edge of the image, the closer the discrimination pixel is to the feature point candidate. Specifically, in the region A2, the distance between the feature point candidate and the discriminant pixel is corrected to be reduced to 67% with respect to the positional relationship in the case of no correction. That is, in the region A2, the abnormality detection device 100 corrects the positional relationship so as to specify the discriminant pixel in the positional relationship (Δx = ± 2) that is two pixels horizontally separated from the feature point candidate.

Next, it is assumed that the feature point candidate is located in the region A3 near the outer edge of the frame image PIC. In this case, the abnormality detection device 100 corrects the discriminant pixel so that the discrimination pixel is closer to the feature point candidate than when the feature point candidate is located in the area A2. Specifically, in the region A3, the distance between the feature point candidate and the discriminant pixel is corrected to be reduced to 33% with respect to the positional relationship in the case of no correction. That is, in the region A3, the abnormality detection device 100 corrects the positional relationship so as to specify the discriminant pixel in the positional relationship (Δx = ± 1) one pixel away from the feature point candidate in the horizontal direction.

As in this embodiment, a wide-angle lens such as a fisheye lens is often used in the optical system of a camera used for peripheral monitoring of a moving body. Therefore, the subject appearing on the outer edge of the image is often smaller than the subject appearing in the center of the image. Even when a fisheye lens is not used, the road surface on the outer edge of the frame image is farther away than the center, so that the image appears smaller in the frame image. That is, the shadow due to the unevenness of the road surface also appears smaller at the outer edge of the image than at the center of the image.

As in the modified example, the abnormality detection device 100 corresponds to the size of the shadow on the frame image by correcting the positional relationship for specifying the discrimination pixel based on the position in the frame image of the feature point candidate. It becomes possible to specify the discriminant pixel from the positional relationship. As a result, feature points can be extracted with high accuracy over the entire frame image.

In the above description, the positions of the feature point candidates in the frame image PIC of the feature point candidates have been described by dividing them into three regions, but the description is not limited to this. The abnormality detection device 100 may continuously correct based on the position in the frame image of the feature point candidate. For example, the abnormality detection device 100 may correct the positional relationship by determining the correction amount by a continuous function in which the distance between the center of the frame image and the feature point candidate is a variable. Further, the abnormality detection device 100 may correct the positional relationship by predetermining the correction amount corresponding to the position of the feature point candidate in the frame image as a table.

Further, the abnormality detection device 100 according to the modified example may correct the threshold value based on the brightness representative value of the frame image. Specifically, the abnormality detection device 100 calculates the average brightness of the frame image and corrects the threshold value based on the average brightness. For example, when the average brightness of the frame image is higher than a predetermined threshold value, the abnormality detection device 100 makes the kth threshold value (k is a natural number of 1 or more and N or less) larger than that of the case of no correction. to correct. For example, it is advisable to take the product of a correction coefficient larger than 1.

When the typical brightness value of the frame image is high, the surroundings are often bright. The frame image taken in such a state has high contrast. When looking at the road surface unevenness, the peaks of the brightness distribution tend to be high in the vicinity of the feature point candidates due to the road surface unevenness.

On the contrary, when the representative brightness value of the image is low, it is often the case that the surroundings are dark or shadows are unlikely to occur such as cloudiness. Therefore, the contrast of the frame image becomes low. When looking at the road surface unevenness, the peaks of the brightness distribution tend to be low in the vicinity of the feature point candidates due to the road surface unevenness.

In other words, when the ambient brightness, that is, the representative brightness value of the frame image fluctuates, the feature point candidates that could be extracted as feature points before the fluctuation may not be extracted after the fluctuation. That is, the feature points may not be extracted stably in time.

The abnormality detection device 100 according to the modified example corrects the threshold value based on the representative brightness value of the image. As a result, even if the brightness distribution in the vicinity of the feature point candidate fluctuates, the feature points can be extracted stably in time.

Further, the abnormality detection device 100 according to the modified example may use a plurality of straight lines when specifying the discrimination pixel. For example, in FIG. 3, the abnormality detection device 100 according to the modified example may set another straight line in a direction orthogonal to the straight line L, and specify the discrimination pixel from the straight line as well.

Further, the abnormality detection device 100 according to the modified example does not have to have an integer positional relationship as the positional relationship for specifying the discrimination pixel. In other words, the positional relationship may be set so as to specify the discriminant pixel from between the pixels. In such a case, it is preferable to calculate the brightness of the discriminant pixel by using the pixel-to-pixel interpolation value. Specifically, consider a case where the discriminant pixel is specified from the positional relationship of Δx = 1.5 and Δy = 1.5. In this case, the brightness of the discriminant pixel may be calculated by a known pixel interpolation method such as a bilinear interpolation method based on Δx = 1 and 2 and four pixels located at Δy = 1 and 2.

Although the embodiment of the present invention has been described above, the above-described embodiment is merely an example for carrying out the present invention. Therefore, the present invention is not limited to the above-described embodiment, and the above-described embodiment can be appropriately modified and implemented within a range that does not deviate from the gist thereof.

The abnormality detection device according to the present invention can be used for abnormality detection of a camera mounted on a moving body.

100 Anomaly detection device 110 Acquisition unit 120 Candidate extraction unit 130 Discrimination pixel identification unit 140 Luminance difference calculation unit 150 Comparison unit 160 Feature point extraction unit 170 Abnormality detection unit 200 Imaging unit 300 Sensor unit 400 Communication bus

Claims (7)

An acquisition unit that acquires images from a camera mounted on a moving body,
A candidate extraction unit that extracts feature point candidates from the image acquired by the acquisition unit, and a candidate extraction unit.
A discriminant pixel identification unit that identifies the first to Nth discriminating pixels having a predetermined first-to-Nth (N is a natural number of 2 or more) positional relationship with the feature point candidate extracted by the candidate extraction unit.
A luminance difference calculation unit that calculates the kth luminance difference, which is the luminance difference between the extracted feature point candidate and the kth (k is a natural number of 1 or more and N or less) discrimination pixel specified by the discrimination pixel identification unit.
A comparison unit that compares the kth luminance difference with the kth threshold value set according to the kth positional relationship, and
A feature point extraction unit that extracts feature points from the feature point candidates based on the result of the comparison unit comparing the first to Nth luminance differences with the first to first N threshold values.
An abnormality detection unit that detects the presence or absence of an abnormality in the camera based on the time change of the position of the feature point extracted by the feature point extraction unit and the movement amount of the moving body.
Anomaly detection device. The discrimination pixel identification unit is
A pixel located on a straight line passing through the extracted feature point candidate is specified as the first discriminating pixel.
A pixel located on the straight line and on the opposite side of the first discrimination pixel with reference to the extracted feature point candidate is specified as a second discrimination pixel.
The first threshold is different from the second threshold.
The abnormality detection device according to claim 1. The abnormality detection device according to claim 2, wherein the direction of the straight line passing through the extracted feature point candidates is determined based on the recording order of the pixel values of the acquired image. The discrimination pixel identification unit is
According to any one of claims 1 to 3, at least one of the first to Nth positional relationships is corrected based on the position of the extracted feature point candidate in the acquired image. The described anomaly detector. The discrimination pixel identification unit is
At least one of the first to Nth positional relationships is corrected so that the closer the feature point candidate is to the outer edge of the acquired image, the closer the distance between the feature point candidate and the discriminating pixel is. The abnormality detection device according to claim 4, wherein the abnormality detection device is characterized. The extraction unit
The abnormality detection device according to any one of claims 1 to 5, wherein the threshold value is corrected based on the brightness representative value of the acquired image. The acquisition process of acquiring images from the camera mounted on the moving body,
A candidate extraction step for extracting feature point candidates from the acquired image, and
A discriminant pixel identification step for identifying the first to Nth discriminant pixels having a predetermined positional relationship between the extracted feature point candidates and a predetermined first to first N (N is a natural number of 2 or more).
A luminance difference calculation step for calculating the luminance difference between the extracted feature point candidate and the specified kth (k is a natural number of 1 or more and N or less) discrimination pixel, and
A comparison step of comparing the k-th luminance difference with the k-th threshold value set according to the k-th positional relationship, and
A feature point extraction step of extracting feature points from the feature point candidates based on the result of comparing the first to Nth luminance differences with the first to Nth threshold values, and
An abnormality detection step of detecting the presence or absence of an abnormality in the camera based on the time change of the position of the extracted feature point and the movement amount of the moving body.
Anomaly detection method comprising.

Images