JP7446445B2 - Image processing device, image processing method, and in-vehicle electronic control device - Google Patents

Description

Ingest by reference

This application claims priority to Japanese Patent Application No. 2020-141436, which was filed on August 25, 2020, and its contents are incorporated into this application by reference.

The present invention relates to an image processing device, and particularly to a method for easily detecting an abnormality in a vehicle-mounted camera.

With the advancement of electronic control technology for vehicles such as advanced driving assistance and autonomous driving, image sensors are now being installed outside the vehicle interior, and the chances of the front part of the image sensor (e.g. lens) becoming dirty is increasing. . If foreign matter adheres to the front lens of the image sensor, images of the relevant area cannot be captured, which may reduce the accuracy of recognizing objects outside the vehicle.

For example, in Patent Document 1 (Japanese Unexamined Patent Publication No. 2007-318355), a differential image generation unit acquires a plurality of image data captured by a camera at different times from an image storage unit, and calculates the difference between these image data. Generate differential image data. The difference image updating unit compares the previous difference image and the current difference image for each pixel, and updates the current difference image with the pixel with the larger difference. When the number of updates of the difference image reaches a predetermined number of times, the latest difference image is output to the dirt determining section. The dirt determination section describes an imaging device that determines the presence or absence of lens dirt in the camera based on this latest difference image.

Furthermore, Patent Document 2 (Japanese Unexamined Patent Publication No. 2012-166705) discloses a detection means for detecting a predetermined elongated detection object formed on a predetermined vehicle surrounding area based on an image taken by an on-vehicle camera. and a determining means for determining the presence or absence of foreign matter adhering to the lens based on the detection result thereof, and the determining means 8 determines a predetermined constancy in the length or formation interval of the detected object to be detected. A foreign matter adhesion determination device for a vehicle-mounted camera lens is described, which performs a determination based on the presence or absence of foreign matter as a determination criterion.

Furthermore, Patent Document 3 (Japanese Unexamined Patent Application Publication No. 2019-29940) discloses a deposit detection device that detects deposits that appear in a photographed image taken by an imaging device installed on a moving object, which detects deposits from the photographed image. a contour extraction section that extracts a contour region of the object as a contour region; an inner extraction section that extracts a region inside the contour of the deposit from the photographed image as an inner region; and a comparison between the contour region and the contour of the deposit. Then, the area that matches either the shape or the brightness is detected as the adhered object outline area, and the inner area and the inside of the outline of the adhered object are compared, and the area that matches either the shape or the brightness is detected as the adhered object outline area. Deposit detection that detects a lens deposit as an inside region, and detects a region of lens deposits consisting of either the outline of the deposit or the inside of the contour from either the deposit contour region or the deposit inner region as a deposit detection region. A deposit detection device is described that includes a region determination section.

The camera has a function of detecting hardware failure of the image sensor. Furthermore, many methods have been proposed for detecting problems caused by deposits on lenses provided on the front surface of a camera, but these methods require a high processing load, and there is a need to reduce the processing load.

Therefore, it is an object of the present invention to detect camera abnormalities with a low processing load.

A typical example of the invention disclosed in this application is as follows. That is, an image processing device that processes images taken by an on-vehicle camera includes an image acquisition unit that acquires images from the on-vehicle camera, and an image processing device that processes images taken by an on-vehicle camera based on the types of structures estimated to exist around the own vehicle. a structure color identification unit that identifies color information of the structure; an image area identification unit that identifies an area in the image where the structure is estimated to exist based on the type of the structure; and the vehicle-mounted camera. and a camera abnormality detection unit that detects an abnormality in the image, and the camera abnormality detection unit detects the color information obtained from the structure color identification unit and the color information obtained from the structure color identification unit when the structure exists around the host vehicle. The method is characterized in that an abnormality in the vehicle-mounted camera is detected by comparing color components of an area in which the structure is estimated to exist.

According to one aspect of the present invention, camera abnormality can be detected with a low processing load. Problems, configurations, and effects other than those described above will be made clear by the description of the following examples.

FIG. 1 is a block diagram showing the configuration of an image processing device according to an embodiment of the present invention. It is a figure showing the example of composition of the structure position table of the example of the present invention. FIG. 3 is a diagram illustrating a configuration example of a structure color component table according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a configuration example of a structure color component table according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a configuration example of a structure color component table according to an embodiment of the present invention. It is a flowchart of the camera abnormality detection process of the Example of this invention. It is a flowchart of the camera abnormality detection process of the Example of this invention. It is a flowchart of the camera abnormality detection process of the Example of this invention.

<Example 1>
FIG. 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention.

The image processing device 1 is installed in an in-vehicle electronic control unit (ECU) or an in-vehicle camera device, and includes a memory 10, a computing unit (CPU) 30, a network interface (CAN) 40, an interface (I/F) 50, and a network interface ( Eth) 60.

Arithmetic device 30 executes a program stored in memory 10. A part of the processing performed by the arithmetic device 30 by executing a program may be executed by another arithmetic device (for example, hardware such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)).

The memory 10 includes ROM and RAM, which are nonvolatile storage elements. The ROM stores unchangeable programs (eg, BIOS) and the like. RAM is a high-speed and volatile storage element such as DRAM (Dynamic Random Access Memory) and a non-volatile storage element (non-temporary storage medium) such as SRAM (Static Random Access Memory). Stores programs executed by the program and data used during program execution.

Specifically, the memory 10 includes an image acquisition unit 11, an own vehicle information acquisition unit 12, a map information acquisition unit 13, an image area identification unit 14, a structure color identification unit 16, a camera abnormality detection unit 18, and an abnormal area notification determination unit. Stores programs for realizing each functional block such as the section 19 and the abnormality notification section 21. The memory 10 also stores data referenced during program execution, such as a structure position table 15, a structure color component table 17, and an abnormal area notification table 20.

A network interface (CAN) 40 controls communication with other devices mounted on the vehicle via a CAN (Control Area Network). The interface (I/F) 50 is a serial or parallel input/output interface, and an in-vehicle camera 51 is connected thereto. A network interface (Eth) 60 controls communication via Ethernet (registered trademark, hereinafter the same) with other devices mounted on the vehicle.

The image acquisition unit 11 acquires an image taken by the in-vehicle camera 51 via the interface 50. The own vehicle information 41 including vehicle behavior (for example, speed, acceleration, steering angle, amount of brake and accelerator operation by the occupant, etc.) is acquired via the own vehicle information acquisition unit 12 and the network interface (CAN) 40 . The map information acquisition unit 13 acquires map information 61 via a network interface (Eth) 60. The acquired map information may be detailed map information for automatic driving or map information for navigation.

The image area specifying unit 14 refers to the structure position table 15 and specifies a structure detection area in the image where a structure is estimated to exist. The structure position table 15 is a table in which information on areas in which structures estimated from map information may exist in an image is recorded, and the details thereof will be explained with reference to FIG. 2.

The structure color specifying unit 16 refers to the structure color component table 17 and specifies the color of the structure to be determined. The structure color component table 17 is a table in which color information for each type of structure is recorded, and its details will be explained with reference to FIGS. 3, 4, and 5.

The camera abnormality detection unit 18 detects a deposit on the front surface of the camera (for example, the lens) based on color information of structures present in the image taken by the vehicle-mounted camera 51.

The abnormal area notification determination unit 19 refers to the abnormal area notification table 20 and determines whether the abnormality detected by the camera abnormality detection unit 18 has been notified. The abnormal area notification table 20 is a table in which abnormalities notified by the abnormality notification unit 21 are recorded. The abnormality notification unit 21 notifies the occupant and other ECUs of the abnormality that the abnormal area notification determination unit 19 has determined has not been reported.

FIG. 2 is a diagram showing an example of the structure of the structure position table 15 according to the embodiment of the present invention.

The structure position table 15 is referred to by the image area specifying unit 14 in order to specify a structure detection area where a structure is estimated to exist (i.e., an area for determining the color of a structure), and is used for each type of structure. Holds the coordinate data in the image.

For example, the structure position table 15 shown in FIG. 2 records the positions of structure detection areas in which traffic lights, road signs, and crosswalks are estimated to exist in an image as structures whose colors are to be determined. The structure detection area recorded in the structure position table 15 shown in FIG. 2 is defined by the three coordinates of the vertices of the triangle. The shape of the structure detection area recorded in the structure position table 15 may be a quadrilateral (trapezoid, square, rectangle) instead of a triangle. Since the image taken by the vehicle-mounted camera 51 appears small from a distance, defining the structure detection area as a triangle or trapezoid accurately represents the area where the structure is present, and the structure detection area can be made small.

Further, a plurality of structure detection areas may be recorded for the same type of structure. For example, since traffic lights are installed on the left side of the road (above the own lane when driving on the left) or on the right side (above the oncoming lane when driving on the left), multiple structure detection areas can be defined depending on the traffic light installation pattern. good.

Since the visible position of the structure in the image changes depending on the distance from the intersection, the structure detection area obtained from the structure position table 15 may be changed. Alternatively, the structure detection area may be calculated based on the distance from the intersection using the data of one structure detection area as a variable, and the structure detection area may be changed depending on the distance from the intersection. By switching the structure detection area according to the position or distance in this way, the range to be scanned for color detection can be reduced, and the processing load can be reduced.

In this example, the structure detection areas of traffic lights, road signs, and crosswalks are recorded because structures of the same type have similar colors and the pattern of installation locations is constant. A structure detection area for objects (eg, road markings (road surface paint), railroad crossings) may be defined.

3, 4, and 5 are diagrams showing configuration examples of the structure color component table 17 according to the embodiment of the present invention, in which a plurality of colors are determined depending on the combination of the environment and the structure. The structure color component table 17 shown in the figure is composed of a plurality of tables divided by weather and day/night as environments, and FIG. 3 shows the color of structures during the daytime on a clear day, FIG. 4 shows the color of structures during the daytime on a clear day, and FIG. represents. Colors may be defined in other formats as long as colors can be determined according to the combination of environment and structures.

The color of the structure is expressed by the chromaticity of red, green, and blue in the RGB color space, but other color spaces (for example, CMYK) may be used.

In each environment, representative colors included in structures are defined. For example, most stop signs are placed at intersections and are red with white letters. For this reason, the background color red is defined. Furthermore, the background color of a sign prohibiting traveling in a designated direction may be defined as blue, or the background color of a sign indicating the shape of an intersection may be defined as yellow.

For traffic lights, it is preferable to define one or both of the colors when the light is blue and the color when the light is red. Note that yellow lighting may be defined, although the lighting time is short. Furthermore, the color of the signal when the light is off (blue, yellow, red) may be defined.

At crosswalks, it is a good idea to define the white color of the paint. In the RGB space, if the chromaticity of each RGB color is equal, the color ranges from white to gray or black depending on the lightness.

6, 7, and 8 are flowcharts of camera abnormality detection processing.

First, the calculation device 30 executes a camera abnormality detection method selection process. Specifically, the image processing device 1 determines whether map information can be acquired (S601). For example, if the vehicle is equipped with an automatic driving function, detailed map information can be obtained. Additionally, if the vehicle is equipped with a navigation system, navigation map information can be acquired. Furthermore, if map information is provided from the map ECU via CAN, the image processing device 1 can acquire the map information via CAN.

If the map information can be acquired, the process proceeds to camera abnormality detection processing based on the map information (FIG. 7). On the other hand, if map information cannot be acquired, the process proceeds to camera abnormality detection processing (FIG. 8) based on own vehicle information.

In the camera abnormality detection process from map information, first, the image acquisition unit 11 acquires image data captured by the vehicle-mounted camera 51 (S701). The image acquisition unit 11 may acquire an image every frame (for example, 30 fps), or may acquire one image every several frames (for example, every 0.5 seconds). The number of images acquired per unit time may be changed depending on the traveling speed. For example, the time interval at which images are acquired is lengthened when the vehicle is traveling at a low speed, and the time interval at which images are acquired is shortened when the vehicle is traveling at a high speed. By changing the image acquisition interval depending on the speed, the structure can be captured in multiple still images regardless of the speed, and the color of the structure can be determined reliably.

Then, the image acquisition unit 11 executes image correction processing (S702). Specifically, the shape of the image is shaped. The image acquired by the on-vehicle camera 51 may be distorted, and the image is shaped by linear transformation or projective transformation. In addition, the brightness of the image is adjusted by exposure adjustment to facilitate color detection.

After that, the map information acquisition unit 13 acquires map information. The map information acquired by the map information acquisition unit 13 includes at least the position of the intersection. Then, the map information acquisition unit 13 extracts information on traffic lights, road signs, crosswalks, and road markings (paint) included in the acquired map information (S703). When detailed map information is obtained, the locations of structures such as traffic lights, road signs, crosswalks, and road markings (paints) can be found. On the other hand, since navigation map information does not include the positions of structures, it is possible to know areas where structures such as traffic lights, road signs, crosswalks, and road markings are present near intersections.

After that, the map information acquisition unit 13 determines whether there is a sign or the like near the own vehicle from the acquired map information (S704). When detailed map information is acquired, the presence or absence of a structure can be determined depending on whether information on the structure can be extracted from the detailed map information. On the other hand, when navigation map information is acquired, if the distance to the intersection is closer than a predetermined threshold, it means that the intersection is approaching, and generally there are structures such as traffic lights, road signs, crosswalks, road markings, etc. near the intersection. It can be determined that an object is near the vehicle.

In this embodiment, the processing by the image acquisition unit 11 and the processing by the map information acquisition unit 13 are executed in this order, but either one may be executed first or they may be executed in parallel.

If it is determined in step S704 that there is no structure near the host vehicle (No in S705), the process returns to step S701 and the next image data is acquired.

On the other hand, if it is determined in step S704 that there is a structure near the own vehicle (Yes in S705), the structure color identification unit 16 refers to the data corresponding to the current environment in the structure color component table 17, and The color of the structure determined to be near the vehicle is specified (S706). Note that for traffic lights, since the lighting color is not known, all the colors when the lights are on (and the colors when the lights are off, if registered) are specified from the structure color component table 17. Further, if the type of installed sign cannot be specified and the color of the sign to be recognized cannot be specified, all possible colors may be specified from the structure color component table 17.

Thereafter, the image area specifying unit 14 refers to the structure position table 15 and specifies a structure detection area in the image where the structure determined to be near the host vehicle is estimated to exist (S707).

In this embodiment, the structure color identification process by the structure color identification unit 16 and the image area identification process by the image area identification unit 14 are executed in this order. good.

After that, the camera abnormality detection unit 18 determines whether pixels of the color component of the structure determined to be near the own vehicle exist within the structure detection area (S708). Specifically, the pixels within the specified structure detection area are scanned, and it is determined whether a predetermined number of consecutive pixels that satisfy the color condition of the structure specified in step S706 exist.

When determining whether color components of a plurality of structures exist, it is preferable to determine the color components in order of ease of color recognition. For example, it is easy to recognize the color of a light-emitting traffic light, so it is better to recognize the color first. Next, it is best to recognize white paint on the road (for example, crosswalks), and then signs.

When the color of the structure is detected in the structure detection area, the continuous counter is initialized to 0, the abnormal area notification table 20 is cleared (S709), the process returns to step S701, and the next image data is get. On the other hand, if the color of the structure is not detected in the structure detection area, the continuous counter is incremented by 1 (S710), and it is determined whether the continuous counter is equal to or greater than a predetermined number (S711). As a result, if the continuous counter does not exceed the predetermined threshold value, the process returns to step S701 and acquires the next image data without determining that there is an abnormality.

On the other hand, if the continuous counter exceeds the predetermined threshold, the camera abnormality detection unit 18 determines that the camera is abnormal because the structure that should exist is not shown in the image. Then, the abnormal area notification determination unit 19 refers to the abnormal area notification table 20 and determines whether the determined abnormality has been notified (S712).

As a result, if the determined abnormality is registered in the abnormal area notification table 20, the dirt abnormality of the camera has already been notified (Yes in S713), and the process returns to step S701 without further notification of the abnormality. Obtain the image data of.

On the other hand, if the determined abnormality is not registered in the abnormal area notification table 20, the dirt abnormality of the vehicle-mounted camera 51 has not been notified (No in S713), and it is determined that a new camera dirt abnormality has been detected. Then, the abnormality notification unit 21 notifies the occupant and other ECUs of the camera dirt abnormality (S714). The passenger sees the camera dirt abnormality warning displayed on the instrument panel and removes dirt from the front of the camera (for example, the lens). Furthermore, upon receiving the camera dirt abnormality notification, the ECU determines that the reliability of the video from the vehicle-mounted camera 51 is low, and performs vehicle control processing without using the video from the vehicle-mounted camera 51 until the dirt is removed. Execute. For example, the ECU may stop automatic driving (AD) control using the video of the vehicle-mounted camera 51 and notify the occupant of the automatic driving stop. In addition, upon detection of camera dirt abnormality, the ECU stops the system including the vehicle-mounted camera 51 where dirt has been detected, stops automatic driving control due to the reduction in redundancy, and activates advanced driving assistance (ADAS). You may switch to

In the above explanation, it is determined whether the color component of the structure exists with the image as one area, but the image is divided into multiple areas (for example, 16 areas (4 vertically x 4 horizontally)) and each area is It may also be determined whether a color component of the structure exists. In this case, a continuous counter and an abnormal area notification table 20 are provided for each area, and in step S708, it is determined for each divided area whether pixels of the color component of the structure exist within the structure detection area, In steps S709 and S710, the continuous counter and abnormal area notification table 20 are updated for each area. By determining whether the color component of the structure is present in each region of the image divided in this way, it is possible to identify not only the presence of foreign matter on the front surface of the in-vehicle camera 51 but also the region to which the foreign matter is attached. .

Next, camera abnormality detection processing based on own vehicle information that is executed when map information cannot be acquired will be described. In the camera abnormality detection process from own vehicle information (Fig. 8), the same processing steps as the above-mentioned camera abnormality detection process from map information (Fig. 7) are given the same reference numerals, and detailed explanations of these steps are omitted. do.

In the camera abnormality detection process based on own vehicle information, the image acquisition unit 11 first acquires image data captured by the vehicle-mounted camera 51 (S701), and executes image correction processing (S702).

After that, the own vehicle information acquisition unit 12 acquires the own vehicle information 41 (S803). The own vehicle information 41 acquired by the own vehicle information acquisition unit 12 includes information on the behavior of the vehicle, such as speed, acceleration, steering angle, and operation amount of the occupant (brake operation amount, accelerator operation amount).

Then, based on the acquired own vehicle information 41, the own vehicle information acquisition unit 12 estimates whether there is a vehicle running ahead of the own vehicle or whether there is a traffic light near the own vehicle (S804). Specifically, if the vehicle repeatedly runs at low speed and stops, it can be assumed that there is a traffic jam and there is a vehicle in front. Additionally, if the vehicle decelerates from a constant speed to a stop, it can be assumed that the vehicle is approaching an intersection with a traffic light.

Thereafter, the structure color specifying unit 16 refers to the data corresponding to the current environment in the structure color component table 17 and specifies the color of the structure determined to be near the own vehicle (S706). Thereafter, the image area specifying unit 14 refers to the structure position table 15 and specifies a structure detection area in the image where the structure determined to be near the host vehicle is estimated to exist (S707). For example, when it is estimated that there is a vehicle in front, the area of the vehicle in front is set as a structure detection area, and the red color of the tail lamp within the structure detection area is detected. Furthermore, when it is estimated that the vehicle is approaching an intersection, the color of each structure is detected within the structure detection area of a traffic light, road sign, or crosswalk, as described above.

Thereafter, the camera abnormality detection unit 18 determines whether pixels of the color component of the structure determined to be near the own vehicle exist within the structure detection area (S708). When the color of the structure is detected in the structure detection area, the continuous counter is initialized to 0, the abnormal area notification table 20 is cleared (S709), and the process returns to step S701 to obtain the next image data. . On the other hand, if the color of the structure is not detected in the structure detection area, the continuous counter is incremented by 1 (S710), and it is determined whether the continuous counter is equal to or greater than a predetermined number (S711). As a result, if the continuous counter does not exceed the predetermined threshold value, the process returns to step S701 and acquires the next image data without determining that there is an abnormality.

On the other hand, if the continuous counter exceeds the predetermined threshold value, the predetermined color component cannot be detected continuously, so the abnormal area notification determination unit 19 refers to the abnormal area notification table 20 and determines the determined abnormality. It is determined whether the notification has been completed (S712).

As a result, if the detected color component is registered in the abnormal area notification table 20, the dirt abnormality of the camera has been notified (Yes in S713), and the process returns to step S701 without further notification of the abnormality. Get the next image data.

On the other hand, if the detected color component is not registered in the abnormal area notification table 20, it means that the dirt abnormality of the camera has not been notified (No in S713), and the abnormal area notification determination unit 19 newly detects the camera dirt. It is determined that an abnormality has been detected, and the abnormality notification unit 21 notifies the occupant and other ECUs of the camera dirt abnormality (S714).

As described above, the image may be divided into a plurality of regions (for example, 16 regions of 4 in the vertical direction and 4 in the horizontal direction), and it may be determined whether the color component of the structure exists in each region.

As described above, the image processing device 1 according to the embodiment of the present invention includes the image acquisition unit 11 that acquires images from the vehicle-mounted camera 51, and the image processing device 1 that acquires images from the vehicle-mounted camera 51. , a structure color specifying unit 16 that specifies color information of the structure; an image area specifying unit 14 that specifies a region in an image where a structure is estimated to exist based on the type of structure; and a vehicle-mounted camera. If there is a structure around the host vehicle, the camera abnormality detection unit 18 uses the color information obtained from the structure color identification unit 16 and the color information in the image. Since an abnormality in the vehicle-mounted camera 51 is detected by comparing the color components of an area in which a structure is estimated to be present, an abnormality in the vehicle-mounted camera 51 can be detected with a low processing load. That is, since the presence or absence of a color component of a structure is determined, the processing load can be reduced by reducing the amount of information used for determination, and detection can be facilitated. Additionally, detection of deposits on the front surface of the camera (lens), which can degrade performance, can be easily implemented in systems that do not require advanced processing performance. Further, by using objects having a common color, such as traffic lights and signs, as structures, it becomes easy to set the color conditions of the structures based on the types of structures.

In addition, since the image area specifying unit 14 acquires information on structures estimated to exist around the own vehicle from the map information, the image area specifying unit 14 acquires information on structures estimated to exist around the own vehicle from the map information. It is possible to accurately identify the area in which a structure is estimated to exist. Furthermore, based on the positions of intersections included in the navigation map, it is possible to accurately identify areas in the image where structures such as traffic lights, road signs, and crosswalks are estimated to exist. For this reason, it can be easily determined whether the color of the structure to be captured in the image exists.

In addition, since the own vehicle information acquisition unit 12 estimates information on structures that are estimated to exist around the own vehicle from the state of the own vehicle, the vehicle information acquisition unit 12 uses the vehicle in front as the structure and can be installed in the vehicle with a low processing load. Abnormalities in the camera 51 can be detected. Furthermore, even if map information cannot be obtained, approaching an intersection can be detected.

Further, since the own vehicle information acquisition unit 12 estimates the state of the vehicle from the vehicle speed, it is possible to detect the presence or absence of a vehicle in front and the approach to an intersection even if map information cannot be acquired.

Furthermore, since the structure position table 15 holds areas where structures are estimated to exist in accordance with the types of structures, the image area specifying unit 14 stores areas where structures are estimated to exist according to the types. You can easily obtain the area that you want to use.

In addition, the structure color identification unit 16 detects the color information obtained from the structure color identification unit 16 consecutively a predetermined number of times when the in-vehicle camera 51 abnormalities are detected. That is, since it is determined that there is dirt on the lens in the area where a predetermined color cannot be detected continuously, which obstructs the photographing of the outside world, temporary non-detection can be eliminated and an abnormality in the vehicle-mounted camera 51 can be reliably detected.

In addition, the abnormal area notification determination unit 19 notifies the passenger and/or other ECUs when the camera abnormality detected by the camera abnormality detection unit 18 is a visual notification. can encourage the removal of The ECU can perform vehicle control processing without using the video from the camera to ensure safety.

Further, when a plurality of structures exist, by having a plurality of continuous counters associated with each structure and its structure detection area, abnormality detection can be determined in more areas at the same time.

Note that the present invention is not limited to the embodiments described above, and includes various modifications and equivalent configurations within the scope of the appended claims. For example, the embodiments described above have been described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described. Further, a part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Further, the configuration of one embodiment may be added to the configuration of another embodiment. Further, other configurations may be added, deleted, or replaced with a part of the configuration of each embodiment.

Further, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example by designing an integrated circuit, and a processor realizes each function. It may also be realized by software by interpreting and executing a program.

Information such as programs, tables, files, etc. that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

Furthermore, the control lines and information lines shown are those considered necessary for explanation, and do not necessarily show all the control lines and information lines necessary for implementation. In reality, almost all configurations can be considered interconnected.

Claims (11)

An image processing device that processes images taken by an in-vehicle camera,
an image acquisition unit that acquires an image from the in-vehicle camera;
a structure color identification unit that identifies color information of the structure based on the type of structure estimated to exist around the host vehicle;
an image area identifying unit that identifies an area in the image where the structure is estimated to exist, based on the type of the structure;
and a camera abnormality detection unit that detects an abnormality in the vehicle-mounted camera,
When the structure exists around the own vehicle, the camera abnormality detection unit uses the color information obtained from the structure color identification unit and the area in the image where the structure is estimated to exist. An image processing device that detects an abnormality in the vehicle-mounted camera by comparing color components. The image processing device according to claim 1,
The image processing device is characterized in that the image area specifying unit acquires information on structures estimated to exist around the own vehicle from map information. The image processing device according to claim 1,
An image processing device comprising a vehicle information acquisition unit that estimates information on structures estimated to exist around the vehicle based on the state of the vehicle. The image processing device according to claim 3,
The image processing device is characterized in that the own vehicle information acquisition unit estimates the state of the vehicle from the vehicle speed. The image processing device according to claim 1,
holding structure position information indicating an area where the structure is estimated to exist, corresponding to the type of structure;
The image processing device is characterized in that the image area specifying unit refers to the structure position information and specifies an area in the image where the structure is estimated to exist. An image processing method in which an electronic control device processes images taken by an on-vehicle camera,
The electronic control device includes a calculation device that executes predetermined calculation processing, and a storage device that stores data necessary for the calculation processing,
The image processing method includes:
an image acquisition procedure in which the arithmetic unit acquires an image from the in-vehicle camera;
a structure color identification procedure in which the calculation device identifies color information of the structure based on the type of structure estimated to exist around the own vehicle;
an image region specifying step in which the arithmetic device specifies a region in the image where the structure is estimated to exist based on the type of the structure;
The computing device includes a camera abnormality detection procedure for detecting an abnormality in the vehicle-mounted camera,
In the camera abnormality detection procedure, the calculation device uses the color information obtained in the structure color identification procedure when the structure is present around the own vehicle, and the color information obtained in the structure color identification procedure when the structure is present in the image. An image processing method characterized in that an abnormality in the vehicle-mounted camera is detected by comparing color components of an estimated region. The image processing method according to claim 6,
The image processing method is characterized in that, in the image region specifying step, the arithmetic unit acquires information on structures estimated to exist around the host vehicle from map information. The image processing method according to claim 6,
An image processing method characterized in that the arithmetic unit includes a vehicle information acquisition procedure for estimating information on structures estimated to exist around the vehicle from the state of the vehicle. The image processing method according to claim 8,
The image processing method is characterized in that, in the self-vehicle information acquisition procedure, the arithmetic unit estimates the state of the vehicle from the vehicle speed. The image processing method according to claim 6,
The storage device stores structure position information indicating an area where a structure is estimated to exist, corresponding to the type of structure,
The image processing method is characterized in that, in the image region specifying step, the arithmetic device refers to the structure position information and specifies a region in the image where the structure is estimated to exist. An in-vehicle electronic control device,
an arithmetic device that performs predetermined arithmetic processing;
and a storage device for storing data necessary for the arithmetic processing,
an image acquisition unit in which the arithmetic unit acquires an image from an in-vehicle camera ;
a structure color identification unit in which the arithmetic unit identifies color information of the structure based on the type of structure estimated to exist around the host vehicle;
an image area specifying unit in which the arithmetic unit specifies an area in the image where the structure is estimated to exist based on the type of the structure;
The arithmetic device has a camera abnormality detection unit that detects an abnormality in the vehicle-mounted camera,
When the structure exists around the own vehicle, the camera abnormality detection unit uses the color information obtained from the structure color identification unit and the area in the image where the structure is estimated to exist. An on-vehicle electronic control device that detects an abnormality in the on-vehicle camera by comparing color components.

Images