CN114841910A

CN114841910A - Vehicle-mounted lens shielding identification method and device

Info

Publication number: CN114841910A
Application number: CN202110139589.3A
Authority: CN
Inventors: 万鹏; 任建乐
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-01-30
Filing date: 2021-01-30
Publication date: 2022-08-02

Abstract

The application provides a method and a device for identifying shielding of a vehicle-mounted lens, relates to the field of auxiliary driving or unmanned driving, and is used for rapidly and efficiently judging whether the vehicle-mounted lens is shielded. The method comprises the following steps: and acquiring a frame difference image, and judging whether the vehicle-mounted lens is shielded or not according to the frame difference image. The frame difference image is an image obtained by performing difference operation on a first image and a second image shot by the vehicle-mounted lens; the frame interval between the first image and the second image is N, and N is a positive integer; neither the first image nor the second image is a background image.

Description

Vehicle-mounted lens shielding identification method and device

Technical Field

The application relates to the technical field of automatic driving (including auxiliary driving and unmanned driving), in particular to a vehicle-mounted lens shielding identification method and device.

Background

Automatic driving (including assisted driving and unmanned driving) is an important direction for the development of intelligent automobiles, and more vehicles are beginning to apply an automatic driving system to realize the automatic driving function of the vehicle. Generally, as a component of an automatic driving system, an onboard lens may be used to capture an image of the surrounding environment of the vehicle, so that the automatic driving system can formulate a corresponding driving strategy according to the captured image, and further control the vehicle to realize automatic driving. It can be seen that the performance of the onboard lens is critical for automatic driving.

However, in some scenarios, the onboard lens is likely to be shielded by foreign objects such as leaves and plastic bags, which may cause the onboard lens to fail to work normally, and thus, the implementation of the automatic driving function is likely to be affected. For example, in fig. 1A, 1B, or 1C, when the vehicle-mounted lens is blocked, the image collected by the vehicle-mounted lens appears blurred, which affects normal driving. In consideration of driving safety and other factors, as shown in fig. 2A, it may be detected whether the vehicle-mounted lens is shielded, once the vehicle-mounted lens is shielded, an alarm mechanism is started, and at this time, an automatic driving function (for example, an Advanced Driver Assistance System (ADAS) function) may be stopped), and driving is controlled by a driver (i.e., manual intervention) instead, so as to reduce driving safety hazards.

Several lens occlusion detection schemes are provided in the prior art, including those based on depth information, motion information, image statistics. These lens occlusion detection schemes are generally high in computational complexity and are not suitable for real-time occlusion detection of a vehicle-mounted lens in a vehicle-mounted scene.

Disclosure of Invention

The application provides a vehicle-mounted lens shielding identification method and device, which improve the accuracy of determining road geometry, reduce the influence of non-road factors and better assist a vehicle in determining a driving strategy.

In order to achieve the purpose, the technical scheme is as follows:

in a first aspect, an embodiment of the present application provides an on-board lens occlusion recognition method, which is applied to a device having an automatic driving (including driving assistance) function, such as a vehicle, a chip system in the vehicle, and an operating system and a driver running on a processor, and the method includes: and acquiring a frame difference image, and judging whether the vehicle-mounted lens is shielded or not according to the frame difference image.

The frame difference image is an image obtained by performing difference operation on a first image and a second image shot by the vehicle-mounted lens; the frame interval between the first image and the second image is N, and N is a positive integer; neither the first image nor the second image is a background image.

In one possible design, the frame difference image is one or more frame difference images, the plurality of frame difference images including a first frame difference image and a second frame difference image; the first frame difference image is an image obtained by performing difference operation on a first image shot by the vehicle-mounted lens at a first moment and a second image before the first image; the second frame difference image is an image obtained by performing difference operation on a first image shot by the vehicle-mounted lens at a second moment and a second image before the first image, and the first moment is different from the second moment.

In one possible design, the method further includes: acquiring a first confidence coefficient and a first area corresponding to one or more frame difference images;

the confidence corresponding to one frame difference image is used for representing the credibility of a first region in a first image corresponding to the frame difference image, wherein the confidence is obtained according to the LBP response score of the local binary pattern similar to the first region, the Laplace variance and the Laplace mean value; the first area corresponding to the frame difference image is the area of a first region in the first image corresponding to the frame difference image;

the first area is an area obtained by removing a preset area from the second area; the second region is included in the first image, and the position of the second region in the first image is the same as the position of a third region in a frame difference image corresponding to the first image; the third area is an area formed by pixel points of which the pixel values are smaller than or equal to a first threshold value in the frame difference image; the preset area comprises any one or more of the following: an imaging area of a road surface, an imaging area of snow, an imaging area of sky, an imaging area of grass.

In one possible design, the frame difference images are M; judging whether the vehicle-mounted lens is shielded or not according to the frame difference image, wherein the judging step comprises the following steps:

if P frame difference images in the M frame difference images meet a first condition, determining that the vehicle-mounted lens is shielded;

wherein the first condition is that the confidence is greater than or equal to a first confidence threshold and the first area is greater than or equal to a first area threshold; m, P is a positive integer, M is greater than or equal to P.

if the continuous Q frame difference images in the M frame difference images meet a second condition, determining that the vehicle-mounted lens is shielded;

the second condition is that the confidence is greater than or equal to a second confidence threshold, and the first area is greater than or equal to a second area threshold; m, Q is a positive integer, M is greater than or equal to Q.

In one possible design, the obtaining a frame difference image includes:

acquiring a vehicle speed and/or a first identification position; the first identification bit is used for identifying whether an image processing result of the image signal processing ISP is normal or not;

and if the image processing result of the ISP identified by the first identification bit is normal and/or the vehicle speed is greater than or equal to a vehicle speed threshold value, acquiring the frame difference image.

In a second aspect, the present application provides an on-vehicle camera lens shelters from recognition device, and the device includes:

the acquisition module is used for acquiring a frame difference image; the frame difference image is an image obtained by performing difference operation on a first image and a second image shot by the vehicle-mounted lens; the frame interval between the first image and the second image is N, and N is a positive integer; neither the first image nor the second image is a background image;

and the judging module is used for judging whether the vehicle-mounted lens is shielded or not according to the frame difference image.

In a possible design, the obtaining module is further configured to obtain a first confidence and a first area corresponding to one or more frame difference images;

the first area is an area obtained by removing a preset area from the second area; the second region is included in the first image, and the position of the second region in the first image is the same as the position of a third region in a frame difference image corresponding to the first image; the third area is an area formed by pixel points with pixel values smaller than or equal to a first threshold value in the frame difference image; the preset area comprises any one or more of the following parts: an imaging area of a road surface, an imaging area of a snow area, an imaging area of a sky area, an imaging area of a grass area.

In one possible design, the frame difference images are M; the judging module is used for judging whether the vehicle-mounted lens is shielded according to the frame difference image, and comprises: the method comprises the steps of determining that the vehicle-mounted lens is shielded if P frame difference images in the M frame difference images meet a first condition;

In one possible design, the frame difference images are M; the judging module is used for judging whether the vehicle-mounted lens is shielded according to the frame difference image, and comprises: the on-board lens is determined to be shielded if the continuous Q frame difference images in the M frame difference images meet a second condition;

In one possible design, the obtaining module is configured to obtain a frame difference image, and includes: the method comprises the steps of obtaining a vehicle speed and/or a first identification position; the first identification bit is used for identifying whether an image processing result of the image signal processing ISP is normal or not; and if the image processing result of the ISP identified by the first identification bit is normal and/or the vehicle speed is greater than or equal to a vehicle speed threshold value, acquiring the frame difference image.

In a third aspect, an on-vehicle lens occlusion recognition device is provided, including: a processor and a memory; the memory is used for storing computer execution instructions, and when the on-vehicle lens occlusion recognition device runs, the processor executes the computer execution instructions stored in the memory, so that the on-vehicle lens occlusion recognition device executes the on-vehicle lens occlusion recognition method according to any one of the first aspect and the first aspect.

In a fourth aspect, an on-vehicle lens shielding and recognizing device is provided, which includes: a processor; the processor is coupled with the memory, and is configured to execute the on-vehicle lens occlusion recognition method according to any one of the first aspect and the first aspect, after reading the instruction in the memory.

In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, which includes instructions that, when executed on a computer, cause the computer to execute the in-vehicle lens occlusion recognition method according to any one of the first aspect and the first aspect.

In a sixth aspect, the present application further provides a computer program product, which includes instructions that, when executed on a computer, cause the computer to execute the in-vehicle lens occlusion recognition method according to any one of the first aspect and the first aspect.

In a seventh aspect, an embodiment of the present application provides a vehicle-mounted lens occlusion identification device, where the device may be a chip system, where the chip system includes a processor and may further include a memory, and is configured to implement the functions of the foregoing method. The chip system may be formed by a chip, and may also include a chip and other discrete devices.

In an eighth aspect, an in-vehicle lens occlusion recognition apparatus is provided, which may be a circuit system including a processing circuit configured to execute the in-vehicle lens occlusion recognition method according to any one of the first aspect and the first aspect.

Drawings

1A-1C are exemplary diagrams of an on-board lens being occluded;

fig. 2A and 2B are schematic flow charts of a conventional shot recognition method;

FIG. 3 is a first schematic structural diagram of an autonomous vehicle according to an embodiment of the present disclosure;

FIG. 4 is a schematic structural diagram of an autonomous vehicle according to an embodiment of the present disclosure;

FIG. 5 is a block diagram of a computer system according to an embodiment of the present disclosure;

fig. 6 is a schematic application diagram of a cloud-side command autonomous driving vehicle according to an embodiment of the present disclosure;

FIG. 7 is a block diagram of a computer program product according to an embodiment of the present application;

FIG. 8, FIG. 9-1, and FIG. 9-2 are schematic flow charts of a vehicle-mounted lens occlusion recognition method according to an embodiment of the present application;

9-3 are schematic diagrams of a rules control module provided by an embodiment of the present application;

10-21 are scene diagrams of a vehicle-mounted lens occlusion recognition method according to an embodiment of the present application;

fig. 22 and 23 are schematic structural diagrams of the vehicle-mounted lens occlusion recognition device according to the embodiment of the application.

Detailed Description

For the sake of understanding, the related terms referred to in the embodiments of the present application are explained as follows:

automatic driving: the automatic driving technology is a technology which depends on the cooperative cooperation of artificial intelligence, visual calculation, radar, a monitoring device and a global positioning system, so that a computer can automatically and safely operate a motor vehicle without any active operation of human beings. According to the classification standard of the Society of Automotive Engineers (SAE), the automatic driving technique is classified into: no automation (L0), driving support (L1), partial automation (L2), conditional automation (L3), highly automated (L4) and fully automated (L5).

The prior art includes several lens occlusion detection schemes based on depth information, motion information, and image statistics. The scheme based on the depth information is mainly used for security scenes, and other sensors except binocular lenses or single lenses are required to acquire the depth information, so that the requirements on the types of the lenses or the sensors are high. The lens shielding identification scheme based on the motion information needs to calculate the motion information of an object in an image, and the calculation complexity is high. The scheme based on image statistical information is the current mainstream scheme, and the method is more suitable for scenes such as security protection, video monitoring and the like. Referring to fig. 2B, this can be implemented as the following steps: first, a suspected occlusion region is extracted from the image by background modeling. Then, calculating statistical information of the suspected occlusion area, such as brightness change, edge information, a gray histogram, and the like, and judging whether the lens is occluded according to the statistical information and a corresponding threshold.

Wherein, the background modeling refers to modeling the background and extracting the background in the image according to the established background model. In general, a portion of the screen whose position is relatively fixed is regarded as a background. As a possible implementation, the current frame may be partitioned, a feature of each block, such as a symmetric-symmetric Local binary pattern (CSLBP), is calculated, and clustering is performed according to the calculation result. For example, blocks of the same features are lined up as the background and blocks of other features are lined up as the foreground. Blocks of the CSLBP texture that meet certain conditions are considered as background. In this way, the background can be extracted from the current frame.

Then, inter-frame difference (frame difference for short) operation is carried out on the current frame image and the extracted background to obtain a frame difference image, and a suspected occlusion area is obtained according to the frame difference image. And the occlusion region refers to the image of the occlusion object on the image sensor through the lens when the lens is occluded by the occlusion object. The suspected occlusion area (which may also be referred to as a candidate occlusion area), that is, the preliminarily determined image area that may be the occlusion area, needs to be further determined so as to determine whether the suspected occlusion area is an image caused by an occlusion of the vehicular lens by an occlusion object. Optionally, after the suspected occlusion area is obtained, boundary retrieval may be performed, so as to obtain a more accurate suspected occlusion area.

It can be understood that in some scenes, the background may be unchanged or slightly changed, and therefore, by using the frame difference between the current frame and the background, whether an object that blocks the background exists can be determined, that is, whether an object that blocks the background exists can be determined. For example, when the frame difference result indicates that the background is different from the current frame by a large amount, it indicates that there may be an obstruction blocking the background.

After the suspected occlusion area is extracted, in order to determine whether the suspected occlusion area is a real occlusion area, specific information of the suspected occlusion area needs to be counted, such as luminance change, edge information, a gradient histogram, a gray histogram, and the like of the suspected occlusion area. Wherein the edge information can be obtained by an edge algorithm. And when the statistical information meets a preset condition, determining that the suspected occlusion area is a real occlusion area.

The occlusion detection method also needs to extract the background and perform background modeling, so that the calculation amount is large, the calculation is complex, and the real-time performance is poor.

In addition, in a moving scene, such as an in-vehicle scene, since the in-vehicle lens moves along with the movement of the vehicle, the captured background of the in-vehicle lens also changes continuously, and the frame difference between the continuously changing background and the current frame makes it difficult to detect whether there is an obstruction. For example, a background is extracted from a previous history frame, a frame difference is made between the background and a current frame, and a frame difference result indicates that the difference between the background and the current frame is large. As can be seen, even if occlusion is determined using background modeling in a moving scene, misdetermination or failure in determination may result, i.e., the accuracy of the suspected occlusion region extracted by background modeling is low.

In addition, the detection judgment result excessively depends on a blocking strategy and a clustering strategy. The clustering result has strong correlation with the blocking strategy and the clustering strategy, namely whether the result is accurate depends on the blocking strategy and the clustering strategy to a great extent. When the blocking strategy and the clustering strategy are selected improperly, the result is possibly inaccurate. Therefore, the target occlusion detection method has high calculation complexity and poor real-time performance, and is difficult to be applied to occlusion detection in a motion scene with high real-time requirement.

In contrast, in the method for recognizing occlusion of a vehicle-mounted lens provided by the embodiments of the present application, generally, under the condition that the vehicle-mounted lens is occluded, a large-area connected region with small changes in brightness and the like exists at the same position of several frames of images acquired by the vehicle-mounted lens within a period of time. For example, as shown in fig. 1C, the imaging area of the foreign object may exist in each of the images of multiple frames captured by the on-board lens. Based on the characteristics, in the embodiment of the application, the suspected occlusion area is directly determined according to the frame difference result of the current frame and the historical frame, operations such as background modeling and the like are not needed, the calculated amount is small, the calculation complexity is low, the calculation time can be shortened, and the calculation real-time performance is improved, so that the occlusion detection requirement in a motion scene can be met.

The vehicle-mounted lens shielding identification method provided by the embodiment of the application is applied to a motion scene. For example, in a motion scene of a mobile terminal such as a vehicle-mounted scene, an outdoor robot, an indoor and outdoor factory automatic navigation vehicle (AVG), and a mobile phone. When the part of the image sensor, which is shielded by the foreign matters, in the motion scene is larger than the total imaging area by a certain proportion and exists for a certain time, the shielding identification method can be used for timely judging and giving an alarm to make subsequent response.

The technical method can be applied to vehicles with automatic driving or auxiliary driving functions, or other devices (such as cloud servers) with automatic driving control functions. The vehicle can implement the method for identifying the occlusion of the vehicle-mounted lens provided by the embodiment of the application through the contained components (including hardware and software), and identify whether the vehicle-mounted lens is occluded or not. Or, other devices (such as a server) are used for implementing the method for identifying the occlusion of the onboard lens according to the embodiment of the present application, and identifying whether the onboard lens is occluded or not so as to formulate a corresponding driving strategy.

Fig. 3 is a functional block diagram of the vehicle 100 according to the embodiment of the present application. In one embodiment, the vehicle 100 is configured in a drive-assisted or fully autonomous mode. For example, the vehicle 100 may recognize whether the on-vehicle lens is occluded while in the assisted driving or full automatic driving mode, and formulate a driving strategy based on the recognition result, thereby controlling the vehicle 100 to perform automatic driving. When the vehicle 100 is in the automatic driving mode, the vehicle 100 does not interact with the driver, and autonomously completes the actions of obstacle avoidance, vehicle following, lane keeping, automatic parking and the like. When the vehicle 100 is in the auxiliary driving mode, the vehicle 100 prompts a driver according to a driving strategy, and the driver completes actions such as obstacle avoidance, vehicle following, lane keeping, automatic parking and the like according to the prompts.

Vehicle 100 may include various subsystems such as a travel system 110, a sensor system 120, a control system 130, one or more peripherals 140, as well as a power supply 150, a computer system 160, and a user interface 170. Alternatively, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple elements. In addition, each of the sub-systems and elements of the vehicle 100 may be interconnected by wire or wirelessly.

The travel system 110 may include components, such as an engine, transmission, etc., that provide power to the vehicle 110.

The sensor system 120 may include several sensors that sense information about the environment surrounding the vehicle 100. For example, the sensor system 120 may include at least one of a positioning system 121 (which may be a GPS system, a beidou system, or other positioning system), an Inertial Measurement Unit (IMU) 122, a radar sensor 123, a laser radar 124, a vision sensor 125, an ultrasonic sensor 126, and a sonar sensor 127. Optionally, the sensor system 120 may also include sensors of internal systems of the monitored vehicle 100 (e.g., an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors may be used to detect the object and its corresponding characteristics (position, shape, orientation, velocity, etc.). Such detection and identification is a critical function of the safe operation of the autonomous vehicle 100.

The positioning system 121 may be used to estimate the geographic location of the vehicle 100. The IMU 122 is used to sense position and orientation changes of the vehicle 100 based on inertial acceleration. In one embodiment, the IMU 122 may be a combination of an accelerometer and a gyroscope.

The radar sensor 123 may utilize electromagnetic wave signals to sense objects within the surrounding environment of the vehicle 100. In some embodiments, in addition to sensing the position of an object, the radar sensor 123 may also be used to sense the radial velocity of an object and/or the radar cross-sectional area RCS of the object.

The ultrasonic sensors 126 may utilize ultrasonic waves to sense objects within the surrounding environment of the vehicle 100. In some embodiments, in addition to sensing the position of an object, the ultrasonic sensor 126 may also be used to sense the radial velocity of an object and/or the echo amplitude of the object.

Sonar sensors 127 may utilize acoustic waves to sense objects within the surrounding environment of vehicle 100. In some embodiments, in addition to sensing the position of an object, sonar sensors 127 may also be used to sense the radial velocity of an object and/or the sonar target strength sonar TS of that object.

Lidar 124 may utilize a laser to sense objects in the environment in which vehicle 100 is located. In some embodiments, lidar 124 may include one or more laser sources, laser scanners, and one or more detectors, among other system components.

The vision sensor 125 may be used to capture multiple images of the surrounding environment of the vehicle 100. The vision sensor 125 may be a still camera or a video camera.

In the embodiment of the present application, the vision sensor 125 may be used for a vehicle-mounted lens, or a vehicle-mounted camera. It is likely to be shielded by foreign matter such as leaves, plastic bags, etc. Then the image it captures may have certain characteristics from which it can be determined that the vision sensor 125 is occluded.

The control system 130 may control the operation of the vehicle 100 and its components. Control system 130 may include various elements, such as at least one of a computer vision system 131, a route control system 132, and an obstacle avoidance system 133.

The computer vision system 131 may be operable to process and analyze images captured by the vision sensor 125 and measurement data obtained by the radar sensor 123 to identify objects and/or features in the environment surrounding the vehicle 100. The objects and/or features may include traffic signals, road boundaries, and obstacles. The computer vision system 131 may use object recognition algorithms, motion from motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 131 may be used to map an environment, track objects, estimate the speed of objects, and so forth. In an embodiment of the present application, the computer vision system 131 may be operable to process and analyze images captured by the onboard lens in order to identify features of the images.

The route control system 132 is used to determine a travel route for the vehicle 100. In some embodiments, the route control system 132 may combine data from the radar sensors 123, the positioning system 121, and one or more predetermined maps to determine a travel route for the vehicle 100.

Obstacle avoidance system 133 is used to identify, evaluate, and avoid or otherwise negotiate potential obstacles in the environment of vehicle 100.

Of course, in one example, the control system 130 may additionally or alternatively include components other than those shown and described. Or may reduce some of the components shown above.

The vehicle 100 may acquire the required information using the wireless communication system 140, wherein the wireless communication system 140 may wirelessly communicate with one or more devices directly or via a communication network. For example, the wireless communication system 140 may use 3G cellular communication, such as CDMA, EVD0, GSM/GPRS, or 4G cellular communication, such as LTE. Or 5G cellular communication. The wireless communication system 140 may communicate with a Wireless Local Area Network (WLAN) using WiFi. In some embodiments, the wireless communication system 140 may communicate directly with the devices using infrared links, bluetooth, or ZigBee. Other wireless protocols, such as various vehicle communication systems, for example, the wireless communication system 140 may include one or more Dedicated Short Range Communications (DSRC) devices.

Some or all of the functions of the vehicle 100 are controlled by the computer system 160. The computer system 160 may include at least one processor 161, the processor 161 executing instructions 1621 stored in a non-transitory computer readable medium, such as the data storage device 162. The computer system 160 may also be a plurality of computing devices that control individual components or subsystems of the vehicle 100 in a distributed manner. In the embodiment of the present application, the processor 161 may obtain multiple frames captured by the in-vehicle lens, and perform frame difference on the historical frame and the current frame, so as to determine the candidate occlusion area according to the frame difference result. Optionally, regions such as sky, road surface, snow and the like can be removed from the candidate occlusion region.

Processor 161 may be any conventional processor, such as a commercially available Central Processing Unit (CPU). Alternatively, the processor may be a dedicated device such as an Application Specific Integrated Circuit (ASIC) or other hardware-based processor. Although fig. 3 functionally illustrates a processor, memory, and other elements within the same physical housing, those skilled in the art will appreciate that the processor, computer system, or memory may actually comprise multiple processors, computer systems, or memories that may or may not be stored within the same physical housing. For example, the memory may be a hard drive, or other storage medium located in a different physical enclosure. Thus, references to a processor or computer system are to be understood as including references to a collection of processors or computer systems or memories that may or may not operate in parallel. Rather than using a single processor to perform the steps described herein, some components, such as the steering component and the retarding component, may each have their own processor that performs only computations related to the component-specific functions.

In an embodiment of the application, the processor includes an Image Signal Processor (ISP) for processing a signal output by the image sensor.

In various aspects described herein, the processor may be located remotely from the vehicle and in wireless communication with the vehicle. In other aspects, some of the processes described herein are executed on a processor disposed within the vehicle and others are executed by a remote processor, including taking the steps necessary to perform a single maneuver.

Optionally, the above components are only an example, in an actual application, components in the above modules may be added or deleted according to an actual need, and fig. 3 should not be construed as limiting the embodiment of the present application.

Alternatively, the autonomous vehicle 100 or a computing device associated with the autonomous vehicle 100 (e.g., the computer system 160, the computer vision system 131, the data storage 162 of fig. 3) may identify whether the onboard lens is occluded by a foreign object based on an image captured by the onboard lens. Alternatively, the vehicle 100 can adjust its driving strategy based on whether the onboard lens is occluded. In some examples, if the onboard lens is occluded by a foreign object, then the automatic driving may be stopped and the vehicle switched to the auxiliary driving based on driving safety considerations.

The vehicle 100 may be a car, a truck, a motorcycle, a bus, a boat, an airplane, a helicopter, a lawn mower, an amusement car, a playground vehicle, construction equipment, a trolley, a golf cart, a train, a trolley, etc., and the embodiment of the present invention is not particularly limited.

In other embodiments of the present application, the autonomous vehicle may further include a hardware structure and/or a software module, and the functions described above are implemented in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether any of the above-described functions is implemented as a hardware structure, a software module, or a hardware structure plus a software module depends upon the particular application and design constraints imposed on the technical solution.

Referring to FIG. 4, exemplary, the following modules may be included in a vehicle:

the environment sensing module 201 is configured to acquire measurement data information of a target object detected by a sensor. The sensors may be of various types, such as cameras, lidar, millimeter wave radar, ultrasonic sensors, sonar sensors, and the like. The context awareness module passes this data to the rules control module 202 to facilitate the rules control module 202 in generating action instructions.

The rule control module 202: the module is a conventional control module of an autonomous vehicle, and is configured to receive acquired image data from the environment sensing module 201, determine whether a foreign object blocks the on-board lens based on the image data, determine a specific position of the blocking, generate a corresponding driving policy and an action instruction corresponding to the driving policy, and send the action instruction to the vehicle control module 203, where the action instruction is used to instruct the vehicle control module 203 to control driving of the vehicle.

The vehicle control module 203: for receiving motion commands from the rules control module 202 to control the driving operation of the vehicle.

In-vehicle communication module 204 (not shown in fig. 4): the method is used for information interaction between the self vehicle and other vehicles.

A storage component 205 (not shown in fig. 4) for storing executable code of the above modules. Running these executable codes may implement some or all of the method flows of the embodiments of the present application.

In one possible implementation of the embodiments of the present application, as shown in fig. 5, the computer system 160 shown in fig. 3 includes a processor 301, and the processor 301 is coupled to a system bus 302. Processor 301 may be one or more processors, each of which may include one or more processor cores. A display adapter (video adapter)303, display adapter 303 may drive a display 309, display 309 coupled to system bus 302. System BUS 302 is coupled to an input/output (I/O) BUS (BUS)305 through a BUS bridge 304. I/O interface 306 and I/O bus 305. The I/O interface 306 communicates with various I/O devices, such as an input device 307 (e.g., keyboard, mouse, touch screen, etc.), a multimedia disk 308 (e.g., CD-ROM, multimedia interface, etc.). A transceiver 315 (which can send and/or receive radio communication signals), a camera 310 (which can capture still and moving digital video images), and an external Universal Serial Bus (USB) interface 311. Wherein, optionally, the interface connected with the I/O interface 306 may be a USB interface.

The processor 301 may be any conventional processor, including a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, or a combination thereof. Alternatively, the processor may be a dedicated device such as an Application Specific Integrated Circuit (ASIC). Alternatively, the processor 301 may be a neural network processor or a combination of a neural network processor and a conventional processor as described above.

Alternatively, in various embodiments described herein, the computer system 160 may be located remotely from the autonomous vehicle and may communicate wirelessly with the autonomous vehicle 100. In other aspects, some of the processes described herein may be provided for execution on a processor within an autonomous vehicle, with other processes being performed by a remote processor, including taking the actions necessary to perform a single maneuver.

Computer system 160 may communicate with a software deploying server (deploying server)313 via a network interface 312. The network interface 312 is a hardware network interface, such as a network card. The network (network)314 may be an external network, such as the internet, or an internal network, such as an ethernet or a Virtual Private Network (VPN). Optionally, the network 314 may also be a wireless network, such as a WiFi network, a cellular network, and the like.

In other embodiments of the present application, the method for recognizing occlusion of a vehicle-mounted lens according to the embodiments of the present application may also be executed by a chip system. The embodiment of the application provides a chip system. The main cpu (host cpu) and the neural Network Processor (NPU) cooperate together to realize a corresponding algorithm for a function required by the vehicle 100 in fig. 3, a corresponding algorithm for a function required by the vehicle shown in fig. 4, and a corresponding algorithm for a function required by the computer system 160 shown in fig. 3.

In other embodiments of the present application, computer system 160 may also receive information from, or transfer information to, other computer systems. Alternatively, sensor data collected from the sensor system 120 of the vehicle 100 may be transferred to another computer, where it is processed. Data from computer system 160 may be transmitted via a network to a computer system on the cloud side for further processing. The network and intermediate nodes may comprise various configurations and protocols, including the internet, world wide web, intranets, virtual private networks, wide area networks, local area networks, private networks using proprietary communication protocols of one or more companies, ethernet, WiFi, and HTTP, as well as various combinations of the foregoing. Such communication may be performed by any device capable of communicating data to and from other computers, such as modems and wireless interfaces.

Referring to fig. 6, an example of autonomous vehicle and cloud service center (cloud server) interaction is shown. The cloud service center may receive information (such as data collected by vehicle sensors or other information) from autonomous vehicles 413, 412 within its environment 400 via a network 411, such as a wireless communication network.

The cloud service center 420 runs the stored program for identifying the occlusion of the vehicle-mounted lens according to the received data, and identifies whether the vehicle-mounted lens of the automatic driving vehicles 413 and 412 is occluded. The relevant procedure for identifying whether the on-vehicle lens is blocked may be: and performing a frame difference program, a color segmentation program, a histogram back projection program, a template matching program or a program for calculating the occlusion confidence coefficient on the historical frame and the current frame.

For example, the cloud service center 420 may provide portions of a map to the vehicles 413, 412 via the network 411. In other examples, operations may be divided among different locations. For example, multiple cloud service centers may receive, validate, combine, and/or send information reports. Information reports and/or sensor data may also be sent between vehicles in some examples. Other configurations are also possible.

As shown in fig. 7, in some examples, signal bearing medium 501 may comprise a computer readable medium 503 such as, but not limited to, a hard disk drive, a Compact Disc (CD), a Digital Video Disc (DVD), a digital tape, a memory, a read-only memory (ROM), a Random Access Memory (RAM), or the like. In some implementations, the signal bearing medium 501 may comprise a computer recordable medium 504 such as, but not limited to, a memory, a read/write (R/W) CD, a R/W DVD, and the like. In some implementations, the signal bearing medium 501 may include a communication medium 505, such as, but not limited to, a digital and/or analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communications link, a wireless communication link, etc.). Thus, for example, the signal bearing medium 501 may be conveyed by a wireless form of communication medium 505 (e.g., a wireless communication medium conforming to the IEEE 802.11 standard or other transmission protocol). The one or more program instructions 502 may be, for example, computer-executable instructions or logic-implementing instructions. In some examples, a computing device such as described with respect to fig. 3-7 may be configured to provide various operations, functions, or actions in response to program instructions 502 communicated to the computing device by one or more of a computer-readable medium 503, and/or a computer-recordable medium 504, and/or a communications medium 505. It should be understood that the arrangements described herein are for illustrative purposes only. Thus, those skilled in the art will appreciate that other arrangements and other elements (e.g., machines, interfaces, functions, orders, and groupings of functions, etc.) can be used instead, and that some elements may be omitted altogether depending upon the desired results. In addition, many of the described elements are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, in any suitable combination and location.

The vehicle-mounted lens shielding identification method provided by the embodiment of the application can be applied to movable scenes. Optionally, the mobile scenario includes an in-vehicle scenario. The in-vehicle scenario includes an auto/semi-auto driving scenario. The method can identify whether the vehicle-mounted lens is occluded, for example, identify the occlusion shown in fig. 1A. The method may be performed by the apparatus of fig. 3-7 (e.g., the vehicle, a component such as a processor of the vehicle, a server). The following describes in detail a vehicle-mounted lens occlusion recognition method according to an embodiment of the present application with reference to each drawing.

The embodiment of the application provides a vehicle-mounted lens occlusion identification method, as shown in fig. 8, the method comprises the following steps:

s101, acquiring a frame difference image.

Neither the first image nor the second image is a background image. That means, in the embodiment of the present application, the background modeling operation does not need to be performed on the image captured by the onboard lens.

In some embodiments, performing the inter-frame difference operation on the current frame and the historical frame refers to performing the difference operation on the grayscale image of the current frame and the grayscale image of the historical frame, which is not limited to this, of course. If the inter-frame difference refers to a difference operation between the gray-scale images, the current frame and the historical frame need to be converted into the respective corresponding gray-scale images, and then the difference operation is performed on the gray-scale image of the current frame and the gray-scale image of the historical frame. The operation of converting the current frame from a color image to a grayscale image and converting the historical frame from a color image to a grayscale image can be referred to in the prior art.

The difference between the frames of the gray level image of the current frame and the gray level image of the historical frame can be implemented as follows: and (3) subtracting pixel points at corresponding positions (namely, the same coordinates) in the gray-scale image of the current frame and the gray-scale image of the historical frame. In this way, a frame difference image can be obtained. Of course, other inter-frame difference modes may also be used, and the specific mode of inter-frame difference in the embodiment of the present application is not limited.

It can be seen that the frame difference image may represent the difference between two frame images.

And S102, judging whether the vehicle-mounted lens is shielded or not according to the frame difference image.

It can be understood that when the vehicle-mounted lens is shielded by the foreign object, the foreign object is likely to shield the vehicle-mounted lens for a long time, which means that the position relationship between the foreign object and the vehicle-mounted lens is likely not to change or not to change greatly, and then the image formed by the foreign object through the vehicle-mounted lens is likely to change less in a long time. In this embodiment of the application, the frame difference image may represent a difference between the first image and the second image, and therefore, based on the frame difference image, it may be detected which pixel points at positions in the first image or the second image may be an image caused by blocking of a foreign object, that is, an image caused by blocking of a suspected foreign object. Specifically, an area formed by pixel points with pixel values smaller than or equal to a pixel threshold value in the frame difference image corresponds to imaging caused by foreign matter shielding. Therefore, whether the vehicle-mounted lens is indeed shielded or not can be further judged according to the imaging caused by the suspected foreign matter shielding.

According to the vehicle-mounted lens occlusion identification method, the first image and the second image are directly subjected to the regions with unchanged and small changes in characteristics such as frame difference statistical brightness, and therefore the suspected occlusion regions can be obtained quickly and efficiently without performing calculation such as background modeling and clustering. Meanwhile, compared with the method of extracting the candidate occlusion region in a background modeling mode, the method has the advantage that the accuracy of the extracted candidate occlusion region is higher.

The method of the examples of the present application is explained in detail as follows.

In some cases, a plurality of frame difference images may be acquired, and it may be determined whether the in-vehicle lens is occluded from the plurality of frame difference images. The plurality of frame difference images includes a first frame difference image and a second frame difference image; the first frame difference image is an image obtained by performing difference operation on a first image and a second image (X frames are arranged between the second image and the first image) shot by the vehicle-mounted lens at a first moment; the second frame difference image is an image obtained by performing difference operation on a first image shot by the vehicle-mounted lens at a first moment and a second image (Y frames are arranged between the second image and the first image) before the first image. As shown in fig. 9-1, the method provided by the embodiment of the present application includes:

s201, vehicle speed information Vcur and an ISP identification bit Flag _ ISP are obtained.

As a possible implementation, the vehicle speed information may be acquired by means of sensing data of one or more sensors. For example, the vehicle-mounted camera collects road information, obtains the real-time position of the vehicle, and can calculate the vehicle speed based on the position and time of the vehicle at different times.

And the ISP identification bit is used for representing whether ISP processing is normal or not. The ISP identification bit is true (true) or set (for example, set to 1), or the ISP identification bit is normal, which means that the ISP processes normally; otherwise, the ISP flag is false or not set (e.g. set to 0), or the ISP flag is abnormal, which indicates that the ISP processes the abnormality.

In general, after an ISP acquires an original image signal from a sensor, the original image signal is processed, and if the processing is normal, the ISP is identified as position 1, and if the processing is abnormal, the ISP is identified as position 0. The ISP identification bit may be stored, for example, in the processor 160, and when the ISP identification bit needs to be acquired, the ISP identification bit may be acquired from the processor 160 so as to know whether the ISP processing is normal. The ISP identification bits may be stored 160 within the processor.

ISP to handle exceptions includes, but is not limited to, overexposure, dullness, color distortion. ISP processing normally includes proper exposure, undistorted color, etc.

S202, judging whether the vehicle speed reaches a first threshold value and whether the ISP identification bit is true. If the vehicle speed reaches the first threshold and the ISP flag is true, step S203 is executed. Otherwise, the flow is ended.

The vehicle speed reaching the first threshold value means that the vehicle speed is greater than or equal to the first threshold value. The first threshold value can be flexibly set according to application scenarios.

Generally, the vehicle speed is relatively low, and when the vehicle speed does not reach the first threshold value, the driving safety hidden danger may not be prominent, or the requirement on the performance of the vehicle-mounted lens is not high. In such a scene, the scene is not called as a motion scene, and whether the vehicle-mounted lens is blocked or not does not need to be detected, so that the detection workload is reduced. Moreover, when the ISP flag is false, it often means that the image collected by the vehicle-mounted lens may have phenomena such as overexposure, excessive darkness, color distortion, and the like, that is, the collected image cannot accurately reflect the state of the object to be shot, and thus, since an accurate image cannot be collected, it cannot be accurately determined whether the vehicle-mounted lens is really blocked by a foreign object based on the image, or even if it is possible to determine whether the vehicle-mounted lens is blocked, the determination result is inaccurate. In this case, even if it is detected whether the on-vehicle lens is blocked, it is difficult to determine whether the on-vehicle lens is actually blocked according to inaccurate data, and it is not detected whether the on-vehicle lens is blocked. Optionally, in this case, the report is made to the ISP or the image acquisition display to give an abnormal alarm.

That is to say, in the embodiment of the present application, only when it is determined that the vehicle speed reaches the first threshold (which indicates that the potential safety hazard is relatively prominent, the performance requirement on the on-board lens is high, and the on-board lens conforms to the motion scene), and the ISP identification bit is true (which means that the image acquired by the on-board lens is relatively accurate), the subsequent steps are executed, so as to determine whether the on-board lens is blocked.

Through the judgment of whether the ISP is abnormal or not in the steps S201 and S202, abnormal scenes such as image overexposure, image darkness, color distortion and the like are eliminated, and the quality of the acquired image is ensured to meet the requirements.

S203, acquiring a current frame FrameCur and L-1 historical frames FrameHistory.

Wherein L is a positive integer, and L is greater than or equal to 2. The current frame is a picture of the current moment acquired by the vehicle-mounted lens. The historical frame is a picture of the vehicle-mounted lens at the previous moment.

As a possible implementation mode, the vehicle-mounted lens collects images in real time, the processor acquires the current frame image in real time, and meanwhile, the L-1 frame historical image frame information is stored to obtain L frame images. The L frames include L-1 history frames and the current frame.

The following steps S204 to S209 are performed for the current frame and each of the L-1 history frames (taking the ith history frame as an example):

s204, performing inter-frame difference operation on the ith history frame FrameHistory and the current frame FrameCur to obtain a frame difference image FrameSub corresponding to the ith history frame (the frame difference image corresponding to the ith history frame may be simply referred to as the ith frame difference image).

Wherein i is a positive integer, i is greater than or equal to 1 and less than or equal to L-1.

For a specific way of the inter-frame difference, see step S101 above.

Alternatively, as shown in fig. 9-3, the rule control module 202 includes a frame difference module, which inputs the current frame and the ith history frame into the frame difference module and outputs the ith frame difference image.

S205, thresholding is carried out on the ith frame difference image to obtain a binary image FrameBinary corresponding to the ith historical frame (the binary image corresponding to the ith historical frame can be simply called as the ith binary image).

It can be understood that the frame difference image represents the difference between the current frame and the historical frame, and then, the pixel points whose pixel values are greater than a certain threshold value in the frame difference image are the pixel points with large change between the current frame and the historical frame.

As a possible implementation, thresholding the frame difference image can be implemented as: a binarization threshold is set, and for a pixel in the frame difference image whose pixel value (or gray value) is greater than the binarization threshold, the pixel value (or gray value) of the pixel is set to a first pixel value, such as 0 (i.e. black). For a pixel in the frame difference image whose pixel value (or grayscale value) is less than or equal to the binarization threshold, the pixel value (or grayscale value) of the pixel is set to a second pixel value, such as 255 (i.e., white). That means, the pixel point with larger change in the frame difference image is set to be black, otherwise, the pixel point with smaller change in the frame difference image is set to be white, and a binary image is obtained. Illustratively, a frame difference operation is performed on a historical frame acquired by a vehicle-mounted lens and a current frame shown in fig. 1A to obtain a frame difference image, and the frame difference image is binarized to obtain a binary image shown in fig. 10.

The binary threshold value can be flexibly set according to the situation.

Of course, there may be other manners of binarizing the image, such as setting the pixel value smaller to 0, the pixel value larger to 255, or other possible manners. The embodiment of the present application does not limit the specific way of image binarization.

Optionally, as shown in fig. 9-3, the rule control module 202 includes a binarization sub-module, and inputs the ith frame difference image into the binarization sub-module, and may output the ith binary image, that is, the binary image corresponding to the ith frame difference image.

S206, detecting a target connected domain in the ith binary image.

First, the definition of the connected domain is given. Connected domain is a region composed of pixels having the same pixel value and located adjacently (or referred to as neighboring pixels).

In general, connected domains may differ under different adjacencies. The following mainly describes how the connected component is divided into four adjacent regions and eight adjacent regions.

As shown in (1) of fig. 11, the positional relationship between the pixel p and the four pixel points of the labels 1 to 4 (i.e., right above, right below, left side, and right side of the pixel p) is referred to as a four-neighbor relationship. In the four-adjacent relationship, some pixels are directly adjacent, such as pixel p and pixel 4. Some pixels are adjacent to each other by other intermediate pixels, such as pixel 2 and pixel 4 by pixel p.

As shown in (2) in fig. 11, the positional relationship between the pixel p and the eight pixel points 1 to 8 on the right below, left side, right side, and diagonal line is referred to as an eight-adjacent relationship. Similar to the four-neighbor relationship, in the eight-neighbor relationship, some pixels may have a neighbor relationship therebetween by other intermediate pixels.

As shown in (3) in fig. 11, in the case of the four-adjacent relationship, for the black pixel of the label 2, since the surrounding black pixels cannot form the four-adjacent relationship with themselves, the pixel is divided into a single connected domain by 2 times. Similarly, for the pixel point marked with the mark 1 indicated by the arrow and the pixel point marked with the mark 3 indicated by the arrow, the two pixel points cannot directly form a four-adjacent relationship, and cannot form the four-adjacent relationship with other intermediate pixels, so that the two pixel points are divided into two different connected domains.

As shown in (4) of fig. 11, in the case of eight neighbors, the black pixels can be in eight-neighbor relationship directly or via other intermediate pixels, and therefore, the black pixels are all divided into the same connected domain.

In the embodiment of the present application, the connected domain divided in the case of four neighbors may be referred to as a four-way connected domain. Namely, fig. 11 (3) shows that 3 four connected domains are included. The connected domain divided in the case of eight neighbors may be called an eight connected domain, i.e., 1 eight connected domain is included as shown in (4) in fig. 11.

As can be seen from fig. 11 (3) or fig. 11 (4), the number of pixels included in an eight-connected domain is generally greater than or equal to the number of pixels included in a four-connected domain. In some embodiments, eight connected regions in the binary image may be detected, and certainly, four connected regions in the binary image may also be detected, which is not limited in this application embodiment.

The location of the connected component can be expressed in coordinates. For example, the position of the connected component can be represented by the coordinates (x, y) of the top left pixel of the connected component. The size of the connected component can be represented by the length h, the width w or the number of pixel points.

And the target connected component refers to a connected component with the area larger than or equal to the threshold value of the connected component and the pixel value of the second pixel value. The pixel point with the second pixel value is a pixel point with small change between the current frame and the ith historical frame, namely the pixel point with the pixel value smaller than or equal to the first threshold value in the corresponding frame difference image.

It can be understood that when the vehicle-mounted lens is shielded by the foreign object, the foreign object is likely to shield the vehicle-mounted lens for a long time, which means that the position relationship between the foreign object and the vehicle-mounted lens is likely not to change or not to change greatly, and then the image formed by the foreign object through the vehicle-mounted lens is likely to change less in a long time. Therefore, in the embodiment of the present application, pixel points with small changes (including brightness changes and the like) between the current frame and the historical frame may be detected (as shown in fig. 10, pixel values of the pixel points are set as the second pixel values in the binary image), and if the pixel points are detected, it may be that the changes of the brightness and the like of the pixel points are small due to the shielding of foreign matters. In order to further eliminate whether the detection is false detection or not, whether the area of the connected domain formed by the pixels with smaller changes in brightness and the like is large enough (namely whether the connected domain forms the target connected domain or not) can be detected, and when the area of the connected domain reaches a certain threshold value, it is likely that the changes in brightness and the like of the pixels are smaller due to the shielding of foreign matters.

For example, assuming that, in a binary image, pixel points with small changes between a current frame and a history frame are set to be white, as shown in (1) in fig. 12, the area of a connected domain formed by the white pixel points is detected. If the area of the connected component 1 is greater than or equal to a certain threshold, the connected component 1 can be regarded as a target connected component. The area of the region 2 is small, and the region 2 is not set as a target connected component.

It should be noted that the number of the target connected component in the binary image corresponding to the ith history frame may be zero, one, or multiple. If the target connected domain does not exist in the binary image corresponding to the ith historical frame, operating the next historical frame of the current ith historical frame, namely the (i + 1) th historical frame, namely skipping to the step S204. If a target connected domain exists in the binary image corresponding to the ith historical frame, which indicates that the target connected domain is likely to be an image corresponding to the vehicle-mounted lens shielded by the shielding object, in order to further confirm whether the image is caused by the shielding object shielding the lens, the subsequent steps need to be continuously executed.

Optionally, as shown in fig. 9-3, the rule control module 202 includes a connected component extracting sub-module, and inputs the ith binary image into the connected component extracting sub-module, and may output a target connected component in the ith binary image.

In other words, if the coordinates of the target connected component in the binary image are (x, y), the coordinates of the third region in the frame difference image are also (x, y), and the shapes and sizes of the third region and the target connected component are the same.

S207, determining a region of interest (ROI) corresponding to the target connected domain, and removing imaging regions of sky, snow, road surface and the like in the ROI to obtain a candidate occlusion region.

First, a method of determining a region of interest (i.e., a second region) is described. The interested region is an image region formed by pixel points in the current frame or the ith historical frame image with the same position coordinates as the target connected domain. I.e. each frame image corresponds to one or more regions of interest.

Illustratively, the position of the target connected component, i.e., connected component 1, in the binary image is shown as (1) in fig. 12, and correspondingly, the position of the region of interest in the current frame is shown as (2) in fig. 12. It can be seen that the position of the target connected component in the binary image is the same as the position of the region of interest (i.e., the second region) in the current frame (i.e., the first image), i.e., the position of the third region in the frame difference image is the same as the position of the region of interest (i.e., the second region) in the current frame (i.e., the first image). The region of interest corresponds to the third region, i.e., to the pixel region with less variation between the current frame and the ith historical frame, i.e., it means that the region of interest is likely to include an occlusion region (i.e., an image formed by an occlusion through the onboard lens).

Of course, in a case that the on-vehicle lens is not occluded, a target connected domain may exist, and as shown in fig. 13, an interested region corresponding to the target connected domain may also be determined.

In addition, it has been pointed out above that the occlusion of the on-vehicle lens by the occlusion object causes the pixel values between the respective images of the occlusion object to change little or no value during a division period. For example, as shown in fig. 1C, the foreign object is blocked at the same position of the on-board lens for a period of time, and a plurality of images formed by the on-board lens are substantially unchanged. In some cases, image areas with small pixel value changes appear in the current and historical frames, possibly due to imaging of some interferents. For example, imaging of a large area of natural scenery such as sky, snow, ground, grass, etc. may also result in image areas with insignificant pixel value variations between different frames. For example, referring to fig. 14 (1) and 14 (2), the pixel value change of a part of the image area in the multi-frame image captured by the on-vehicle lens is always small due to the existence of a large area of sky. For example, the pixel values in the two elliptical regions are substantially close.

In order to further avoid judging interferents such as sky in the region of interest as an occlusion region, the influence of the interferents may also be rejected, that is, an imaging region of a preset region (one or more of sky, snow, road surface, grass, lake, and the like) is rejected in the region of interest (i.e., the second region). That is, the candidate occlusion region (i.e., the first region) is a region of the region of interest (i.e., the second region) from which the predetermined region is removed.

As a possible implementation mode, the specified features in the region of interest are detected, and pixel points without the specified features in the region of interest are used as candidate occlusion regions.

The pixel value of the pixel having the specified feature in the region of interest is set to a specified pixel value.

Wherein the specified characteristic is a characteristic of the interfering substance. The interference objects include but are not limited to sky, road surface, snow, grassland, lake and other natural landscapes with large imaging areas and consistent imaging pixel values.

That is, the imaging region representing sky, road surface, etc. in the region of interest is removed, and the rest region constitutes a candidate occlusion region.

A method of removing an imaged region such as sky and road surface from a region of interest is described as follows.

Taking the ROI cut from the current frame as an example, for example, in the case of the occlusion object, the ROI extracted from the current frame is shown in (2) in fig. 12, and in the case of the occlusion object, the ROI extracted from the current frame is shown in the frame in fig. 13. As a possible implementation manner, the elimination of the imaging area of the road surface from the ROI may be implemented as: a road surface template (TemplateRoad) such as that shown in fig. 15 is window-matched on the ROI by means of template matching. The process of template matching can be seen in fig. 16, where the road template is slid from left to right and top to bottom across the ROI. When the template slides to reach a pixel area (or called sliding window), each pixel in the sliding window is compared with the corresponding pixel of the pavement template, and when the matching metric value between the area to be detected and the template in the sliding window meets the threshold condition, the template is judged to be matched, namely the pixel area is likely to be the image of the pavement. Then, the pixel value of the pixel in the pixel region is set to 0 (i.e., set to black), i.e., the pixel region that may be the road surface is set to black. For example, in the pixel region circled by the dashed line frame shown in fig. 16, each pixel has a small difference from the corresponding pixel in the road surface template, and the pixel region is set to black.

The metric values between the sliding window and the template include, but are not limited to, the square error, the standard square error, the correlation value, and the like.

The pixel region shown by the dashed line box in fig. 16 is a pixel region having a specified feature, and the pixel region is a pixel region kicked away in the occlusion detection process of this time, that is, a pixel region that is not used as the occlusion candidate region.

In the template matching process, the size of the sliding window depends on the size of the pavement template. It should be noted that the method of detecting the road surface imaging region from the image is not limited to the above-described template matching method.

As a possible implementation, the elimination of the imaged region of sky from the ROI can be implemented as: a sky color segmentation range [ r 1-r 2, g 1-g 2, b 1-b 2] is set, and all pixels in the ROI are color-segmented according to the color segmentation range. Specifically, for a certain pixel point in the ROI, if the r component of the pixel point is in the range of r 1-r 2, the g component is in the range of g 1-g 2, and the b component is in the range of b 1-b 2, the pixel point is considered as an imaging pixel of the sky. And, the pixel is set to 0 (i.e., black).

Fig. 17 (2) shows the result of removing the sky-imaging region from the ROI shown in fig. 17 (1) (i.e., the ROI shown in fig. 13). The black pixel part is the pixel point which is eliminated, namely the pixel point of the sky imaging area.

As a possible implementation, the removing of the imaging region of the snow from the ROI may be implemented as: setting a color direction projection histogram (called color histogram for short) of the snow, carrying out back projection of the color histogram on the ROI, searching a region which is most matched with the snow imaging in the ROI, and rejecting the region.

The color histogram is used for representing the distribution condition of the image in the color space.

Backprojection is typically used to find the point or region in the input image (i.e., ROI) that best matches a particular image (i.e., snow image), i.e., to locate the snow in the ROI. Specifically, the ROI may be divided into a plurality of image blocks having a size identical to that of the snow image, and the following operation may be performed for each of the plurality of image blocks: comparing the color histogram of the image block with the color histogram of the snow image, and if the two color histograms are close to each other, for example, if the similarity is greater than or equal to the threshold, which indicates that the image block in the ROI is likely to be the imaging block of the snow, setting the upper left pixel of the image block to 0 (i.e., to black). That is, the portion of the ROI suspected of snow imaging is set to black. In the embodiment of the present application, the method of detecting snow formation from an image is not limited to the back projection method, and if the back projection method is adopted, the method of comparing whether the color histograms are similar is not limited to comparing the similarity of the color histograms.

Wherein, acquiring a color histogram of an image block in the ROI may be implemented as: first, the color is quantized, i.e. a color space, such as an RGB (red, green, blue) color space, is divided into a plurality of subspaces, each referred to as a bin. Then, the number of pixels in each bin of the color in the image block is counted to obtain a color histogram of the image block.

It should be noted that, for a certain historical frame or current frame, it may correspond to one or more candidate occlusion regions.

Due to the fact that the rain fog days, the snow days, the large-area sky and the large-area road surface bring fuzzy effects and cause misjudgment, the sky, the road surface and the snow part of the suspected occlusion area can be removed based on the modes of color segmentation, color histogram back projection and template matching, the candidate occlusion area is obtained, and accuracy of occlusion judgment is improved conveniently. Therefore, the method can ensure that the shielded and shielded area is the imaging caused by the real shielding object and is not misjudged in the natural scene. The method can accurately judge whether the shielding exists or not in all-weather and complex scenes. Meanwhile, the influence of rain and fog weather can be eliminated through the frame difference result, and in the rain and fog weather, if the vehicle-mounted lens is not shielded, a large-area with small brightness change cannot exist at the same position all the time.

Optionally, as shown in fig. 9-3, the rule control module 202 includes a candidate occlusion region extraction sub-module, taking the candidate occlusion region in the current frame as an example, the occlusion region extraction sub-module may obtain an interest region in the current frame according to the current frame and a target connected domain corresponding to the current frame, and remove regions such as sky and snow from the interest region to obtain the candidate occlusion region.

And S208, calculating the occlusion confidence coefficient of the candidate occlusion region and the area of the candidate occlusion region.

For example, the area of the candidate occlusion region (i.e. the first area corresponding to the frame difference image) may be represented by the number of pixels included in the candidate occlusion region. Other representations are possible.

The occlusion confidence level (i.e. the first confidence level corresponding to the frame difference image) is used to indicate the degree of blurring of the candidate occlusion region (i.e. the first region) in the current frame (i.e. the first image) corresponding to the frame difference image. Generally, when the lens is shielded by the shielding object, the shielding object is often closer to the lens, and the imaging characteristic is that the image of the shielded area is blurred. In the embodiment of the application, the higher the occlusion confidence is, the more fuzzy the candidate occlusion region is, and the higher the probability that the candidate occlusion region is imaged by the occlusion lens is. The first confidence is obtained according to a Local Binary Pattern (LBP) response score, a laplacian variance, and a laplacian mean of the first region. The first confidence is calculated as described in detail below.

Optionally, as shown in fig. 9-3, the rule control module 202 includes a first computation sub-module, and the confidence and the area of the candidate occlusion region may be output by inputting the candidate occlusion region into the first computation sub-module.

And converting the candidate occlusion area into a gray map, respectively calculating the class LBP response score, the Laplace variance and the Laplace mean value of the candidate occlusion area (the gray map), and then calculating the weighted sum of the class LBP response score, the Laplace variance and the Laplace mean value to obtain the occlusion confidence coefficient.

The LBP-like response score can be used for representing the texture richness degree of the image.

First, the way the class LBP response score is calculated is described. As a possible implementation manner, as shown in fig. 18, the following operations are performed on each pixel point (taking the pixel P as an example) in the candidate occlusion region (I (x, y)): comparing the pixel value of the pixel point P with the pixel value of the 8 neighborhood pixels, if the absolute value difference between the pixel value of the pixel point P and the pixel value of a certain pixel point in the neighborhood is greater than or equal to a preset threshold (for example, 3), the pixel point is correspondingly set to 1, otherwise, if the absolute value difference is less than the preset threshold, the pixel point is correspondingly set to 0. For example, if the pixel value of the top left pixel in fig. 18 is 5, the absolute value of the difference between the pixel value of the top left pixel and the pixel value of the pixel P is 1, and the absolute value is smaller than the threshold 3, the pixel point is set to 0. By analogy, a binary string corresponding to 8 pixels in the neighborhood of the pixel P is obtained, i.e., the binary string 01100101 shown in fig. 18. Then, the number of 1 values in the binary string (e.g., 01100101 shown in fig. 18) is compared, and if the number is greater than or equal to a preset threshold (e.g., 4), the pixel point P is marked as a strong response point. Of course, the number of 0 values in the giant binary string can be counted, and when the number is less than or equal to a certain threshold, the pixel P is marked as a strong response point. The strong response points are pixels with larger pixel difference with more neighborhood pixels or pixels with rich peripheral textures. For example, the pixel values are different from more than 4 neighboring pixels.

It should be noted that, as shown in fig. 18, the binary strings obtained by spreading binary values corresponding to each pixel point in a clockwise direction are mainly taken as an example, in other scenarios, other spreading manners may also be adopted, such as spreading in a counterclockwise direction to obtain 11010100.

As another possible implementation manner, taking marking a strong response point or a weak response point for the pixel P as an example, as shown in fig. 18, after obtaining a neighborhood binary string, that is, 01100101, corresponding to the pixel P, it may be further determined whether the pixel P is a strong response point by counting the number of value transitions in the binary string. Specifically, the jump frequency from 0 to 1 in the binary string and the change frequency from 1 to 0 are counted, and when the change frequency is greater than or equal to a preset threshold, the pixel point P is marked as a strong response point. Taking the preset threshold of 4 as an example, if there are 3 transitions from 0 to 1, 2 transitions from 1 to 0, and 5 total transitions (greater than the threshold of 4) in 01100101, the pixel P is marked as a strong response point.

It should be noted that, for the pixel P, the step of determining whether the pixel P is a strong response point in any of the two manners may be performed, that is, the step may be determined by the number of the value 1 in the binary string, or may be determined by the number of transitions in the binary string. If one of the determination methods determines that the pixel P is a strong response point, the pixel P is marked as a strong response point. If the pixel P is judged not to be a strong response point in the two modes, the pixel point is marked as a weak response point. And weak response points refer to pixel points with sparse peripheral textures.

Similarly, referring to the way of marking the strong response point or the weak response point on the pixel point P, similar judgment is performed on the rest of the pixel points in the candidate shielding region, so as to mark the strong response point or the weak response point on all the pixel points in the candidate shielding region.

Next, the number Num _ Strong of Strong response points in the candidate occlusion region and the total number Num _ Sum of pixels in the candidate occlusion region are counted. Optionally, according to the number Num _ Strong of the Strong response points and the number Num _ Sum of the total pixels, the class LBP response score D is calculated according to the following formula.

The LBP-like response score is used for representing the fuzzy degree and the texture richness of the candidate occlusion area. It can be seen from formula 1 that, when the number of strong response points in the image is smaller, the LBP-like response score is lower, which indicates that the pixel values of the image tend to be more consistent, i.e. the difference of the pixel values between pixels is smaller, and the blurred texture of the image is more sparse.

The Laplace operator is a second order differential algorithm. Typically, the Laplace operator can be used to process the images to detect the degree of blurring or occlusion of the different images. Specifically, the clear image is processed by the Laplace operator, so that the image contrast is enhanced, the image boundary is sharpened, the texture is clearer, and the deviation of the obtained pixel value is increased. The pixel values obtained after the blurred image is processed by a Laplace operator are distributed uniformly. By the characteristics, the statistical variance or mean value can be used for evaluating the blurring degree of the image, and specifically, the clearer image is subjected to Laplace operator processing to obtain the bigger variance and mean value of the pixel value, and the blurred image is subjected to Laplace operator processing to obtain the smaller variance and mean value of the pixel value.

In order to obtain the Laplace variance and the Laplace mean value, firstly, a Laplace operator is used for processing a candidate occlusion area to obtain a Laplace response, then the variance of the Laplace response, namely the Laplace variance, is obtained, and the mean value of the Laplace response, namely the Laplace mean value, is obtained.

First, a method of obtaining a Laplace response will be described. As a possible implementation manner, a Laplace response G (x, y) is obtained by convolving the candidate occlusion region I (x, y) with a Laplace convolution kernel H (x, y) (i.e., a Laplace operator). Wherein G (x, y) ═ I (x, y) × H (x, y). Is the convolution operator.

The convolution kernel H (x, y) may be a two-dimensional matrix, and the size of the convolution kernel may be selected according to actual requirements, for example, it may be a 3 x 3 (i.e. 3 times 3) matrix as shown in fig. 19

Candidate occlusion region I (x, y) is also providedCan be regarded as a two-dimensional matrix, such as the candidate occlusion region shown in FIG. 19

And (3) the convolution process of I (x, y) and H (x, y), namely, for each pixel point of the I (x, y) image, calculating the product of each pixel in the eight neighborhood pixels and the corresponding element in the Laplace convolution kernel, and then adding the products to serve as the pixel value of the pixel. As shown in fig. 19, taking the pixel value of the updated pixel Q as an example, the initial pixel value of the pixel Q in the candidate occlusion region is 1, and the pixel value of the updated pixel Q is 0 × 4+0 × 0+0 × 0+1 × 0+0 × 0+2 (-4) — 8. With reference to the way of updating the pixel value of the pixel Q, the remaining pixels in the candidate occlusion region I (x, y) are updated with the pixel value H (x, y), that is, the convolution result of I (x, y) and H (x, y) is obtained, that is, the Laplace response G (x, y) of the candidate occlusion region I (x, y) is obtained.

Of course, there are some boundary pixels where there is no eight neighborhood, such as pixel R shown in fig. 19. For these boundary pixels, a boundary filling mode may be adopted to update the pixel values of the boundary pixels. For example, the pixel values of the boundary pixels are updated by 0 padding, boundary copy padding, mirror padding, block padding, and the like.

Next, the variance of the Laplace response G (x, y) is calculated, resulting in a Laplace variance Var _ Laplace. And calculating the average value of G (x, y) to obtain Laplace average value Mean _ Laplace.

As already indicated above, a more blurred image typically has smaller Var _ Laplace and Mean _ Laplace values, and a sharper image has larger Var _ Laplace and Mean _ Laplace values.

And after the Laplace variance, the Laplace mean value and the similar LBP response score are calculated, calculating the weighted sum of the three to obtain the shielding confidence coefficient. Optionally, the occlusion confidence is calculated according to the following formula 2:

wherein, the SCORE is an occlusion confidence coefficient and is used for representing the characteristics of the image, such as the fuzzy degree, the texture abundance degree and the like. But also to characterize the confidence level that the candidate occlusion region (first region) is identified as an occlusion region. D is LBP-like response fraction, and a is the weighting coefficient of D; var _ Laplace is Laplace variance, and b is a weighting coefficient of the Laplace variance; mean _ Laplace is a Laplace Mean value, and c is a weighting coefficient of the Laplace Mean value; n is a constant value, and is related to the size of the candidate occlusion region.

It has been noted above that the smaller the LBP-like response score, the greater the degree of image blur; the smaller the Laplace mean value is, the larger the image blurring degree is; the smaller the Laplace variance, the greater the degree of image blur. Then, when the LBP-like response score, Laplace mean and Laplace variance are all small, the occlusion confidence is usually large, which represents that the image blur degree is high. Namely, the fuzzy degree of the candidate occlusion region can be judged by judging the occlusion confidence.

In the embodiment of the application, the occlusion confidence coefficient is obtained as the occlusion judgment criterion according to the Laplace mean value, the variance and the similar LBP response score of the candidate occlusion area, whether the suspected occlusion area is fuzzy or not is judged through the multi-dimensional characteristics, and the accuracy and the robustness of fuzzy judgment are improved. And further, robustness and accuracy of judgment results of the shielding areas can be improved.

S209, judging whether the vehicle-mounted lens is shielded or not according to the shielding confidence coefficient and the shielding area of the candidate shielding area.

Optionally, as shown in fig. 9-3, the rule control module 202 includes a judging sub-module, and the confidence and the area are input into the judging sub-module, and a judgment result, that is, whether the on-board lens is shielded and a specific shielding position, may be output.

As shown in fig. 9-1, the embodiment of the present application provides two determination mechanisms, which are suitable for occlusion determination in different scenes. And when the candidate occlusion area corresponding to the ith historical frame meets the first branch, judging whether the vehicle-mounted lens is occluded according to a first judgment mechanism. And when the candidate shielding area corresponding to the ith historical frame meets the second branch, judging whether the vehicle-mounted lens is shielded or not according to a second judgment mechanism. Therefore, the scene sheltered by the foreign matters in a large area can be judged more quickly, and the sheltering in a small area can be found more accurately. The two mechanisms for determining whether the on-vehicle lens is blocked are described below.

In the judgment mechanism corresponding to the first branch, a queue with the length of L is used for storing the judgment results corresponding to the current frame and the historical L-1 frame. And judging whether the occlusion confidence score is greater than or equal to a first confidence threshold and whether two conditions of the area greater than or equal to the first area threshold are both satisfied or not for a candidate occlusion region corresponding to a certain frame, if so, storing 1 into a queue, otherwise, storing 0 into the queue if any one of the two conditions is not satisfied. And judging the number of numerical values 1 in the L numerical values of the queue after judging the candidate occlusion areas corresponding to the L-1 historical frames and the current frame, and determining that the vehicle-mounted lens is occluded when the number of 1 is greater than or equal to a certain threshold value. Otherwise, determining that the vehicle-mounted lens is not shielded.

It can be understood that if the occlusion confidence score of the candidate occlusion region corresponding to a certain frame is greater than or equal to the first confidence threshold and both conditions that the area is greater than or equal to the first area threshold are satisfied, the candidate occlusion region is considered as an occlusion region, that is, an image formed by the lens is occluded by the occlusion object. Through judging the candidate occlusion areas corresponding to the multiple frames, the influence of misjudgment on the candidate occlusion area corresponding to a certain frame can be avoided, and the accuracy of the occlusion detection result is improved.

For example, as shown in fig. 20, assuming that the threshold is 4, the number of 1 s in the queue is 6 (greater than the threshold 4), which indicates candidate occlusion areas corresponding to a plurality of frames, it is determined that the in-vehicle lens is occluded.

In other embodiments, the determination mechanism corresponding to the second branch is generally used for determining large-area occlusion. In some scenes, due to the large size of the shelter or other factors, the area imaged by the shelter lens (i.e. the sheltered area) is large, and the sheltered area usually appears in continuous multiframe pictures and is not easy to fluctuate. Based on the characteristic, in the embodiment of the present application, a frame number Counter is set, when the area of the candidate occlusion region is greater than or equal to the second area threshold and the occlusion confidence of the candidate occlusion region is greater than or equal to the second confidence threshold, the frame number Counter (Counter) is incremented by one, otherwise, the Counter is set to 0. If the Counter is larger than or equal to the threshold value, the situation that the shielding areas exist in the continuous multiframes is indicated, and in this case, the vehicle-mounted lens is determined to be shielded.

In the second judgment mechanism, in the case where the on-vehicle lens is largely blocked by a foreign object, for example, a block occurs as shown in fig. 21, the block can be found and detected more quickly by means of continuous cumulative judgment.

In the embodiment of the application, the mode of queue accumulation judgment is adopted, so that the abnormal conditions such as intermittent brightness mutation and the like can be smoothed, and the robustness of the judgment result is improved.

Optionally, after it is determined that the on-vehicle lens is completely or partially occluded, an alarm mechanism is triggered. For example, the automatic driving may be stopped and the driver may take over the driving.

In other cases, multiple frame difference images may be acquired and a determination may be made as to whether the onboard lens is occluded based on the multiple frame difference images. The plurality of frame difference images includes a first frame difference image and a second frame difference image; the first frame difference image is an image obtained by performing difference operation on a first image shot by the vehicle-mounted lens at a first moment and a second image before the first image; the second frame difference image is an image obtained by performing difference operation on a first image shot by the vehicle-mounted lens at a second moment and a second image before the first image, and the first moment is different from the second moment.

As shown in fig. 9-2, the method provided by the embodiment of the present application includes:

s301, vehicle speed information and an ISP identification position are obtained.

This step can be seen in step S201 described above.

S302, whether the vehicle speed reaches a first threshold value and whether the ISP identification bit is true or not are judged. If the vehicle speed reaches the first threshold and the ISP flag is true, step S303 is executed. Otherwise, the flow is ended.

This step can be seen in step S202 described above.

And S303, acquiring the current frame and the historical frame.

The current frame is the first image and the historical frame is the second image.

It should be noted that, in the embodiment of the present application, a plurality of frame difference images may be acquired, and whether the on-vehicle lens is occluded or not may be determined according to the plurality of frame difference images. Fig. 9-2 illustrates an example of one frame difference image among a plurality of frame difference images. In one example, a current frame at a first time and a history frame X frames away from the current frame may be acquired, and a frame difference image between the current frame and the history frame may be calculated, and a current frame at a second time and a history frame X frames away from the current frame may be acquired, and a frame difference image between the current frame and the history frame may be calculated. X is a positive integer. By analogy, M (M is a positive integer) frame difference images can be obtained, and a subsequent process is performed according to the M frame difference images, so as to judge whether the on-board lens is blocked.

In some embodiments, X history frames may be stored using a queue of length X. For a current frame at a certain time, a frame difference image is calculated using a history frame that is X frames away from the current frame and the current frame. After the frame difference image is calculated, the current frame can be inserted into the tail of the queue, and the corresponding calculated historical frame is subjected to dequeuing operation. Thus, the current frame becomes a history frame at a subsequent time. The same operation may be performed for the current frame at the other time instant and the history frame that is X frames away from the current frame at the other time instant. Therefore, the current frame at a certain moment after calculation is finished can be inserted into the queue in real time to become a historical frame at a subsequent moment, and the calculated historical frame is deleted from the queue to update elements in the queue, so that the subsequent calculation can be finished by using a queue with a smaller length, and the storage space can be effectively saved.

S304, performing inter-frame difference operation on the historical frame and the current frame to obtain a frame difference image.

S305, thresholding is carried out on the frame difference image to obtain a binary image.

S306, detecting a target connected domain in the binary image.

S307, determining an interested area corresponding to the target connected area, and eliminating imaging areas of sky, snow, road surface and the like in the interested area to obtain a candidate blocking area.

S308, calculating the occlusion confidence coefficient of the candidate occlusion region and the area of the candidate occlusion region.

S309, judging whether the vehicle-mounted lens is shielded or not according to the shielding confidence coefficient and the shielding area of the candidate shielding area.

As a possible implementation manner, if M frame difference images are obtained in total, and P frame difference images in the M frame difference images satisfy the first condition, it is determined that the on-vehicle lens is blocked. The first condition is that the confidence is greater than or equal to a first confidence threshold, and the first area is greater than or equal to a first area threshold; m, P is a positive integer, M is greater than or equal to P. Or if the continuous Q frame difference images in the M frame difference images meet a second condition, determining that the vehicle-mounted lens is blocked; the second condition is that the confidence is greater than or equal to a second confidence threshold, and the first area is greater than or equal to a second area threshold; m, Q is a positive integer, M is greater than or equal to Q.

For example, referring to fig. 9-2, if the number of 1 s in the queue is greater than P, it indicates that there are occlusion areas in all P frames, and in this case, it is determined that the in-vehicle lens is occluded. For another example, in another example, if the count of the frame counter reaches Q, it indicates that there is an occlusion region in each of the consecutive Q frames, and in this case, it is determined that the in-vehicle lens is occluded.

The specific implementation of steps S303-S309 can be seen in steps S203-S209 described above.

In the embodiment of the present application, functional modules of the on-board lens blocking recognition device may be divided according to the above method example, and in a case that the functional modules are divided according to the corresponding functions, fig. 22 shows a schematic diagram of a possible structure of the on-board lens blocking recognition device in the above embodiment. As shown in fig. 22, the in-vehicle lens blocking recognition device includes an acquisition module 401 and a determination module 402. Of course, the on-board lens shielding identification device may also include other functional modules, or the on-board lens shielding identification device may include fewer functional modules.

The acquiring module 401 is configured to acquire a frame difference image; the frame difference image is an image obtained by performing difference operation on a first image and a second image shot by the vehicle-mounted lens; the frame interval between the first image and the second image is N, and N is a positive integer; neither the first image nor the second image is a background image;

a judging module 402, configured to judge whether the on-vehicle lens is blocked according to the frame difference image.

In a possible design, the obtaining module 401 is further configured to obtain a first confidence and a first area corresponding to one or more frame difference images;

the first area is an area obtained by removing a preset area from the second area; the second region is included in the first image, and the position of the second region in the first image is the same as the position of a third region in a frame difference image corresponding to the first image; the third area is an area formed by pixel points of which the pixel values are smaller than or equal to a first threshold value in the frame difference image; the preset area comprises any one or more of the following parts: an imaging area of a road surface, an imaging area of a snow area, an imaging area of a sky area, an imaging area of a grass area.

In one possible design, the frame difference images are M; the determining module 402 is configured to determine whether the on-vehicle lens is blocked according to the frame difference image, and includes: the method comprises the steps of determining that the vehicle-mounted lens is shielded if P frame difference images in the M frame difference images meet a first condition;

In one possible design, the frame difference images are M; the determining module 402 is configured to determine whether the on-vehicle lens is blocked according to the frame difference image, and includes: the on-board lens is determined to be shielded if the continuous Q frame difference images in the M frame difference images meet a second condition;

In one possible design, the obtaining module 401 is configured to obtain a frame difference image, and includes: the method comprises the steps of obtaining a vehicle speed and/or a first identification position; the first identification bit is used for identifying whether an image processing result of the image signal processing ISP is normal or not; and if the image processing result of the ISP identified by the first identification bit is normal and/or the vehicle speed is greater than or equal to a vehicle speed threshold value, acquiring the frame difference image.

Referring to fig. 23, the present application further provides an in-vehicle lens occlusion recognition apparatus, which includes a processor 510 and a memory 520. The processor 510 is coupled to the memory 520 (e.g., via bus 540).

Optionally, the in-vehicle lens blocking recognition device may further include a transceiver 530, and the transceiver 530 is connected to the processor 510 and the memory 520, and the transceiver is configured to receive/transmit data. Such as receiving data from other vehicles.

The processor 510 may perform the operations of any one of the embodiments corresponding to fig. 8, 9-1, and 9-2, as well as various possible embodiments thereof. For example, the module may be configured to perform the operations of the obtaining module 401, the determining module 402, and/or other operations described in the embodiments of the present application.

The processor 510 (or alternatively described as a controller) may implement or execute the various illustrative logical blocks, unit modules, and circuits described in connection with the disclosure herein. The processor or controller may be a central processing unit, general purpose processor, digital signal processor, application specific integrated circuit, field programmable gate array or other programmable logic device, transistor logic device, hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, unit modules, and circuits described in connection with the disclosure herein. The processor may also be a combination of computing functions, e.g., comprising one or more microprocessors, DSPs, and microprocessors, among others.

The bus 540 may be an Extended Industry Standard Architecture (EISA) bus or the like. The bus 540 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 23, but it is not intended that there be only one bus or one type of bus.

For the specific working procedures of the processor, the memory, the bus and the transceiver, reference is made to the above, and the detailed description is omitted here.

The application also provides a vehicle-mounted lens occlusion recognition device, which comprises a nonvolatile storage medium and a central processing unit, wherein the nonvolatile storage medium stores an executable program, and the central processing unit is connected with the nonvolatile storage medium and executes the executable program to realize the vehicle-mounted lens occlusion recognition method shown in fig. 8-9-1 and 9-2.

Another embodiment of the present application also provides a computer-readable storage medium including one or more program codes, the one or more programs including instructions, which when executed by a processor, cause the in-vehicle lens occlusion recognition apparatus to perform the in-vehicle lens occlusion recognition method as shown in fig. 8, 9-1, and 9-2.

In another embodiment of the present application, there is also provided a computer program product comprising computer executable instructions stored in a computer readable storage medium. The at least one processor of the on-vehicle lens occlusion recognition device can read the computer-executable instructions from the computer-readable storage medium, and the at least one processor executes the computer-executable instructions to enable the on-vehicle lens occlusion recognition device to implement and execute corresponding steps in the on-vehicle lens occlusion recognition method shown in fig. 8, 9-1 and 9-2.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the above embodiments, all or part of the implementation may be realized by software, hardware, firmware or any combination thereof. When implemented using a software program, may take the form of a computer program product, either entirely or partially. The computer program product includes one or more computer instructions. The procedures or functions according to the embodiments of the present application are all or partially generated when the computer program instructions are loaded and executed on a computer.

The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, digital subscriber line (DSL 0)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk, SSD), among others.

Through the description of the foregoing embodiments, it will be clear to those skilled in the art that, for convenience and simplicity of description, only the division of the functional modules is illustrated, and in practical applications, the above function distribution may be completed by different functional modules as needed, that is, the internal structure of the apparatus may be divided into different functional modules to complete all or part of the above described functions.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical functional division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another device, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may be one physical unit or a plurality of physical units, that is, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only an embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions within the technical scope of the present disclosure should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A vehicle-mounted lens shielding identification method is characterized by comprising the following steps:

acquiring a frame difference image; the frame difference image is an image obtained by performing difference operation on a first image and a second image shot by the vehicle-mounted lens; the frame interval between the first image and the second image is N, and N is a positive integer; neither the first image nor the second image is a background image;

and judging whether the vehicle-mounted lens is shielded or not according to the frame difference image.

2. The on-vehicle lens occlusion recognition method according to claim 1, wherein the frame difference image is one or more frame difference images, the plurality of frame difference images including a first frame difference image and a second frame difference image; the first frame difference image is an image obtained by performing difference operation on a first image shot by the vehicle-mounted lens at a first moment and a second image before the first image; the second frame difference image is an image obtained by performing difference operation on a first image shot by the vehicle-mounted lens at a second moment and a second image before the first image, and the first moment is different from the second moment.

3. The on-vehicle lens occlusion recognition method of claim 2, further comprising: acquiring a first confidence coefficient and a first area corresponding to one or more frame difference images;

4. The on-vehicle lens shielding identification method according to any one of claims 1 to 3, wherein the number of the frame difference images is M; judging whether the vehicle-mounted lens is shielded or not according to the frame difference image, wherein the judging step comprises the following steps:

5. The on-vehicle lens shielding identification method according to any one of claims 1 to 4, wherein the number of the frame difference images is M; judging whether the vehicle-mounted lens is shielded or not according to the frame difference image, wherein the judging comprises the following steps:

6. The on-vehicle lens shielding identification method according to any one of claims 1 to 5, wherein the acquiring a frame difference image comprises:

7. The utility model provides an on-vehicle camera lens shelters from recognition device which characterized in that includes:

8. The on-vehicle lens shading recognition device according to claim 7, wherein the frame difference image is one or more frame difference images, the plurality of frame difference images including a first frame difference image and a second frame difference image; the first frame difference image is an image obtained by performing difference operation on a first image shot by the vehicle-mounted lens at a first moment and a second image before the first image; the second frame difference image is an image obtained by performing difference operation on a first image shot by the vehicle-mounted lens at a second moment and a second image before the first image, and the first moment is different from the second moment.

9. The vehicle-mounted lens occlusion recognition device according to claim 8, wherein the obtaining module is further configured to obtain a first confidence and a first area corresponding to one or more frame difference images;

10. The on-vehicle lens shading recognition device according to any one of claims 7 to 9, wherein the number of the frame difference images is M; the judging module is used for judging whether the vehicle-mounted lens is shielded according to the frame difference image, and comprises: the method comprises the steps of determining that the vehicle-mounted lens is shielded if P frame difference images in the M frame difference images meet a first condition;

11. The on-vehicle lens shading recognition device according to any one of claims 7 to 10, wherein the number of the frame difference images is M; the judging module is used for judging whether the vehicle-mounted lens is shielded according to the frame difference image, and comprises: the on-board lens is determined to be shielded if the continuous Q frame difference images in the M frame difference images meet a second condition;

the second condition is that the confidence is greater than or equal to a second confidence threshold, the first area is greater than or equal to a second area threshold; m, Q is a positive integer, M is greater than or equal to Q.

12. The vehicular lens shading recognition device according to any one of claims 7 to 11, wherein the obtaining module is configured to obtain a frame difference image, and includes: the method comprises the steps of obtaining a vehicle speed and/or a first identification position; the first identification bit is used for identifying whether an image processing result of the image signal processing ISP is normal or not; and if the image processing result of the ISP identified by the first identification bit is normal and/or the vehicle speed is greater than or equal to a vehicle speed threshold value, acquiring the frame difference image.

13. The utility model provides an on-vehicle camera lens shelters from recognition device which characterized in that includes: a processor, a memory, and a communication interface; wherein the communication interface is configured to communicate with another device or a communication network, and the memory is configured to store one or more programs, the one or more programs including computer executable instructions, and when the apparatus is running, the processor executes the computer executable instructions stored in the memory to cause the apparatus to perform the in-vehicle lens occlusion recognition method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized by comprising a program and instructions, which when run on a computer, implement the in-vehicle lens occlusion recognition method according to any of claims 1-6.

15. A computer program product comprising instructions for causing a computer to perform the in-vehicle lens occlusion recognition method according to any of claims 1-6 when the computer program product is run on the computer.

16. A chip system comprising a processor coupled to a memory, the memory storing program instructions that, when executed by the processor, implement the in-vehicle lens occlusion recognition method of any of claims 1-6.