WO2018028974A1 - Method and apparatus for soiling detection, image processing system and advanced driver assistance system - Google Patents

Method and apparatus for soiling detection, image processing system and advanced driver assistance system Download PDF

Info

Publication number
WO2018028974A1
WO2018028974A1 PCT/EP2017/068612 EP2017068612W WO2018028974A1 WO 2018028974 A1 WO2018028974 A1 WO 2018028974A1 EP 2017068612 W EP2017068612 W EP 2017068612W WO 2018028974 A1 WO2018028974 A1 WO 2018028974A1
Authority
WO
WIPO (PCT)
Prior art keywords
soiling
imaging system
image processing
frame
processing data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/EP2017/068612
Other languages
French (fr)
Inventor
Rui Guerreiro
Alexander Kadyrov
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Aumovio Germany GmbH
Original Assignee
Continental Automotive Technologies GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Continental Automotive Technologies GmbH filed Critical Continental Automotive Technologies GmbH
Publication of WO2018028974A1 publication Critical patent/WO2018028974A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/0002Inspection of images, e.g. flaw detection
    • G06T7/0004Industrial image inspection
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10016Video; Image sequence
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30248Vehicle exterior or interior
    • G06T2207/30252Vehicle exterior; Vicinity of vehicle

Definitions

  • the present invention relates to a method and an apparatus for soiling detection on an image system. Further, the present invention relates to an image processing system and an advanced driver assistance system comprising an image pro ⁇ cessing system.
  • One factor that affects the image quality and which is diffi ⁇ cult to control is the degree of contamination of the optical system of the camera. Cameras may be positioned at places with less risk of contamination, or cameras may be cleaned, for instance, by an electric viper. Despite of these provi ⁇ sions, it is impossible to avoid the contamination of the op ⁇ tical system completely. Therefore, it is necessary to detect soiling on a camera lens automatically in order to trigger an appropriate action.
  • EP 2 351 351 Al discloses a method for detecting the presence of an impediment on a lens of an image capture device. At least two image frames are captured to detect similar por ⁇ tions in the image frames. Presence of an impediment on the lens is determined by analyzing the luminance of the respec ⁇ tive area. In particular, it is suggested that the method is performed in a vehicle which is moving at a speed in the range of 5 km/h to 30 km/h.
  • the present invention provides a method for soiling detection on an image system with the features of independent claim 1. Further, the present invention provides a soiling detection apparatus with the features of independent claim 9.
  • the present invention provides a method for soiling detection on an imaging system comprising the steps of capturing, by the imaging system, a first frame, while the imaging system is moving; computing first image processing data based on the captured first frame; capturing, by the imaging system, a second frame, while the imaging sys ⁇ tem is not moving; computing second image processing data based on the captured second frame; and determining a soiling measure for each pixel position covered by the imaging system based on the computed first image processing data and the computed second image processing data, wherein a said soiling measure comprising a first value specifying a degree of soil ⁇ ing and a second value specifying a confidence value of the first value.
  • the present invention provides a soiling detection apparatus for detecting soiling on an imaging system.
  • the apparatus comprises an image receiving unit, an image data processor, and a soiling detector.
  • the image receiving unit is adapted to receive a first frame, which is captured by the imaging system while the imaging system is moving.
  • the image receiving unit is further adapted to receive a second frame, captured by the imaging system while the imaging system is not moving.
  • the image data pro ⁇ cessor is adapted to compute first image processing data based on the received first frame and compute second image processing data based on the received second frame.
  • the soil- ing detector is adapted to determine a soiling measure for each pixel position covered by the imaging system based on the computed first image processing data and the computed second image processing data.
  • the soiling measure comprises a first value specifying a degree of soiling and a second value specifying a confidence value of the first value.
  • the present invention provides an image processing system comprising an imaging system including a camera.
  • the imaging system is adapted to capture image data.
  • the image processing system further comprises a soiling detection apparatus according to the second aspect of the present inven ⁇ tion.
  • the present invention provides an advanced driver assistance system comprising an image processing system according with a soiling detection apparatus.
  • reference image data are generated while the imaging system is moving, and these reference image data are used as a basis for analyzing the image data which are captured when the imaging system is standing still.
  • the reference image data as well as the image data which are captured while the imaging system stands still are processed in order to remove elements in the image data caused by moving objects in the coverage area of the imaging system. In this way, it is possible to obtain image pro ⁇ cessing data without any elements of moving objects.
  • a reliable detection of artifacts caused by a contamination of the imaging system can be achieved.
  • a fast and reliable soiling detection of an imaging system is provided.
  • the reliability of the indi ⁇ vidual pixels of the image data provided by the imaging sys ⁇ tem can be evaluated.
  • the reliability of the image da ⁇ ta provided by the imaging system can be increased.
  • the soiling measure which is provided by the present inven ⁇ tion comprises at least two values.
  • a first value specifies a degree of soiling for each pixel provided by the imaging sys ⁇ tem.
  • a second value specifies a confidence val ⁇ ue of the first value. In this way, the reliability of the detected soiling of the imaging system can be further specified .
  • the soiling measure is updated iteratively. In this way, continuous evaluation of the pro ⁇ vided image data and the soiling of the imaging system can be achieved .
  • determining the soiling measure comprises applying an infinite impulse response (IIR) filter of the first value and/or the second value of the soiling measure.
  • IIR infinite impulse response
  • computing the first image processing data and computing the second image processing data comprises computing a transition between a current frame and a previous frame. In this way, static elements in the im ⁇ age data can be eliminated. Hence, the further processing of the image processing data only has to take into account changes in the image data.
  • computing a transition between the current frame and the previous frame comprises applying a Laplacian filter.
  • a Laplacian filter is a very efficient and reliable way for filtering static elements in the image data.
  • computing the first image processing data and the second image processing data comprising ⁇ es applying an infinite impulse response filtering to the computed transition between the current frame and the previ ⁇ ous frame.
  • computing first image pro ⁇ cessing data and computing second image processing data comprises computing a sharpness of a captured first frame and a captured second frame.
  • the sharpness of a cap ⁇ tured frame can be classified into a plurality of predeter ⁇ mined classes. For example, sharpness can be classified into "no edge”, “blurred edge” and "sharp edge”.
  • determining a soiling measure comprises comparing the computed sharpness of the first frame and the computed sharpness of the second frame.
  • Figure 1 shows a schematic drawing of a vehicle compris ⁇ ing an imaging system according to an embodiment
  • Figure 2 shows a schematic diagram of a flowchart of a soiling detection method underlying an embodiment
  • Figure 3 shows a schematic diagram for determining sharpness in a frame according to an embodiment
  • Figure 4 shows a schematic illustration of image data
  • Figure 5 shows a schematic illustration of a soiling de ⁇ tection apparatus according to an embodiment.
  • FIG. 1 shows an image processing system according to an embodiment of the present invention.
  • the image processing sys ⁇ tem comprises an imaging system 10.
  • the image processing system may be a part of an advanced driver assis ⁇ tance system (ADAS) of a vehicle 20.
  • ADAS advanced driver assis ⁇ tance system
  • the imaging system 10 and the later described soiling detection apparatus of the image processing system is not limited to an ADAS or any other application of a vehicle.
  • soiling de- tection of an imaging system may be also applied to any other image processing system.
  • the image processing system of Figure 1 comprises an imaging system 10.
  • the imaging system 10 includes a camera 11 and a lens 12.
  • Camera 11 captures image data and provides the image data to a further device for processing the image data.
  • the image data may be provided in form of image frames.
  • Camera 11 may capture image data in form of frames having a predetermined resolution. Further, camera 11 may capture frames in regular time intervals. Alternatively, any other form of capturing image data is possible, too.
  • the optical system of the imaging system 10 may be contaminated.
  • dust, particles, raindrops or any other kind of contamination may be located on lens 12.
  • the imaging system 10 may be located behind a wind ⁇ shield of a vehicle or any other transparent protecting ele ⁇ ment.
  • the contamination 13 may be located on the windshield or the further protecting element. Due to this contamination 13, the image data captured by camera 11 may be disturbed .
  • a vehicle 20 may stand still at an intersection, and another vehicle may splash water onto the windshield of the vehicle or on the lens 12 of imaging system 10.
  • the image areas covered by this contamination do not provide appropriate image data for a further processing, in particular for use in an ADAS.
  • Figure 2 illustrates a flowchart of a method for soiling de ⁇ tection on the imaging system 10 underlying an embodiment of the present invention.
  • the imaging system 10 captures image data while the imaging system 10 is moving.
  • the imaging system 10 may successively capture first frames while the imaging system 10 is moving.
  • first image pro ⁇ cessing data are computed.
  • Several different methods or algo ⁇ rithm may be used for computing the first image processing data. Some examples for computing first image processing data will be described below.
  • step S3 Capturing SI of first frames and computing S2 of first image processing data is performed until the imaging system 10 stops moving. If the imaging system 10 stands still, in step S3 one or a plurality of successive second frames are cap ⁇ tured. Based on the captured second frames, in step S4 second image processing data are computed.
  • the computing of the sec ⁇ ond image processing data may be similar to the computing S2 of the first image processing data and will be also described below in more detail.
  • a soiling measure is determined based on the computed first image processing data and the computed second image processing data.
  • the first im ⁇ age processing data may be used as a reference image data and the second image processing data may be used as current image data.
  • the determining step S5 for de ⁇ termining a soiling measure computes a first value specifying a degree soiling in the image data relating to the second frame, and a second value specifying a confidence value for the determined degree of soiling.
  • the soiling measure may be computed separately for each pixel in the im ⁇ age data of the frames provided by the imaging system 10.
  • This first example is based on a computation of transitions in the image data of the frames provided by the imaging system 10.
  • the soiling detection may be also performed by other analysis of the frames provided by the imaging system 10.
  • a magnitude of derivatives E is computed for each frame I provided by the imaging system 10. Since the direction of the derivatives is not relevant for this method, the computation can be performed, for instance, by applying a simple Laplaci- an filter F to the image data of a frame I:
  • This transition matrix T can be determined, for instance, by the following formula:
  • ThEdge is a threshold value that guarantees that a clear transition is detected.
  • transition T represents a matrix having the size of the frames provided by imaging system 10, wherein each el ⁇ ement of this matrix represents the probability of a transi ⁇ tion for the corresponding pixel.
  • first image processing data are computed as reference image data by the following formula:
  • R_0 T_0 (for a first frame)
  • R_f+1 (1- ) -R_f + cx-T_f+l (for the next frames), wherein f and f+1 are the number of frames, and a is a coef ⁇ ficient between 0 and 1.
  • a small value for example 0.1 may be used for a .
  • IIR infinite im ⁇ pulse response
  • the goal of this filtering is to remove the influence of moving objects in the field of view of the imaging system 10.
  • a small value of a implies that higher filtering occurs.
  • small values of a also may lead to the effect, that the past, where the imaging system 10 was observing something different, has bigger influence to the final reference image data .
  • This reference image data (first image processing data) is updated based on the frames provided by the imaging system 10 as long as the imaging system 10 is moving. If the imaging system stops, the updating of the reference image data stops, too. When the imaging system 10 starts moving again, updating of the reference image data resumes.
  • second image pro ⁇ cessing data N is generated.
  • the calculation of the second image processing data N is performed in a similar way to the calculation of the reference image data (first image pro ⁇ cessing data) :
  • N_0 T_0 (for a first frame)
  • N_f+1 (l- ⁇ ) -N_f + p-T_f+l (for the next frames), wherein ⁇ is also a small number, for instance 0.1.
  • the goal of this filtering step is to remove the influence of moving objects in the field of view of the imaging system 10 from a more permanent effect on the lens due to soiling. Similar to a, the choice of ⁇ is important. A small value implies that higher filtering occurs but also that the detection takes more time to occur.
  • a soiling measure is determined by two variables S and C.
  • a second soiling variable C represents a measure for the confidence of the determined soiling status S. Both variables can be initialized with one ("1") when the imaging system 10 stops. After that, the variables are updated according to the following formulas:
  • C_f+1 C_f-(l-8-R) + ⁇ -R, wherein ⁇ is a coefficient similar to a and ⁇ .
  • the soiling measure comprising the two soiling variables S and C may be determined for each pixel of an im ⁇ age frame provides by the imaging system 10. Accordingly, separate soiling variables may be obtained for each pixel.
  • soiling can be detected based on the confidence val ⁇ ue C and the satisfactory S of the soiling if the confidence C is larger than a predetermined confidence threshold value and the satisfactory S is less than a predetermined soiling threshold value.
  • the image is clear if the confidence C is larger than the confidence threshold value and the soiling is larger than the soiling threshold.
  • the status is un ⁇ known if the confidence value C is less than the confidence threshold value.
  • step SI captures one or more first frames while the imaging system 10 is moving.
  • step S3 captures one or more second frames while the imaging sys ⁇ tem 10 is not moving.
  • This embodiment mainly differs from the previous embodiment in the way of computing the first and the second imaging processing data in steps S2 and S4.
  • the steps S2 and S4 for computing first image pro ⁇ cessing data and second image processing data comprises com ⁇ puting sharpness at each pixel of a frame provided by the im ⁇ aging system 10.
  • the steps S2 and S4 for computing first image pro ⁇ cessing data and second image processing data comprises com ⁇ puting sharpness at each pixel of a frame provided by the im ⁇ aging system 10.
  • any other method for determining the sharpness of a frame provided by the im ⁇ aging system 10 may be possible, too.
  • FIG. 3 illustrates schematic drawing for understanding the principle of a sharpness metric considering high frequency and low frequency derivatives.
  • High frequency derivatives HF consider a transition amount within a small distance of pixels
  • low frequency derivatives LF con ⁇ sider a transition amount over larger distances.
  • the low frequency deriva ⁇ tive LF which is given, for example, by a convolution of the image with a kernel [1 0 0 0 -1] will have approximately the same value as a high frequency derivative HF, which is given, for example, by a convolution of the image with a kernel [1 0 -1] .
  • a sharpness which is defined as a ratio of HF/LF will be approximately 1.
  • the sharpness will be smaller than 1, since the high frequency derivative HF is much smaller than the total edge size which is given by the low frequency derivative LF.
  • a first frame is captured by the imaging system 10 when the imaging system 10 stops (i.e. the end of the movement of the imaging system 10) .
  • a sharpness detection is per ⁇ formed by computing first image processing data classifying the sharpness of the first frame as mentioned above.
  • This classification of the sharpness for each pixel in the first frame can be stored as reference image data (first image pro ⁇ cessing data) .
  • the imaging system 10 further provides second frames. For these second frames, a classification of the sharpness for each pixel is performed, too.
  • the classification of the sharpness is the same as described above.
  • soiling detec ⁇ tion can be performed. For example, in a first step the first image processing data R and the second image processing data H are analyzed in order to determine if there is an edge in the first image processing data R (reference image data) and if this edge disappears in the second image processing data H (current image data) . In this case, a further array Dl with the size of the frames provided by the imaging system 10 is created, wherein an element of this array Dl is set to "1" if the above condition is fulfilled, otherwise the respective element is set to "0".
  • the respective element of D12 will be set to "1" if either Dl or D2 is 1.
  • this map D12 it is possible to determine entire areas that are connected and which are candidates for detecting a soiling. Such connected areas relate to large areas which can be aggregated and treated as a single entity, for example a segment.
  • the con ⁇ nected areas may have, for example a shape of a raindrop or another known element causing soiling on the imaging system 10.
  • Figure 4 illustrates a map comprising such a connected area A.
  • this area N is used to update a potential soiling map. This can be performed similar to the previous embodiment by updating soiling variables.
  • the soiling variables S and C can be updated based on the following formula
  • FIG. 5 shows a schematic illustration of a soiling detec ⁇ tion apparatus according to an embodiment.
  • the apparatus com ⁇ prises an image receiving unit 1, an image data processor 2 and a soiling detector 3.
  • the image receiving unit 1 is adapted to receive the first frame captured by the imaging system 10 while the imaging system 10 is moving. Further, the image receiving 1 is further adapted to receive a second frame, captured by the imaging system 10, while the imaging system 10 is not moving.
  • the image data processor 2 is adapted to compute first image processing data based on a re ⁇ ceived first frame and to compute second image processing da ⁇ ta based on a received second frame.
  • the computation of the first and the second image processing data can be performed, for instance, as already described above in connection with the soiling detection method.
  • the soiling detector 3 is adapted to determine a soiling measure.
  • the determination of the soiling measure can be performed as already described above in connection with the soiling detection method.
  • the determination of the soiling measure can be performed for each pixel position by considering the first image processing data and the second image processing data. Based on these image processing data, a soiling measure can be computed comprising two soiling variables, in particular a first value specifying a degree of soiling and a second value specifying a confidence value for the first value.
  • This soiling detection apparatus can be employed, for example, in an image processing system, for example in a camera system of a vehicle.
  • the image processing system may be part of an advanced driver assistance system for fully or partially automated driving of a vehicle.
  • the present invention relates to a soiling detection method for detecting soiling of an imaging system if the imaging system is standing still. It is for this purpose, that reference image data are generated while the imaging system is moving and based on this reference image data a further analysis of a captured image data can be performed while the imaging system is standing still. By comparing the reference data with image data generated while the imaging system is standing still, a soiling detection measure can be computed comprising a degree of soiling and a confidence val ⁇ ue for this degree. In this way, a fast and reliable soiling detection of a still standing imaging system can be performed .

Landscapes

  • Engineering & Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
  • Traffic Control Systems (AREA)
  • Studio Devices (AREA)
  • Image Analysis (AREA)

Abstract

The present invention provided a soiling detection method for detecting soiling of an imaging system if the imaging system is standing still. It is for this purpose, that reference image data are generated while the imaging system is moving and based on this reference image data a further analysis of a captured image data can be performed while the imaging system is standing still. By comparing the reference data with image data generated while the imaging system is standing still, a soiling detection measure can be computed comprising a degree of soiling and a confidence value for this degree. In this way,a fast and reliable soiling detection of a still standing imaging system can be performed.

Description

METHOD AND APPARATUS FOR SOILING DETECTION, IMAGE PROCESSING SYSTEM AND ADVANCED DRIVER ASSISTANCE SYSTEM
The present invention relates to a method and an apparatus for soiling detection on an image system. Further, the present invention relates to an image processing system and an advanced driver assistance system comprising an image pro¬ cessing system.
Prior art
Present day passenger vehicles are increasingly equipped with camera-based assistance systems. These systems capture the surrounding environment of the vehicle, and provide a variety of functions for improving driving safety and comfort. The functionality of these systems is based on the analysis of the recorded image data. Therefore, the quality of the sys¬ tem' s predictions is directly related to the quality of the image data.
One factor that affects the image quality and which is diffi¬ cult to control is the degree of contamination of the optical system of the camera. Cameras may be positioned at places with less risk of contamination, or cameras may be cleaned, for instance, by an electric viper. Despite of these provi¬ sions, it is impossible to avoid the contamination of the op¬ tical system completely. Therefore, it is necessary to detect soiling on a camera lens automatically in order to trigger an appropriate action.
EP 2 351 351 Al discloses a method for detecting the presence of an impediment on a lens of an image capture device. At least two image frames are captured to detect similar por¬ tions in the image frames. Presence of an impediment on the lens is determined by analyzing the luminance of the respec¬ tive area. In particular, it is suggested that the method is performed in a vehicle which is moving at a speed in the range of 5 km/h to 30 km/h.
Disclosure of the invention
The present invention provides a method for soiling detection on an image system with the features of independent claim 1. Further, the present invention provides a soiling detection apparatus with the features of independent claim 9.
In a first aspect, the present invention provides a method for soiling detection on an imaging system comprising the steps of capturing, by the imaging system, a first frame, while the imaging system is moving; computing first image processing data based on the captured first frame; capturing, by the imaging system, a second frame, while the imaging sys¬ tem is not moving; computing second image processing data based on the captured second frame; and determining a soiling measure for each pixel position covered by the imaging system based on the computed first image processing data and the computed second image processing data, wherein a said soiling measure comprising a first value specifying a degree of soil¬ ing and a second value specifying a confidence value of the first value.
According to a second aspect, the present invention provides a soiling detection apparatus for detecting soiling on an imaging system. The apparatus comprises an image receiving unit, an image data processor, and a soiling detector. The image receiving unit is adapted to receive a first frame, which is captured by the imaging system while the imaging system is moving. The image receiving unit is further adapted to receive a second frame, captured by the imaging system while the imaging system is not moving. The image data pro¬ cessor is adapted to compute first image processing data based on the received first frame and compute second image processing data based on the received second frame. The soil- ing detector is adapted to determine a soiling measure for each pixel position covered by the imaging system based on the computed first image processing data and the computed second image processing data. The soiling measure comprises a first value specifying a degree of soiling and a second value specifying a confidence value of the first value.
Further, the present invention provides an image processing system comprising an imaging system including a camera. The imaging system is adapted to capture image data. The image processing system further comprises a soiling detection apparatus according to the second aspect of the present inven¬ tion.
Further, the present invention provides an advanced driver assistance system comprising an image processing system according with a soiling detection apparatus.
Advantages of the invention
When using image data as a basis for security-related appli¬ cations, for instance, automated control of a vehicle, it is important to detect soiling on the imaging system. Some conventional methods are able to detect soiling when the vehicle is moving. However, these methods cannot perform soiling detection when the vehicle is standing still. Hence, when a ve¬ hicle is standing still at an intersection and another vehicle splashes water onto the lens, such a contamination cannot be detected by a soiling detection method, since a conven¬ tional soiling detection method requires a moving vehicle.
It is an idea of the present invention, to take into account the above-mentioned problem and to provide soiling detection on an imaging system which is able to detect a contamination of the imaging system while the imaging system is standing still. Accordingly, reference image data are generated while the imaging system is moving, and these reference image data are used as a basis for analyzing the image data which are captured when the imaging system is standing still. In particular, the reference image data as well as the image data which are captured while the imaging system stands still, are processed in order to remove elements in the image data caused by moving objects in the coverage area of the imaging system. In this way, it is possible to obtain image pro¬ cessing data without any elements of moving objects. Thus, a reliable detection of artifacts caused by a contamination of the imaging system can be achieved.
Hence, a fast and reliable soiling detection of an imaging system is provided. In this way, the reliability of the indi¬ vidual pixels of the image data provided by the imaging sys¬ tem can be evaluated. Hence, the reliability of the image da¬ ta provided by the imaging system can be increased.
The soiling measure which is provided by the present inven¬ tion comprises at least two values. A first value specifies a degree of soiling for each pixel provided by the imaging sys¬ tem. Additionally, a second value specifies a confidence val¬ ue of the first value. In this way, the reliability of the detected soiling of the imaging system can be further specified .
According to an embodiment, the soiling measure is updated iteratively. In this way, continuous evaluation of the pro¬ vided image data and the soiling of the imaging system can be achieved .
According to an embodiment, determining the soiling measure comprises applying an infinite impulse response (IIR) filter of the first value and/or the second value of the soiling measure. In this way, the reliability of the soiling measure can be further increased. According to a further embodiment, computing the first image processing data and computing the second image processing data comprises computing a transition between a current frame and a previous frame. In this way, static elements in the im¬ age data can be eliminated. Hence, the further processing of the image processing data only has to take into account changes in the image data.
According to an embodiment, computing a transition between the current frame and the previous frame comprises applying a Laplacian filter. A Laplacian filter is a very efficient and reliable way for filtering static elements in the image data.
According to a further embodiment, computing the first image processing data and the second image processing data compris¬ es applying an infinite impulse response filtering to the computed transition between the current frame and the previ¬ ous frame. By applying such an infinite impulse response fil¬ tering, the effects of moving objects in the coverage area of the imaging system can be removed.
According to a further embodiment, computing first image pro¬ cessing data and computing second image processing data comprises computing a sharpness of a captured first frame and a captured second frame. For example, the sharpness of a cap¬ tured frame can be classified into a plurality of predeter¬ mined classes. For example, sharpness can be classified into "no edge", "blurred edge" and "sharp edge".
According to an embodiment, determining a soiling measure comprises comparing the computed sharpness of the first frame and the computed sharpness of the second frame.
Brief description of the drawings The subject-matter of the present specification will be now explained in further detail with respect to the following figures, in which:
Figure 1 shows a schematic drawing of a vehicle compris¬ ing an imaging system according to an embodiment ;
Figure 2 shows a schematic diagram of a flowchart of a soiling detection method underlying an embodiment ;
Figure 3 shows a schematic diagram for determining sharpness in a frame according to an embodiment;
Figure 4 shows a schematic illustration of image data
provided by an imaging system according to an embodiment; and
Figure 5 shows a schematic illustration of a soiling de¬ tection apparatus according to an embodiment.
Description of embodiments
In the following description, details are provided to de¬ scribe the embodiments of the present invention. It shall be apparent to once skilled in the art, however, that the embod¬ iments may be also practiced without such details.
Figure 1 shows an image processing system according to an embodiment of the present invention. The image processing sys¬ tem comprises an imaging system 10. For example, the image processing system may be a part of an advanced driver assis¬ tance system (ADAS) of a vehicle 20. However, the imaging system 10 and the later described soiling detection apparatus of the image processing system is not limited to an ADAS or any other application of a vehicle. Furthermore, soiling de- tection of an imaging system may be also applied to any other image processing system.
The image processing system of Figure 1 comprises an imaging system 10. The imaging system 10 includes a camera 11 and a lens 12. Camera 11 captures image data and provides the image data to a further device for processing the image data. In particular, the image data may be provided in form of image frames. Camera 11 may capture image data in form of frames having a predetermined resolution. Further, camera 11 may capture frames in regular time intervals. Alternatively, any other form of capturing image data is possible, too.
During operation of the imaging system 10, the optical system of the imaging system 10 may be contaminated. For example, dust, particles, raindrops or any other kind of contamination may be located on lens 12. In order to avoid contamination of lens 12, the imaging system 10 may be located behind a wind¬ shield of a vehicle or any other transparent protecting ele¬ ment. In this case, the contamination 13 may be located on the windshield or the further protecting element. Due to this contamination 13, the image data captured by camera 11 may be disturbed .
For example, a vehicle 20 may stand still at an intersection, and another vehicle may splash water onto the windshield of the vehicle or on the lens 12 of imaging system 10. As a re¬ sult, the image areas covered by this contamination do not provide appropriate image data for a further processing, in particular for use in an ADAS.
Figure 2 illustrates a flowchart of a method for soiling de¬ tection on the imaging system 10 underlying an embodiment of the present invention. In a first step SI, the imaging system 10 captures image data while the imaging system 10 is moving. For example, the imaging system 10 may successively capture first frames while the imaging system 10 is moving. Based on the captured first frame, or based on a plurality of succes¬ sively captured first frames, in step S2 first image pro¬ cessing data are computed. Several different methods or algo¬ rithm may be used for computing the first image processing data. Some examples for computing first image processing data will be described below.
Capturing SI of first frames and computing S2 of first image processing data is performed until the imaging system 10 stops moving. If the imaging system 10 stands still, in step S3 one or a plurality of successive second frames are cap¬ tured. Based on the captured second frames, in step S4 second image processing data are computed. The computing of the sec¬ ond image processing data may be similar to the computing S2 of the first image processing data and will be also described below in more detail.
Finally, in step S5 a soiling measure is determined based on the computed first image processing data and the computed second image processing data. For this purpose, the first im¬ age processing data may be used as a reference image data and the second image processing data may be used as current image data. By computing a measure between the reference image data and the current image data, the determining step S5 for de¬ termining a soiling measure computes a first value specifying a degree soiling in the image data relating to the second frame, and a second value specifying a confidence value for the determined degree of soiling. In particular, the soiling measure may be computed separately for each pixel in the im¬ age data of the frames provided by the imaging system 10.
In the following, an example for soiling detection on an imaging system 10 is described. This first example is based on a computation of transitions in the image data of the frames provided by the imaging system 10. However, as will be seen further below, the soiling detection may be also performed by other analysis of the frames provided by the imaging system 10.
For each frame I provided by the imaging system 10, a magnitude of derivatives E is computed. Since the direction of the derivatives is not relevant for this method, the computation can be performed, for instance, by applying a simple Laplaci- an filter F to the image data of a frame I:
wherein: F
Figure imgf000010_0001
is a Laplacian filter.
Since the actual value of the derivatives is not important, it is sufficient to determine a transition matrix T. This transition matrix T can be determined, for instance, by the following formula:
T = min(l, E / ThEdge) , wherein ThEdge is a threshold value that guarantees that a clear transition is detected.
Accordingly, transition T represents a matrix having the size of the frames provided by imaging system 10, wherein each el¬ ement of this matrix represents the probability of a transi¬ tion for the corresponding pixel.
In order to perform soiling detection on the imaging system 10, two sets of image processing data are used. The first im¬ age processing data sets are computed based on first frames provided by the imaging system 10 while the imaging system 10 is moving. Alternatively, if the method is started when the imaging system 10 is already standing still, a single first frame is captured and first image processing data sets are computed based on this single frame. Accordingly, first image processing data are computed as reference image data by the following formula:
R_0 = T_0 (for a first frame) , and
R_f+1 = (1- ) -R_f + cx-T_f+l (for the next frames), wherein f and f+1 are the number of frames, and a is a coef¬ ficient between 0 and 1. In particular, a small value, for example 0.1 may be used for a . In this way, an infinite im¬ pulse response (IIR) filter is applied to the transition im¬ age T. The goal of this filtering is to remove the influence of moving objects in the field of view of the imaging system 10. A small value of a implies that higher filtering occurs. However, small values of a also may lead to the effect, that the past, where the imaging system 10 was observing something different, has bigger influence to the final reference image data .
This reference image data (first image processing data) is updated based on the frames provided by the imaging system 10 as long as the imaging system 10 is moving. If the imaging system stops, the updating of the reference image data stops, too. When the imaging system 10 starts moving again, updating of the reference image data resumes.
If the imaging system 10 stops moving, second image pro¬ cessing data N is generated. The calculation of the second image processing data N is performed in a similar way to the calculation of the reference image data (first image pro¬ cessing data) :
N_0 = T_0 (for a first frame) , and
N_f+1 = (l-β ) -N_f + p-T_f+l (for the next frames), wherein β is also a small number, for instance 0.1. The goal of this filtering step is to remove the influence of moving objects in the field of view of the imaging system 10 from a more permanent effect on the lens due to soiling. Similar to a, the choice of β is important. A small value implies that higher filtering occurs but also that the detection takes more time to occur.
After the first image processing data (reference image data) and the second image processing data (current image data) have been calculated, detection of soiling is performed simp¬ ly by updating soiling variables using the first and the sec¬ ond image processing data R and N. In particular, a soiling measure is determined by two variables S and C. A first soil¬ ing variable S represents a measure how satisfactory the soiling status is: S = 1 means that the lens is clear, and S = 0 means that the lens is soiled. A second soiling variable C represents a measure for the confidence of the determined soiling status S. Both variables can be initialized with one ("1") when the imaging system 10 stops. After that, the variables are updated according to the following formulas:
S_f+1 = S_f-(l-8-R) + s-R-N_f+l, and
C_f+1 = C_f-(l-8-R) + ε-R, wherein ε is a coefficient similar to a and β .
In particular, the soiling measure comprising the two soiling variables S and C may be determined for each pixel of an im¬ age frame provides by the imaging system 10. Accordingly, separate soiling variables may be obtained for each pixel.
If a transition is not present in the reference image data (first image processing data) for a pixel (i.e. R(x, y) = 0), then the two soiling variables S and C are not updated. This means, that there is no detail in the reference image data at this position and consequently, if there is soiling on the lens on that location, it is not possible to detect this soiling .
If a transition is present in the reference image data (first image data) for a pixel (i.e. R(x, y) = 1), then the first soiling variable S is updated with a portion of the value of the second image processing data corresponding to the current image data N. If the second image processing data also has a transition, then the satisfactory S will tend towards being classified as clear (i.e. S = 1) . If the current image is not clear, the satisfactory S will start tending towards not being clear (i.e. S = 0) . Since there is a transition in the first image data (reference image data) a confidence value C can be computed.
Finally, soiling can be detected based on the confidence val¬ ue C and the satisfactory S of the soiling if the confidence C is larger than a predetermined confidence threshold value and the satisfactory S is less than a predetermined soiling threshold value. The image is clear if the confidence C is larger than the confidence threshold value and the soiling is larger than the soiling threshold. Finally, the status is un¬ known if the confidence value C is less than the confidence threshold value.
In the following, a further example for soiling detection on an imaging system 10 is described. This example considers edge sharpness in the image data provided by the frames of the imaging system 10. The steps SI and S2 for capturing a first frame and a second frame by the imaging system 10 are similar to the capturing of the first and the second frame in the previous example. In particular, step SI captures one or more first frames while the imaging system 10 is moving. Step S3 captures one or more second frames while the imaging sys¬ tem 10 is not moving. This embodiment mainly differs from the previous embodiment in the way of computing the first and the second imaging processing data in steps S2 and S4. In partic- ular, the steps S2 and S4 for computing first image pro¬ cessing data and second image processing data comprises com¬ puting sharpness at each pixel of a frame provided by the im¬ aging system 10. In the following one example for computing the sharpness will be presented. However, any other method for determining the sharpness of a frame provided by the im¬ aging system 10 may be possible, too.
The basic idea behind the sharpness metric in this example is to divide a high frequency transition amount by the total size of an edge. Figure 3 illustrates schematic drawing for understanding the principle of a sharpness metric considering high frequency and low frequency derivatives. High frequency derivatives HF consider a transition amount within a small distance of pixels, while low frequency derivatives LF con¬ sider a transition amount over larger distances.
In a scenario having a sharp edge, the low frequency deriva¬ tive LF, which is given, for example, by a convolution of the image with a kernel [1 0 0 0 -1] will have approximately the same value as a high frequency derivative HF, which is given, for example, by a convolution of the image with a kernel [1 0 -1] . Thus, a sharpness which is defined as a ratio of HF/LF will be approximately 1. In a blurred edge like the example of Figure 2, the sharpness will be smaller than 1, since the high frequency derivative HF is much smaller than the total edge size which is given by the low frequency derivative LF.
This principle is only valid when the low frequency deriva¬ tive LF captures the size of the entire edge, which does not happen if the image resolution is too high with regard to the low frequency kernel that is being used and/or the kernel is shifted with regard to the center of the edge. If necessary, further processing needs to occur to guarantee that the out¬ put of the sharpness detector truly indicates the sharpness of the image for each pixel. However, this processing is well known and will not be discussed in further detail. In the following, it is assumed that each pixel of the image is classified as being part of either no edge (value "0") , having a blurred edge (value "1") or having a sharp edge (value "2") . However, in principle, another classification may be also possible, if the soiling detection method is adapted accordingly.
In order to perform soiling detection, a first frame is captured by the imaging system 10 when the imaging system 10 stops (i.e. the end of the movement of the imaging system 10) . Based on this first image, a sharpness detection is per¬ formed by computing first image processing data classifying the sharpness of the first frame as mentioned above. This classification of the sharpness for each pixel in the first frame can be stored as reference image data (first image pro¬ cessing data) .
While the imaging system 10 is standing still, the imaging system 10 further provides second frames. For these second frames, a classification of the sharpness for each pixel is performed, too. The classification of the sharpness is the same as described above.
Based on a calculated sharpness R of the first frame and a calculated sharpness H of the second frames, soiling detec¬ tion can be performed. For example, in a first step the first image processing data R and the second image processing data H are analyzed in order to determine if there is an edge in the first image processing data R (reference image data) and if this edge disappears in the second image processing data H (current image data) . In this case, a further array Dl with the size of the frames provided by the imaging system 10 is created, wherein an element of this array Dl is set to "1" if the above condition is fulfilled, otherwise the respective element is set to "0". This can be represented by the follow¬ ing formula: If ((R(x,y)=l) or (R(x,y)=2)) and (H(x,y)=0) then Dl(x,y)=l; else Dl (x, y) =0
Further, it is determined if a pixel which does not have an edge or does not have a sharp edge in the first image pro¬ cessing data R becomes blurred in the second image processing data H. In this case, a respective element of a further ma¬ trix D2 is set to "1". Otherwise, the respective element is set to "0". This can be represented by the following formula:
If ((R(x,y)≠l) and (H(x,y)=l) then D2(x,y)=l;
else D2 (x, y) =0
Based on these two additional maps Dl and D2, the detection of a soiled area can be performed in two steps. First, a map D12(x,y) is created by
D12(x,y) = max (Dl (x, y) , D2(x,y)).
In other words, the respective element of D12 will be set to "1" if either Dl or D2 is 1. With use of this map D12 it is possible to determine entire areas that are connected and which are candidates for detecting a soiling. Such connected areas relate to large areas which can be aggregated and treated as a single entity, for example a segment. The con¬ nected areas may have, for example a shape of a raindrop or another known element causing soiling on the imaging system 10.
Figure 4 illustrates a map comprising such a connected area A.
In a further step, only those areas are considered as soiled areas, in which at least one pixel inside such an area is newly blurred, i.e. at least one element of D2 (x y) = 1. The reason behind this step is that soiling on a lens creates an area of no edge content, but also an area with blurred con¬ tent. Thus, it is expected that any soiling on an imaging system 10 will create some blurring. If no blurring is created close to edges that disappear, it might be because some object that was captured on the reference image (for example a person or a vehicle) has moved. Hence, such an element is not considered as a soiling area.
Once a blurred area has been found, this area N is used to update a potential soiling map. This can be performed similar to the previous embodiment by updating soiling variables. For example, the soiling variables S and C can be updated based on the following formula
S_f+1 = S_f-(l-8-R) + s-R-N_f+l, and
C_f+1 = C_f-(l-8-R) + ε-R,
This formula has been already explained above. A similar way to take this information into account is simply to compute a variable which indicates how much particular region is soiled by a simple infinite impulse response based on the following formula
S_f+1 = S_f-(l-sR) + s-N_f+l.
Figure 5 shows a schematic illustration of a soiling detec¬ tion apparatus according to an embodiment. The apparatus com¬ prises an image receiving unit 1, an image data processor 2 and a soiling detector 3. The image receiving unit 1 is adapted to receive the first frame captured by the imaging system 10 while the imaging system 10 is moving. Further, the image receiving 1 is further adapted to receive a second frame, captured by the imaging system 10, while the imaging system 10 is not moving. The image data processor 2 is adapted to compute first image processing data based on a re¬ ceived first frame and to compute second image processing da¬ ta based on a received second frame. The computation of the first and the second image processing data can be performed, for instance, as already described above in connection with the soiling detection method.
The soiling detector 3 is adapted to determine a soiling measure. The determination of the soiling measure can be performed as already described above in connection with the soiling detection method. In particular, the determination of the soiling measure can be performed for each pixel position by considering the first image processing data and the second image processing data. Based on these image processing data, a soiling measure can be computed comprising two soiling variables, in particular a first value specifying a degree of soiling and a second value specifying a confidence value for the first value.
This soiling detection apparatus can be employed, for example, in an image processing system, for example in a camera system of a vehicle. In particular, the image processing system may be part of an advanced driver assistance system for fully or partially automated driving of a vehicle.
Summarizing, the present invention relates to a soiling detection method for detecting soiling of an imaging system if the imaging system is standing still. It is for this purpose, that reference image data are generated while the imaging system is moving and based on this reference image data a further analysis of a captured image data can be performed while the imaging system is standing still. By comparing the reference data with image data generated while the imaging system is standing still, a soiling detection measure can be computed comprising a degree of soiling and a confidence val¬ ue for this degree. In this way, a fast and reliable soiling detection of a still standing imaging system can be performed .

Claims

PATENTANSPRliCHE
1. A method for soiling detection on an imaging system
(10), comprising the steps of: capturing (SI), by the imaging system (10), a first frame, while the imaging system (10) is moving; computing (S2) first image processing data based on the captured first frame; capturing (S3), by the imaging system (10), a second frame, while the imaging system (10) is not moving; computing (S4) second image processing data based on the captured second frame; and determining (S5) a soiling measure for each pixel position covered by the imaging system (10) based on the computed first image processing data and the computed second image processing data, wherein said soiling meas¬ ure comprises a first value specifying a degree of soil¬ ing and a second value specifying a confidence value of the first value.
2. The method according to claim 1, wherein soiling measure is updated iteratively.
3. The method according to claim 1 or 2, wherein determining (S5) a soiling measure comprises applying an infinite impulse filtering of the first value and/or the second value of the soiling measure.
4. The method according to any of claims 1 to 3, wherein computing (SI) first image processing data and computing (S2) second image processing data comprises computing a transition between a current frame and a previous frame.
The method according to claim 4, wherein computing a transition between the current frame and the previous frame comprises applying a Laplacian filter.
The method according to claims 4 or 5, wherein computing
(51) first image processing data and computing (S2) sec¬ ond image processing data comprises applying an infinite impulse response filtering to the computed transition between the current frame and the previous frame.
The method according to any of claims 1 to 3, wherein computing (SI) first image processing data and computing
(52) second image processing data comprises computing sharpness of a captured first frame and a captures sec¬ ond frame .
The method according to claim 7, wherein determining (S5) a soiling measure comprises comparing the computed sharpness of the first frame and the computed sharpness of the second frame.
A soiling detection apparatus for detecting soiling on an imaging system (10), the apparatus comprising: an image receiving unit (1), adapted to receive a first frame, captured by the imaging system (10) while the im¬ aging system (10) is moving, and to receive a second frame, captured by the imaging system (10) while the im¬ aging system (10) is not moving; an image data processor (2) adapted to compute first im¬ age processing data based on the received first frame, and to compute second image processing data based on the received second frame; and a soiling detector (3) , adapted to determine a soiling measure for each pixel position covered by the imaging system (10) based on the computed first image processing data and the computed second image processing data, wherein said soiling measure comprises a first value specifying a degree of soiling and a second value speci¬ fying a confidence value of the first value.
10. An image processing system, comprising an imaging system (10) including a camera (11), adapted to capture image data; a soiling detection apparatus according to claim 9.
11. An advanced driver assistance system comprising an image processing system according to claim 10.
PCT/EP2017/068612 2016-08-08 2017-07-24 Method and apparatus for soiling detection, image processing system and advanced driver assistance system Ceased WO2018028974A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
EP16183187.0A EP3282420B1 (en) 2016-08-08 2016-08-08 Method and apparatus for soiling detection, image processing system and advanced driver assistance system
EP16183187.0 2016-08-08

Publications (1)

Publication Number Publication Date
WO2018028974A1 true WO2018028974A1 (en) 2018-02-15

Family

ID=56740858

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2017/068612 Ceased WO2018028974A1 (en) 2016-08-08 2017-07-24 Method and apparatus for soiling detection, image processing system and advanced driver assistance system

Country Status (2)

Country Link
EP (1) EP3282420B1 (en)
WO (1) WO2018028974A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2024537563A (en) * 2021-08-31 2024-10-11 メルセデス・ベンツ グループ アクチェンゲゼルシャフト Method for detecting optical blockage of a vehicle camera and method for operating a vehicle - Patents.com

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2351351A1 (en) 2008-10-01 2011-08-03 HI-KEY Limited A method and a system for detecting the presence of an impediment on a lens of an image capture device to light passing through the lens of an image capture device

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102012222126B4 (en) * 2012-12-04 2020-06-04 Robert Bosch Gmbh Method and device for controlling a cleaning device for a vehicle camera, method for cleaning a vehicle camera and camera system for a vehicle

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2351351A1 (en) 2008-10-01 2011-08-03 HI-KEY Limited A method and a system for detecting the presence of an impediment on a lens of an image capture device to light passing through the lens of an image capture device

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
CHEN LI ET AL: "Dust particle detection in traffic surveillance video using motion singularity analysis", DIGITAL SIGNAL PROCESSING, ACADEMIC PRESS, ORLANDO,FL, US, vol. 58, 1 August 2016 (2016-08-01), pages 127 - 133, XP029727491, ISSN: 1051-2004, DOI: 10.1016/J.DSP.2016.07.020 *
KATAJAMAEKI J ET AL: "OBJECTIVE QUALITY POTENTIAL MEASURES OF NATURAL COLOR IMAGES", JOURNAL OF IMAGING SCIENCE AND TECHNOLOGY, SPIE - THE INTERNATIONAL SOCIETY FOR OPTICAL ENGINEERING, US, vol. 42, no. 3, 1 May 1998 (1998-05-01), pages 250 - 263, XP000979385, ISSN: 1062-3701 *
S K SATPATHY ET AL: "Adaptive Non-Linear Filtering Techniques for Image Restoration", INTERNATIONAL JURNAL OF ELECTRICAL, COMPUTER, ENERGETIC AND COMMUNICATION ENGINEERING, 1 January 2003 (2003-01-01), pages 948 - 955, XP055340549, Retrieved from the Internet <URL:http://www.waset.org/publications/15180> [retrieved on 20170131] *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2024537563A (en) * 2021-08-31 2024-10-11 メルセデス・ベンツ グループ アクチェンゲゼルシャフト Method for detecting optical blockage of a vehicle camera and method for operating a vehicle - Patents.com
JP7649921B2 (en) 2021-08-31 2025-03-21 メルセデス・ベンツ グループ アクチェンゲゼルシャフト Method for detecting optical blockage of a vehicle camera and method for operating a vehicle - Patents.com

Also Published As

Publication number Publication date
EP3282420A1 (en) 2018-02-14
EP3282420B1 (en) 2022-05-04

Similar Documents

Publication Publication Date Title
US11138698B2 (en) Static soiling detection and correction
JP4353127B2 (en) Rain sensor
US8797417B2 (en) Image restoration method in computer vision system, including method and apparatus for identifying raindrops on a windshield
US8391612B2 (en) Edge detection with adaptive threshold
US9934442B2 (en) Passenger counting device, passenger counting method, and program recording medium
EP2351351B1 (en) A method and a system for detecting the presence of an impediment on a lens of an image capture device to light passing through the lens of an image capture device
JP6744336B2 (en) Detection of lens contamination using expected edge trajectories
CN111860120B (en) Automatic occlusion detection method and device for vehicle-mounted cameras
US7646890B2 (en) Image processor for automotive vehicle
CN112215794B (en) Method and device for detecting dirt of binocular ADAS camera
JP3444192B2 (en) Imaging environment estimation device
US8520953B2 (en) Apparatus and method for extracting edges of image
US20110013839A1 (en) Procede de determination d&#39;une region d&#39;interet dans une image
CN115035000A (en) Road raise dust image identification method and system
JP4087600B2 (en) Image monitoring device
WO2018028974A1 (en) Method and apparatus for soiling detection, image processing system and advanced driver assistance system
JP2020088420A (en) Photographing abnormality diagnostic device and vehicle
EP3336747A1 (en) Rain detection with a camera
CN116682111B (en) Intelligent vehicle camera dirt detection methods, operating systems and in-vehicle equipment
CN114757947B (en) Method, device and system for detecting fouling of camera lens
EP3306522A1 (en) Device for determining a region of interest on and/or within a vehicle windscreen
JP2020160735A (en) Vehicle and imaging device
JP3081660B2 (en) Distance detection method
KR101728670B1 (en) AVM system having a function of identifying foreign material on camera and method thereof
CN117351402A (en) Video stream identification method and device and readable storage medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17746036

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17746036

Country of ref document: EP

Kind code of ref document: A1