CN113468976A - Garbage detection method, garbage detection system and computer readable storage medium - Google Patents

Garbage detection method, garbage detection system and computer readable storage medium Download PDF

Info

Publication number
CN113468976A
CN113468976A CN202110646613.2A CN202110646613A CN113468976A CN 113468976 A CN113468976 A CN 113468976A CN 202110646613 A CN202110646613 A CN 202110646613A CN 113468976 A CN113468976 A CN 113468976A
Authority
CN
China
Prior art keywords
target
detected
garbage
preset threshold
detection
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110646613.2A
Other languages
Chinese (zh)
Inventor
白家男
章合群
周祥明
傅凯
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang Dahua Technology Co Ltd
Original Assignee
Zhejiang Dahua Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang Dahua Technology Co Ltd filed Critical Zhejiang Dahua Technology Co Ltd
Priority to CN202110646613.2A priority Critical patent/CN113468976A/en
Publication of CN113468976A publication Critical patent/CN113468976A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/21Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/24Classification techniques
    • G06F18/241Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/20Analysis of motion
    • G06T7/246Analysis of motion using feature-based methods, e.g. the tracking of corners or segments

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Probability & Statistics with Applications (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

The application discloses a garbage detection method, a garbage detection system and a computer readable storage medium, wherein the garbage detection method comprises the following steps: acquiring a plurality of video frames, and acquiring a tracking result of a target to be detected in a target detection area from each video frame; wherein the tracking result comprises the category of the target to be detected; responding to the fact that the class of the target to be detected is a garbage can and the overflow confidence coefficient of the target to be detected is larger than a first preset threshold value, or responding to the fact that the class of the target to be detected is exposed garbage and the detection classification joint confidence coefficient of the target to be detected is larger than a second preset threshold value, taking the target to be detected as an alarm target, and accumulating the time of video frames corresponding to all the alarm targets to obtain first accumulated time; and responding to the situation that the first accumulated time is larger than a third preset threshold value, emptying the first accumulated time, and reporting all alarm targets for alarm display. Therefore, the alarm accuracy rate of the overflow and the garbage exposure of the garbage can be improved, and the influence of the false detection of individual video frames on result judgment is avoided.

Description

Garbage detection method, garbage detection system and computer readable storage medium
Technical Field
The present application relates to the field of computer vision technologies, and in particular, to a spam detection method, a spam detection system, and a computer-readable storage medium.
Background
In recent years, governments in various places strive to advance the process of national civilized city construction, and a large number of garbage cans are additionally arranged in urban roads, commercial districts with large people flow and residential districts. People can produce a lot of garbage in daily life, the garbage can is easily overflowed to cause various garbage to be stacked together, the appearance of a city is influenced, a civilized city is not easy to construct, pungent peculiar smell can be generated, bacteria are easily bred to cause air pollution, and particularly, the situation is more serious in hot summer. Therefore, it is necessary to identify the garbage bin overflowing or exposed and notify the relevant personnel to clean the garbage bin.
The method comprises the steps of carrying out region detection on an obtained original image to obtain a target region in the original image, carrying out target detection on the target region to obtain position information and type information of garbage, wherein the garbage detection region is a simple region, and the target region is a simple region.
Disclosure of Invention
The technical problem mainly solved by the application is to provide a garbage detection method, a garbage detection system and a computer readable storage medium, which can improve the alarm accuracy rate of garbage can overflow and garbage exposure.
In order to solve the technical problem, the application adopts a technical scheme that: provided is a garbage detection method, including: acquiring a plurality of video frames, and acquiring a tracking result of a target to be detected in a target detection area from each video frame; wherein the tracking result comprises the category of the target to be detected; responding to the fact that the category of the target to be detected is a garbage can and the overflow confidence coefficient of the target to be detected is larger than a first preset threshold value, or responding to the fact that the category of the target to be detected is exposed garbage and the detection classification joint confidence coefficient of the target to be detected is larger than a second preset threshold value, taking the target to be detected as an alarm target, and accumulating the time of video frames corresponding to all the alarm targets to obtain first accumulated time; and responding to the situation that the first accumulated time is larger than a third preset threshold value, emptying the first accumulated time, and reporting all the alarm targets for alarm display.
After the step of obtaining the tracking result of the target to be detected in the target detection area from each video frame, the method further includes: in response to the fact that the category of the target to be detected is a garbage can and the overflow confidence coefficient of the target to be detected is smaller than or equal to the first preset threshold, or in response to the fact that the category of the target to be detected is exposed garbage and the detection classification joint confidence coefficient of the target to be detected is smaller than or equal to the second preset threshold, summarizing the target to be detected, and accumulating the time of video frames corresponding to all the target to be detected to obtain second accumulated time; and in response to the second accumulated time being greater than a fourth preset threshold, emptying the first accumulated time and the second accumulated time, and returning to the step of acquiring a plurality of video frames.
Before the step of taking the target to be detected as an alarm target and accumulating the time of the video frames corresponding to all the alarm targets to obtain a first accumulated time, the method comprises the following steps: setting an alarm flag bit of a video frame corresponding to the target to be detected as a first flag bit; before the step of summarizing the targets to be detected and accumulating the time of the video frames corresponding to all the targets to be detected to obtain a second accumulated time, the method comprises the following steps: and setting the alarm zone bit of the video frame corresponding to the target to be detected as a second zone bit.
Wherein, after the step of obtaining the tracking result of the target to be detected in the target detection area from each video frame, the method comprises the following steps: obtaining a standard value of a coordinate frame corresponding to each target to be detected in a current video frame and a coordinate frame corresponding to the target to be detected in an existing video frame through a matching algorithm; and in response to the fact that the standard value is larger than a fifth preset threshold value, setting the state bit of the target to be detected to be updated, and obtaining a tracking result of the target to be detected.
After the step of obtaining the standard value of the coordinate frame corresponding to each target to be detected in the current video frame and the standard value of the coordinate frame corresponding to the target to be detected in the existing video frame through the matching algorithm, the method further comprises the following steps: responding to the standard value smaller than or equal to a fifth preset threshold value, and judging whether the target to be detected is a tracking target appearing for the first time in all the video frames; if so, setting the state bit of the target to be detected as creation; otherwise, setting the state bit of the target to be detected as lost; after the step of setting the status bit of the target to be detected to be lost, the method comprises the following steps: and in response to the condition that the number of the lost video frames of the state bits of the target to be detected is larger than a sixth preset threshold value, setting the state bits of the target to be detected to be deleted and deleting the tracking result of the target to be detected.
The step of taking the target to be detected as an alarm target and accumulating the time of all the alarm targets corresponding to the video frames to obtain a first accumulated time includes: judging whether the center point coordinate of the coordinate frame corresponding to the target to be detected is located in the target detection area; if so, entering a step of responding to that the category of the target to be detected is a garbage can and the overflow confidence coefficient of the target to be detected is greater than a first preset threshold value, or responding to that the category of the target to be detected is exposed garbage and the detection classification joint confidence coefficient of the target to be detected is greater than a second preset threshold value, taking the target to be detected as an alarm target, and accumulating the time of video frames corresponding to all the alarm targets to obtain first accumulated time; otherwise, returning to the step of acquiring a plurality of video frames.
Before the step of taking the target to be detected as an alarm target and accumulating video frame time corresponding to all the alarm targets to obtain first accumulated time in response to the fact that the category of the target to be detected is a garbage can and the overflow confidence coefficient of the target to be detected is greater than a first preset threshold, the method comprises the following steps: acquiring a first coordinate frame corresponding to the garbage can in the target detection area; obtaining a second coordinate frame corresponding to the garbage can according to the first coordinate frame, wherein the top of the second coordinate frame extends upwards for a preset proportion of the height of the garbage can relative to the first coordinate frame; and inputting the image corresponding to the second coordinate frame into a first target classification network to obtain the overflow confidence corresponding to the garbage can.
Wherein, before the step of responding to that the category of the target to be detected is exposed garbage and the detection classification joint confidence coefficient of the target to be detected is greater than a second preset threshold, the step of taking the target to be detected as an alarm target and accumulating video frame time corresponding to all the alarm targets to obtain first accumulated time comprises the following steps: acquiring a third coordinate frame corresponding to the exposed garbage in the target detection area; inputting the image corresponding to the third coordinate frame into a second target classification network to obtain a garbage confidence coefficient corresponding to the exposed garbage; judging whether the category of the exposed garbage is garbage or not according to the garbage confidence coefficient; if so, obtaining a sum of a product of the first weight coefficient and a target detection confidence coefficient and a product of the second weight coefficient and the garbage confidence coefficient, and taking the sum as a detection classification joint confidence coefficient of the exposed garbage; otherwise, deleting the exposed garbage.
Before the step of obtaining the tracking result of the target to be detected in the target detection area from each video frame, the method further includes: acquiring the category of a target to be detected in the target detection area; and obtaining the target detection confidence corresponding to each target to be detected according to the category.
In order to solve the above technical problem, another technical solution adopted by the present application is: there is provided a garbage detection system comprising a memory and a processor coupled to each other, the memory having stored therein program instructions, the processor being configured to execute the program instructions to implement the garbage detection method mentioned in any of the above embodiments.
In order to solve the above technical problem, the present application adopts another technical solution: there is provided a computer-readable storage medium storing a computer program for implementing the spam detection method mentioned in any of the above embodiments.
Different from the prior art, the beneficial effects of the application are that: the method comprises the steps of obtaining a plurality of video frames, obtaining a tracking result of a target to be detected in a target detection area from each video frame, wherein the tracking result comprises the category of the target to be detected, taking the target to be detected as an alarm target when the category of the target to be detected is a garbage can and the overflow confidence coefficient of the target to be detected is greater than a first preset threshold, or taking the target to be detected as exposed garbage and the detection classification joint confidence coefficient of the target to be detected is greater than a second preset threshold, accumulating the time of all the alarm targets corresponding to the video frames to obtain first accumulated time, emptying the first accumulated time when the first accumulated time is greater than a third preset threshold, and reporting all the alarm targets for alarm display. By the aid of the design mode, alarm accuracy of overflow and garbage exposure of the trash can be improved, and influence of false detection of individual video frames on result judgment is avoided.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts. Wherein:
FIG. 1 is a schematic flow chart diagram illustrating an embodiment of a spam detection method according to the present application;
FIG. 2 is a schematic view of a target detection area;
FIG. 3 is a schematic flow chart illustrating an embodiment of the method before step S1 in FIG. 1;
FIG. 4 is a schematic flow chart diagram illustrating an embodiment after step S1 in FIG. 1;
FIG. 5 is a schematic flow chart diagram illustrating an embodiment after step S25 in FIG. 4;
FIG. 6 is a schematic flow chart illustrating an embodiment of the method before step S2 in FIG. 1;
FIG. 7 is a schematic flow chart illustrating an embodiment of the method before step S3 in FIG. 1;
FIG. 8 is a schematic flow chart diagram illustrating one embodiment of the method before step S2 or before step S3 in FIG. 1;
FIG. 9 is a block diagram of an embodiment of the garbage detection system of the present application;
FIG. 10 is a schematic structural diagram of an embodiment of the garbage detection system of the present application;
FIG. 11 is a schematic structural diagram of an embodiment of a computer-readable storage medium according to the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1-2, fig. 1 is a schematic flow chart of an embodiment of a spam detection method according to the present application, and fig. 2 is a schematic view of a target detection area. Specifically, the garbage detection method includes:
s1: and acquiring a plurality of video frames, and acquiring a tracking result of the target to be detected in the target detection area from each video frame.
Specifically, the tracking result includes the category of the target to be detected. Specifically, in this embodiment, an intelligent camera may be installed near the trash can area, and the camera needs to clearly capture real-time images in the trash can and the trash can, and decode data through a video encoding and decoding technology to obtain a plurality of video frames. A closed target detection area is set in each video frame, and as shown in fig. 2, a corresponding coordinate frame is set in the target detection area for a target to be detected. Of course, in other embodiments, the target to be detected may also be labeled, which is not limited in this application.
Specifically, in the present embodiment, please refer to fig. 3, and fig. 3 is a flowchart illustrating an embodiment before step S1 in fig. 1. Specifically, step S1 includes:
s10: and acquiring the category of the target to be detected in the target detection area.
Specifically, the Type of the object to be detectedODTo trash cans and exposed trash.
S11: and obtaining the target detection confidence corresponding to each target to be detected according to the category.
Specifically, in the present embodiment, the Type according to the object to be detected isODObtaining a target detection confidence Conf corresponding to a target to be detectedODWherein the target detection confidence ConfODIs a decimal fraction between 0 and 1.
In addition, in the present embodiment, please refer to fig. 4, and fig. 4 is a flowchart illustrating an embodiment after step S1 in fig. 1. Specifically, step S1 is followed by:
s20: and obtaining the standard value of the coordinate frame corresponding to each target to be detected in the current video frame and the coordinate frame corresponding to the target to be detected in the existing video frame through a matching algorithm.
Specifically, the method adopts a detection tracking mode to match each target to be detected in the current video frame with the target to be detected in the existing video frame through a matching algorithm according to the detection and classification result of each frame of image so as to form a new tracking track. Specifically, the standard value IOU of the coordinate frame corresponding to each target to be detected in the current video frame and the coordinate frame corresponding to the target to be detected in the existing video frame is obtained through calculation.
S21: and judging whether the standard value is larger than a fifth preset threshold value.
Specifically, it is determined whether the criterion value IOU is greater than a fifth preset threshold, where the fifth preset threshold is configured by the user according to an actual situation, and the application is not limited herein.
S22: and if so, setting the state bit of the target to be detected as an update, and obtaining the tracking result of the target to be detected.
Specifically, in this embodiment, if the standard value IOU is greater than the fifth preset threshold, which indicates that the target to be detected and the target to be detected in the previous existing video frame are successfully matched, the state bit of the target to be detected is set to be the Update, and the tracking result of the target to be detected is obtained. Of course, in other embodiments, when the coordinate frame corresponding to the target to be detected in the current video frame can be matched with the coordinate frame corresponding to the target to be detected in the plurality of existing video frames, the video frame corresponding to the tracking coordinate frame with the maximum standard value IOU is taken as the tracking target.
S23: otherwise, judging whether the target to be detected is the tracking target appearing for the first time in all the video frames.
Specifically, in this embodiment, if the standard value IOU is less than or equal to the fifth preset threshold, it indicates that the target to be detected cannot find a tracking track matching with the target in all video frames, and determines whether the target to be detected is a tracking target appearing for the first time in all video frames.
S24: and if so, setting the status bit of the target to be detected as the creation.
Specifically, if the target to be detected is a tracking target appearing for the first time in all video frames, the status bit of the target to be detected is set to Create, and the status bit is numbered, i.e., ID, for the target to be detected.
S25: otherwise, the status bit of the target to be detected is set to be lost.
Specifically, if the target to be detected is not the tracking target appearing for the first time in all video frames, it is indicated that the target to be detected is Lost in the video frames, and the status bit of the target to be detected is set to Lost.
Specifically, in the present embodiment, please refer to fig. 5, and fig. 5 is a flowchart illustrating an embodiment after step S25 in fig. 4. Specifically, after step S25, the method includes:
s30: and judging whether the number of the video frames with the status bits of the target to be detected being lost is larger than a sixth preset threshold value.
Specifically, in this embodiment, if the status bit of the target to be detected is Lost, it is determined whether the number of video frames of which the status bit of the target to be detected is Lost is greater than a sixth preset threshold, where the sixth preset threshold may be 12 frames, or may be another numerical value set according to an actual situation, and the application is not limited herein.
S31: and if so, setting the state bit of the target to be detected as deleted and deleting the tracking result of the target to be detected.
Specifically, if the number of the lost video frames of the status bits of the target to be detected is greater than the sixth preset threshold, the status bits of the target to be detected are set to Delete and Delete the tracking result of the target to be detected, so that the condition of invalid resource occupation can be avoided.
S32: otherwise, returning to the step of setting the status bit of the target to be detected as lost.
Specifically, if the status bit of the target to be detected is that the number of lost video frames is less than or equal to the sixth preset threshold, the process returns to step S25.
By the design mode, the motion track and the identity information of the target to be detected can be obtained, and the logic judgment of neglected loading is facilitated, so that the accuracy and the reliability of detection are improved.
S2: and judging whether the overflow confidence coefficient of the target to be detected is greater than a first preset threshold value when the type of the target to be detected is the garbage can.
Specifically, in this embodiment, when the category of the target to be detected is the trash can, the overflow confidence Conf of the target to be detected is determinedOC—FullIs greater than the first predetermined threshold th1, wherein the first predetermined threshold th1 is a floating point number between 0 and 1.
In addition, in the present embodiment, please refer to fig. 6, and fig. 6 is a flowchart illustrating an embodiment before step S2 in fig. 1. Specifically, the step S2 is preceded by:
s40: and acquiring a first coordinate frame corresponding to the garbage can in the target detection area.
Specifically, in this embodiment, when the target to be detected in the target detection area is a trash can, the coordinate point at the upper left corner of the first coordinate frame Rect corresponding to the trash can is (x)1,y1) The coordinate point of the lower right corner is (x)2,y2)。
S41: and obtaining a second coordinate frame corresponding to the garbage can according to the first coordinate frame.
Specifically, since the output first coordinate frame Rect may be in a state of being able to detect only the body of the trash can without including the opening, and thus the stacking of the trash in the trash can cannot be seen, the top of the first coordinate frame Rect needs to be extended upward by a preset ratio of the height of the trash can to obtain a second coordinate frame Rect corresponding to the trash canOCThat is, the second coordinate frame RectOCExtends upward by a preset proportion of the height of the trash can relative to the first coordinate frame Rect. Specifically, the preset ratio may be 15%, or may be other ratios, and the application is not limited herein. In the present embodiment, y is adjusted1Position of (a) is obtained as1', the specific formula is as follows:
y1’=max(y1-(y2-y1)*0.15,0)
obtaining a second coordinate frame Rect corresponding to the garbage can according to the formulaOCSecond coordinate frame RectOCHas a coordinate point of (x) at the upper left corner1,y1'), the coordinate point of the lower right corner is (x)2,y2). Therefore, the overflow degree of the garbage can be fully judged later, the condition of false detection is avoided, and the overflow detection accuracy of the garbage can is improved.
S42: and inputting the image corresponding to the second coordinate frame into the first target classification network to obtain the overflow confidence corresponding to the trash can.
Specifically, in the present embodiment, the second coordinate frame Rect is set toOCExtracting corresponding images, inputting the images into a first target classification network, and obtaining corresponding overflow confidence Conf of the garbage canOC—FullWherein the overflow confidence ConfOC—FullIs a decimal fraction between 0 and 1. Specifically, the first target classification network in the present application is a two-class network, and the detected classes are overflow or non-overflow, respectively.
S3: and judging whether the detection classification combined confidence coefficient of the target to be detected is greater than a second preset threshold value when the category of the target to be detected is exposed garbage.
Specifically, in this embodiment, when the class of the target to be detected is exposed garbage, it is determined whether the detection classification joint confidence Conf of the target to be detected is greater than a second preset threshold th2, where the second preset threshold th2 is a floating point number between 0 and 1.
In addition, in the present embodiment, please refer to fig. 7, and fig. 7 is a flowchart illustrating an embodiment before step S3 in fig. 1. Specifically, the step S3 is preceded by:
s50: and acquiring a third coordinate frame corresponding to the exposed garbage in the target detection area.
Specifically, in this embodiment, when the target to be detected in the target detection area is exposed trash, the third coordinate frame corresponding to the exposed trash is obtained.
S51: and inputting the image corresponding to the third coordinate frame into a second target classification network to obtain a garbage confidence coefficient corresponding to the exposed garbage.
Specifically, in this embodiment, the image corresponding to the third coordinate frame is extracted and input into the second target classification network to obtain the garbage confidence Conf corresponding to the exposed garbageOCWherein the confidence of garbage ConfOCIs a decimal fraction between 0 and 1. Specifically, the second target classification network in the present application is a binary classification network, and the detected classes are respectively garbage or non-garbage.
S52: and judging whether the exposed garbage category is garbage according to the garbage confidence.
Specifically, since the types of the trash are numerous, various articles near the trash can are stacked together, false detection of objects such as ground water stain, dust, leaves and the like is likely to occur, and if only a single detection algorithm is adopted, many false alarms are generated, so that whether the category of the exposed trash is the trash can be judged according to the trash confidence obtained in the step S61, and the detection reliability is improved.
S53: and if so, obtaining a sum of the product of the first weight coefficient and the target detection confidence coefficient and the product of the second weight coefficient and the garbage confidence coefficient, and taking the sum as the detection classification joint confidence coefficient of the exposed garbage.
Specifically, if the type of the exposed garbage is determined to be garbage, the first weight coefficient α and the target detection confidence Conf are obtainedODThe product of (a) and the second weight coefficient beta and the spam confidence factor ConfOCThe sum value is taken as the detection classification joint confidence Conf of the exposed garbage, and the specific formula is as follows:
Conf=αConfOD+βConfOC
note that β is 1- α, and in this embodiment, α is 0.5, but in other embodiments, α may be other values, and is determined according to actual circumstances.
S54: otherwise, the exposed garbage is deleted.
Specifically, if the type of the exposed garbage is judged to be non-garbage, the exposed garbage is deleted, and subsequent operation is not needed, so that resources are saved, and the reliability of exposed garbage detection is improved.
In the present application, before step S2 or S3, the detection and classification information corresponding to the target to be detected in steps S40-S42 and steps S50-S54 is integrated, and the integrated information is applied to the subsequent steps to detect the garbage. Specifically, when the target to be detected is a trash can, the integrated information includes a first coordinate frame Rect and a target detection confidence factor ConfODAnd overflow confidence ConfOC—Full(ii) a When the target to be detected is exposed garbage, the integrated information comprises a third coordinate frame and a detection classification joint confidence Conf. Through the design mode, the result of the overflow confidence coefficient can improve the accuracy of detecting whether the garbage bin is overflow, and the result of the detection classification combined confidence coefficient can improve the reliability of detecting the exposed garbage.
Specifically, in the present embodiment, referring to fig. 8, fig. 8 is a schematic flowchart of an implementation manner before step S2 or before step S3 in fig. 1. Specifically, the step S2 or the step S3 includes:
s60: and judging whether the center point coordinate of the coordinate frame corresponding to the target to be detected is located in the target detection area.
Specifically, in this embodiment, the coordinate point at the upper left corner of the coordinate frame corresponding to the target to be detected is (x)1,y1x, the coordinate point at the lower right corner is (x)2,y2) The coordinate of the central point of the coordinate frame corresponding to the target to be detected is
Figure BDA0003110045460000101
Judging the coordinates of the center point
Figure BDA0003110045460000102
Whether it is located within the target detection area.
S61: if so, the step of judging whether the overflow confidence coefficient of the target to be detected is larger than a first preset threshold value when the category of the target to be detected is a garbage can or the step of judging whether the detection classification combined confidence coefficient of the target to be detected is larger than a second preset threshold value when the category of the target to be detected is exposed garbage is carried out.
Specifically, if the center point coordinates
Figure BDA0003110045460000111
If the target detection area is located, the process proceeds to step S2 or step S3.
S62: otherwise, returning to the step of acquiring a plurality of video frames.
Specifically, if the center point coordinates
Figure BDA0003110045460000112
And is not located within the target detection area, the process returns to step S1.
S4: if yes, the target to be detected is used as an alarm target, and the time of the video frames corresponding to all the alarm targets is accumulated to obtain first accumulated time.
Specifically, when the class of the target to be detected is the trash can, if the overflow confidence Conf of the target to be detected is higher than the overflow confidence Conf of the target to be detectedOC—FullIf the value is greater than the first preset threshold th1, the target to be detected is taken as an alarm target, the time of the video frames corresponding to all the alarm targets is accumulated to obtain a first accumulated time t, and the alarm is regularly accumulated when the alarm is givenThe time interval is t +1, where the unit of t may be a frame or other time unit, and the application is not limited herein. In addition, in the present embodiment, before step S4, the method further includes: and setting the alarm zone bit of the video frame corresponding to the target to be detected as a first zone bit. By means of accumulative counting, the influence of false detection of individual frames on result judgment can be avoided.
Specifically, when the type of the target to be detected is exposed garbage, if the detection classification joint confidence Conf of the target to be detected is greater than the second preset threshold th2, the target to be detected is taken as an alarm target, and the time of the video frames corresponding to all the alarm targets is accumulated to obtain a first accumulated time t, where the regular accumulated alarm time is t +1, where the unit of t may be a frame or other time units, and the application is not limited herein. In addition, in this embodiment, before step S4, the method further includes setting the alarm flag bit of the video frame corresponding to the target to be detected as the first flag bit. By means of accumulative counting, the influence of false detection of individual frames on result judgment can be avoided.
S5: otherwise, summarizing the targets to be detected, and accumulating the time of the video frames corresponding to all the targets to be detected to obtain a second accumulated time.
Specifically, when the class of the target to be detected is the trash can, if the overflow confidence Conf of the target to be detected is higher than the overflow confidence Conf of the target to be detectedOC—FullAnd if the sum is less than or equal to the first preset threshold th1, summarizing all the targets to be detected meeting the above conditions, and accumulating the time of the video frames corresponding to all the targets to be detected to obtain a second accumulated time n, where the accumulated time that does not meet the alarm is n +1, where the unit of n may be a frame or other time units, and the application is not limited herein. In addition, in the present embodiment, before step S5, the method further includes: and setting the alarm zone bit of the video frame corresponding to the target to be detected as a second zone bit, wherein the second zone bit is different from the first zone bit.
Specifically, when the type of the target to be detected is exposed garbage, if the detection classification joint confidence Conf of the target to be detected is less than or equal to the second preset threshold th2, summarizing all the targets to be detected that satisfy the above condition, and accumulating the time of the video frames corresponding to all the targets to be detected to obtain a second accumulated time n, where the accumulated time that does not satisfy the alarm is n +1, where the unit of n may be a frame or other time units, and the application is not limited herein. In addition, in this embodiment, before step S5, the method further includes setting the warning flag bit of the video frame corresponding to the target to be detected as a second flag bit, where the second flag bit is different from the first flag bit.
S6: and judging whether the first accumulated time is greater than a third preset threshold value.
Specifically, in the present embodiment, it is determined whether the first accumulated time t is greater than a third preset threshold t1, where the third preset threshold t1 is configured by the user according to practical situations, and the application is not limited herein. Through the design mode, the alarm time interval can be set to remind a monitor to clear up garbage in time.
S7: if so, emptying the first accumulated time, and reporting all alarm targets for alarm display.
Specifically, if the first accumulated time t is greater than a third preset threshold t1, the first accumulated time t is set to 0, and all the alarm targets are reported to perform alarm display. Specifically, if the reported alarm target is an overflow trash can, the reported information includes the category, the coordinate frame, and the overflow confidence Conf of the alarm targetOC—Full(ii) a If the reported alarm target is exposed garbage, the reported information includes the category of the alarm target, a corresponding coordinate frame and a detection classification joint confidence factor Conf, which is not limited herein.
S8: otherwise, returning to the step of acquiring a plurality of video frames.
Specifically, if the first accumulated time t is less than or equal to the third preset threshold t1, the process returns to step S1.
S9: and judging whether the second accumulated time is greater than a fourth preset threshold value.
Specifically, in this embodiment, it is determined whether the second accumulated time n is greater than a fourth preset threshold n1, where the fourth preset threshold n1 may be 50 frames, or may be another value configured by the user according to an actual situation, and the application is not limited herein.
S10: and if so, clearing the first accumulation time and the second accumulation time, and returning to the step of acquiring a plurality of video frames.
Specifically, if the second integrated time n is greater than the fourth preset threshold n1, the first integrated time t and the second integrated time n are set to 0, and the process returns to step S1. Therefore, by setting the alarm accumulation time which is not met, when the alarm condition is not met when the alarm accumulation time continuously exceeds the fourth preset threshold n1, the alarm accumulation time is accumulated again, and the influence of the false detection of individual frames on the result judgment can be avoided.
S11: otherwise, returning to the step of acquiring a plurality of video frames.
Specifically, if the second accumulated time n is less than or equal to the fourth preset threshold n1, the process returns to step S1.
Through the design mode, the alarm accuracy rate of overflowing and exposed garbage of the garbage can be improved, the influence of false detection of individual video frames on result judgment is avoided, and the purpose of repeated alarm reminding is achieved by setting the alarm time threshold, so that a monitor is reminded of clearing garbage in time.
In addition, in this embodiment, before step S1, a video sequence of the target detection area is further acquired, and an image obtained by acquiring a video frame is used as a training data set. Firstly, a training data set for target detection is made, a garbage can and exposed garbage are marked as positive samples, and a first target classification network is obtained after deep learning is carried out on the positive samples, so that two categories of the garbage can and the exposed garbage can be detected simultaneously; and then, a data set for target classification is produced, the classified data set is divided into two types, the first classification data set is a garbage can picture set and is divided into two types of an overflowing garbage can and a non-overflowing garbage can, the second classification data set is an exposed garbage and non-garbage picture set, and a second target classification network is obtained after deep learning is carried out on the exposed garbage and non-garbage picture set, so that overflow or non-overflow of the garbage can and garbage or garbage exposure can be detected at the same time. Through such design, can promote the garbage bin overflow and expose the rate of accuracy of reporting an emergency and asking for help or increased vigilance of rubbish to remind the supervisor in time to clear up rubbish.
Referring to fig. 9, fig. 9 is a schematic diagram of a framework of an embodiment of the spam detection system according to the present application. The above-mentioned rubbish detecting system specifically includes:
the acquisition module 10 is configured to acquire a plurality of video frames and obtain a tracking result of a target to be detected in a target detection area from each video frame; and the tracking result comprises the category of the target to be detected.
The judging module 12 is coupled to the obtaining module 10, and configured to judge whether the category of the target to be detected is a trash can and an overflow confidence level of the target is greater than a first preset threshold, or judge whether the category of the target to be detected is exposed trash and a detection classification joint confidence level of the target to be detected is greater than a second preset threshold. Of course, the determining module 12 is further configured to determine whether the first accumulated time is greater than a third preset threshold.
And the processing module 14 is coupled to the obtaining module 12, and configured to, in response to that the category of the target to be detected is a trash can and the overflow confidence of the target to be detected is greater than a first preset threshold, or in response to that the category of the target to be detected is exposed trash and the detection classification joint confidence of the target to be detected is greater than a second preset threshold, take the target to be detected as an alarm target, and accumulate the time of video frames corresponding to all the alarm targets to obtain a first accumulated time. Certainly, the processing module 14 is further configured to empty the first accumulated time and report all the alarm targets for alarm display in response to that the first accumulated time is greater than a third preset threshold.
Please refer to fig. 10, fig. 10 is a schematic structural diagram of an embodiment of the garbage detection system according to the present application. The garbage detection system includes a processor 100 and a memory 102 coupled to each other. Specifically, in the present embodiment, the processor 100 and the memory 102 cooperate with each other to implement the garbage detection method mentioned in any of the above embodiments.
Specifically, processor 100 may also be referred to as a CPU (Central Processing Unit). Processor 100 may be an integrated circuit chip having signal processing capabilities. The Processor 100 may also be a general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. Additionally, processor 100 may be commonly implemented by multiple integrated circuit chips.
Referring to fig. 11, fig. 11 is a schematic structural diagram of an embodiment of a computer-readable storage medium according to the present application. The computer-readable storage medium 20 stores a computer program 200, which can be read by a computer, and the computer program 200 can be executed by a processor to implement the garbage detection method mentioned in any of the above embodiments. The computer program 200 may be stored in the computer-readable storage medium 20 in the form of a software product, and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. The computer-readable storage medium 20 having a storage function may be various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, or may be a terminal device, such as a computer, a server, a mobile phone, or a tablet.
In summary, different from the situation of the prior art, in the present application, a plurality of video frames are obtained, and a tracking result of an object to be detected in an object detection area is obtained from each video frame, where the tracking result includes a category of the object to be detected, when the category of the object to be detected is a trash can and an overflow confidence thereof is greater than a first preset threshold, or when the category of the object to be detected is exposed trash and a detection classification joint confidence thereof is greater than a second preset threshold, the object to be detected is taken as an alarm object, and times of all the alarm objects corresponding to the video frames are accumulated to obtain a first accumulated time, and when the first accumulated time is greater than a third preset threshold, the first accumulated time is cleared, and all the alarm objects are reported to perform alarm display. Through the design mode, the alarm accuracy rate of overflowing and exposed garbage of the garbage can be improved, the influence of false detection of individual video frames on result judgment is avoided, and the purpose of repeated alarm reminding is achieved by setting the alarm time threshold, so that a monitor is reminded of clearing garbage in time.
The above description is only for the purpose of illustrating embodiments of the present application and is not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application or are directly or indirectly applied to other related technical fields, are also included in the scope of the present application.

Claims (11)

1. A garbage detection method is characterized by comprising the following steps:
acquiring a plurality of video frames, and acquiring a tracking result of a target to be detected in a target detection area from each video frame; wherein the tracking result comprises the category of the target to be detected;
responding to the fact that the category of the target to be detected is a garbage can and the overflow confidence coefficient of the target to be detected is larger than a first preset threshold value, or responding to the fact that the category of the target to be detected is exposed garbage and the detection classification joint confidence coefficient of the target to be detected is larger than a second preset threshold value, taking the target to be detected as an alarm target, and accumulating the time of video frames corresponding to all the alarm targets to obtain first accumulated time;
and responding to the situation that the first accumulated time is larger than a third preset threshold value, emptying the first accumulated time, and reporting all the alarm targets for alarm display.
2. The spam detection method according to claim 1, wherein the step of obtaining a tracking result of the object to be detected in the object detection area from each of the video frames further comprises:
in response to the fact that the category of the target to be detected is a garbage can and the overflow confidence coefficient of the target to be detected is smaller than or equal to the first preset threshold, or in response to the fact that the category of the target to be detected is exposed garbage and the detection classification joint confidence coefficient of the target to be detected is smaller than or equal to the second preset threshold, summarizing the target to be detected, and accumulating the time of video frames corresponding to all the target to be detected to obtain second accumulated time;
and in response to the second accumulated time being greater than a fourth preset threshold, emptying the first accumulated time and the second accumulated time, and returning to the step of acquiring a plurality of video frames.
3. The debris detection method according to claim 2,
before the step of taking the target to be detected as the warning target and accumulating the time of the video frames corresponding to all the warning targets to obtain the first accumulated time, the method comprises the following steps:
setting an alarm flag bit of a video frame corresponding to the target to be detected as a first flag bit;
before the step of summarizing the targets to be detected and accumulating the time of the video frames corresponding to all the targets to be detected to obtain a second accumulated time, the method comprises the following steps:
and setting the alarm zone bit of the video frame corresponding to the target to be detected as a second zone bit.
4. The spam detection method according to claim 1, wherein said step of obtaining a tracking result of the object to be detected in the object detection area from each of said video frames is followed by:
obtaining a standard value of a coordinate frame corresponding to each target to be detected in a current video frame and a coordinate frame corresponding to the target to be detected in an existing video frame through a matching algorithm;
and in response to the fact that the standard value is larger than a fifth preset threshold value, setting the state bit of the target to be detected to be updated, and obtaining a tracking result of the target to be detected.
5. The debris detection method according to claim 4,
after the step of obtaining the standard value of the coordinate frame corresponding to each target to be detected in the current video frame and the standard value of the coordinate frame corresponding to the target to be detected in the existing video frame through the matching algorithm, the method further comprises the following steps:
responding to the standard value smaller than or equal to a fifth preset threshold value, and judging whether the target to be detected is a tracking target appearing for the first time in all the video frames;
if so, setting the state bit of the target to be detected as creation;
otherwise, setting the state bit of the target to be detected as lost;
after the step of setting the status bit of the target to be detected to be lost, the method comprises the following steps:
and in response to the condition that the number of the lost video frames of the state bits of the target to be detected is larger than a sixth preset threshold value, setting the state bits of the target to be detected to be deleted and deleting the tracking result of the target to be detected.
6. The spam detection method according to claim 1, wherein the step of taking the target to be detected as an alert target and accumulating the time of all the alert targets corresponding to video frames to obtain a first accumulated time is preceded by the step of responding to the target to be detected as a trash can with an overflow confidence greater than a first preset threshold, or responding to the target to be detected as exposed trash with a detection classification joint confidence greater than a second preset threshold:
judging whether the center point coordinate of the coordinate frame corresponding to the target to be detected is located in the target detection area;
if so, entering a step of responding to that the category of the target to be detected is a garbage can and the overflow confidence coefficient of the target to be detected is greater than a first preset threshold value, or responding to that the category of the target to be detected is exposed garbage and the detection classification joint confidence coefficient of the target to be detected is greater than a second preset threshold value, taking the target to be detected as an alarm target, and accumulating the time of video frames corresponding to all the alarm targets to obtain first accumulated time;
otherwise, returning to the step of acquiring a plurality of video frames.
7. The spam detection method according to claim 1, wherein before the step of taking the target to be detected as an alert target and accumulating video frame times corresponding to all the alert targets to obtain a first accumulated time in response to the class of the target to be detected being a trash can and the overflow confidence thereof being greater than a first preset threshold, the method comprises:
acquiring a first coordinate frame corresponding to the garbage can in the target detection area;
obtaining a second coordinate frame corresponding to the garbage can according to the first coordinate frame, wherein the top of the second coordinate frame extends upwards for a preset proportion of the height of the garbage can relative to the first coordinate frame;
and inputting the image corresponding to the second coordinate frame into a first target classification network to obtain the overflow confidence corresponding to the garbage can.
8. The spam detection method according to claim 1, wherein before the step of taking the target to be detected as an alert target and accumulating video frame times corresponding to all the alert targets to obtain a first accumulated time in response to the fact that the category of the target to be detected is spam and the detection classification joint confidence thereof is greater than a second preset threshold, the method comprises:
acquiring a third coordinate frame corresponding to the exposed garbage in the target detection area;
inputting the image corresponding to the third coordinate frame into a second target classification network to obtain a garbage confidence coefficient corresponding to the exposed garbage;
judging whether the category of the exposed garbage is garbage or not according to the garbage confidence coefficient;
if so, obtaining a sum of a product of the first weight coefficient and a target detection confidence coefficient and a product of the second weight coefficient and the garbage confidence coefficient, and taking the sum as a detection classification joint confidence coefficient of the exposed garbage;
otherwise, deleting the exposed garbage.
9. The spam detection method according to claim 1, wherein the step of obtaining the tracking result of the object to be detected in the object detection area from each of the video frames is preceded by the step of:
acquiring the category of a target to be detected in the target detection area;
and obtaining the target detection confidence corresponding to each target to be detected according to the category.
10. A spam detection system comprising a memory and a processor coupled to each other, the memory having stored therein program instructions, the processor being configured to execute the program instructions to implement the spam detection method of any of claims 1-9.
11. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for implementing the spam detection method according to any one of claims 1 to 9.
CN202110646613.2A 2021-06-10 2021-06-10 Garbage detection method, garbage detection system and computer readable storage medium Pending CN113468976A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110646613.2A CN113468976A (en) 2021-06-10 2021-06-10 Garbage detection method, garbage detection system and computer readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110646613.2A CN113468976A (en) 2021-06-10 2021-06-10 Garbage detection method, garbage detection system and computer readable storage medium

Publications (1)

Publication Number Publication Date
CN113468976A true CN113468976A (en) 2021-10-01

Family

ID=77869530

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110646613.2A Pending CN113468976A (en) 2021-06-10 2021-06-10 Garbage detection method, garbage detection system and computer readable storage medium

Country Status (1)

Country Link
CN (1) CN113468976A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114283387A (en) * 2022-03-08 2022-04-05 深圳市万物云科技有限公司 Intelligent garbage point cleaning work order generation method and device and related medium
CN114532918A (en) * 2022-01-26 2022-05-27 深圳市杉川机器人有限公司 Cleaning robot, target detection method and device thereof, and storage medium
CN116189099A (en) * 2023-04-25 2023-05-30 南京华苏科技有限公司 Method for detecting and stacking exposed garbage based on improved yolov8

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160167233A1 (en) * 2014-12-11 2016-06-16 Xiaomi Inc. Methods and devices for cleaning garbage
CN108960174A (en) * 2018-07-12 2018-12-07 广东工业大学 A kind of object detection results optimization method and device
CN110956104A (en) * 2019-11-20 2020-04-03 河南华衍智能科技有限公司 Method, device and system for detecting overflow of garbage can
CN111458721A (en) * 2020-03-31 2020-07-28 江苏集萃华科智能装备科技有限公司 Exposed garbage identification and positioning method, device and system
CN112560755A (en) * 2020-12-24 2021-03-26 中再云图技术有限公司 Target detection method for identifying urban exposed garbage
CN112883921A (en) * 2021-03-22 2021-06-01 北京易华录信息技术股份有限公司 Garbage can overflow detection model training method and garbage can overflow detection method
CN112926431A (en) * 2021-02-20 2021-06-08 北京市商汤科技开发有限公司 Garbage detection method, device, equipment and computer storage medium

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160167233A1 (en) * 2014-12-11 2016-06-16 Xiaomi Inc. Methods and devices for cleaning garbage
CN108960174A (en) * 2018-07-12 2018-12-07 广东工业大学 A kind of object detection results optimization method and device
CN110956104A (en) * 2019-11-20 2020-04-03 河南华衍智能科技有限公司 Method, device and system for detecting overflow of garbage can
CN111458721A (en) * 2020-03-31 2020-07-28 江苏集萃华科智能装备科技有限公司 Exposed garbage identification and positioning method, device and system
CN112560755A (en) * 2020-12-24 2021-03-26 中再云图技术有限公司 Target detection method for identifying urban exposed garbage
CN112926431A (en) * 2021-02-20 2021-06-08 北京市商汤科技开发有限公司 Garbage detection method, device, equipment and computer storage medium
CN112883921A (en) * 2021-03-22 2021-06-01 北京易华录信息技术股份有限公司 Garbage can overflow detection model training method and garbage can overflow detection method

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
陶威远;姜太平;郑一然;: "基于Faster R-CNN算法的垃圾分类识别系统", 网络安全技术与应用, no. 03, 15 March 2020 (2020-03-15) *
魏书法;程章林;: "基于图像的城市场景垃圾自动检测", 集成技术, no. 01, 15 January 2017 (2017-01-15) *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114532918A (en) * 2022-01-26 2022-05-27 深圳市杉川机器人有限公司 Cleaning robot, target detection method and device thereof, and storage medium
CN114283387A (en) * 2022-03-08 2022-04-05 深圳市万物云科技有限公司 Intelligent garbage point cleaning work order generation method and device and related medium
CN116189099A (en) * 2023-04-25 2023-05-30 南京华苏科技有限公司 Method for detecting and stacking exposed garbage based on improved yolov8
CN116189099B (en) * 2023-04-25 2023-10-10 南京华苏科技有限公司 Method for detecting and stacking exposed garbage based on improved yolov8

Similar Documents

Publication Publication Date Title
CN113468976A (en) Garbage detection method, garbage detection system and computer readable storage medium
WO2020206861A1 (en) Yolo v3-based detection method for key object at transportation junction
US11455805B2 (en) Method and apparatus for detecting parking space usage condition, electronic device, and storage medium
CN111178182A (en) Real-time detection method for garbage loss behavior
WO2019000929A1 (en) Garbage sorting and recycling method, garbage sorting equipment, and garbage sorting and recycling system
WO2021004344A1 (en) Data analysis-based risk identification method and related device
CN111126252A (en) Stall behavior detection method and related device
US11587327B2 (en) Methods and systems for accurately recognizing vehicle license plates
KR20140036990A (en) Device and method for detecting the presence of a logo in a picture
WO2022174541A1 (en) Garbage detection method and apparatus, device, storage medium, and program product
Chetouane et al. Vision‐based vehicle detection for road traffic congestion classification
CN114943936B (en) Target behavior recognition method and device, electronic equipment and storage medium
CN113052158A (en) Night infrared scene target identification method and device, electronic equipment and storage medium
CN112560755A (en) Target detection method for identifying urban exposed garbage
CN110490150A (en) A kind of automatic auditing system of picture violating the regulations and method based on vehicle retrieval
CN104463232A (en) Density crowd counting method based on HOG characteristic and color histogram characteristic
US20210004600A1 (en) Assessing video stream quality
CN115545441A (en) Road garbage detection method, system, terminal and storage medium
Chen et al. A precise information extraction algorithm for lane lines
CN113920585A (en) Behavior recognition method and device, equipment and storage medium
CN113139434A (en) City management event processing method and device, electronic equipment and readable storage medium
CN113963316A (en) Target event determination method and device, storage medium and electronic device
CN112183460A (en) Method and device for intelligently identifying environmental sanitation
Kerouh et al. Real-time Android application for traffic density estimation
Tiwari et al. Real Time Traffic Management Using Machine Learning

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination