WO2022174541A1

WO2022174541A1 - Garbage detection method and apparatus, device, storage medium, and program product

Info

Publication number: WO2022174541A1
Application number: PCT/CN2021/103078
Authority: WO
Inventors: 窦浩轩; 王意如; 甘伟豪
Original assignee: 北京市商汤科技开发有限公司
Priority date: 2021-02-20
Filing date: 2021-06-29
Publication date: 2022-08-25
Also published as: CN112926431A

Abstract

A garbage detection method and apparatus, a device, a storage medium, and a program product. The method comprises: obtaining a first image to be detected (S101); when it is determined that a target object exists in the first image to be detected, obtaining a second image to be detected, wherein an overlapping ratio between an acquisition region of the first image to be detected and an acquisition region of the second image to be detected is greater than a preset threshold, and acquisition time of the first image to be detected and acquisition time of the second image to be detected have a preset time interval (S102); and when it is determined that the target object exists in the second image to be detected, determining that the target object is garbage (S103).

Description

A garbage detection method, device, equipment, storage medium and program product

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on the Chinese patent application with the application number of 202110193692.6, the application date of February 20, 2021, and the application name of "garbage detection method, device, equipment and computer storage medium", and claims the priority of the Chinese patent application, The entire contents of this Chinese patent application are hereby incorporated by reference into this application.

technical field

The present application relates to, but is not limited to, the field of intelligent management, and in particular, relates to a method, apparatus, device, storage medium and program product for garbage detection.

Background technique

It has a very important role in garbage detection through images. The prior art mainly uses a lot of manpower to label unlabeled data, which consumes manpower and computing resources. Using a single frame of image cannot effectively exclude everyday objects that are confusingly similar to garbage.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a garbage detection method, apparatus, device, storage medium, and program product.

A first aspect provides a method for detecting garbage, the method comprising:

Acquire a first image to be detected; in the case where it is determined that a target object exists in the first image to be detected, acquire a second image to be detected, wherein the acquisition area of the first image to be detected is the same as the second image to be detected The overlap ratio between the image acquisition areas is greater than a preset threshold, and the acquisition time of the first to-be-detected image and the second to-be-detected image is separated by a preset time interval; in determining the second to-be-detected image When the target object exists, it is determined that the target object is garbage.

In some embodiments, the method further includes: analyzing scene information in the first image to be detected; and determining the target object according to the scene information in the first image to be detected.

In this way, determining the target object according to the scene information in the first image to be detected can effectively improve the efficiency and accuracy of determining the target object.

In some embodiments, the method further includes: determining the time interval according to attribute parameters of the first image to be detected and/or attribute parameters of the target object; wherein the first image to be detected The attribute parameter includes at least one of the following: scene information of the first image to be detected, and a time period or season to which the acquisition time of the first image to be detected belongs.

In this way, the time interval determined according to the attribute parameters of the first image to be detected and/or the attribute parameters of the target object can further improve the efficiency of garbage detection, so that the suspected garbage can be confirmed as soon as possible.

In some embodiments, the method further includes: using a target garbage detection model to detect the first image to be detected extracted from the online video stream to obtain a first detection result; In the case where the target object exists in the detected image, acquiring the second image to be detected includes: in the case that the target object exists in the first image to be detected according to the first detection result, from a stored video library Obtaining the second image to be detected in ; correspondingly, the method further includes: using a target garbage detection model to detect from the second image to be detected to obtain a second detection result; In the case that the target object exists in the to-be-detected image, determining that the target object is garbage includes: in the case of determining that the target object exists in the second to-be-detected image according to the second detection result, determining The target object is garbage.

In this way, the data mining method is used to mine the online video stream and the stored video database data in the real scene, to quickly determine the difficult samples that are difficult to detect and identify by the target garbage detection model, and solve the problem that the traditional method cannot comprehensively collect data.

In some embodiments, the detecting the first to-be-detected image extracted from the online video stream by using the target garbage detection model to obtain a first detection result includes: using the target garbage detection model to analyze the first image to be detected attribute information of the object in the image to be detected; determine whether the corresponding object belongs to the target object or garbage according to the attribute information of the object in the first image to be detected, and obtain a first detection result; the use of the target garbage detection model Detecting the second to-be-detected image to obtain a second detection result includes: using the target garbage detection model to analyze the attribute information of the object in the second to-be-detected image; The attribute information of the object is used to determine whether the corresponding object belongs to the target object or the garbage, and a second detection result is obtained. The attribute information of the object includes at least one of the following: the shape, material, size, and location of the object.

In this way, because the garbage has some specific shape, material, size and location, it can be directly determined whether the target object is garbage according to the attribute information of the object. In this way, the efficiency and accuracy of garbage determination are greatly improved.

In some embodiments, when the first detection result is that the object belongs to garbage, the method further includes: determining, according to attribute information of the object, a garbage category to which the garbage belongs and where the garbage is located. the location of the garbage alarm; determine the content corresponding to the garbage alarm according to the garbage category and the location of the garbage; send the content corresponding to the garbage alarm to the garbage management platform.

In this way, when it is determined that there is garbage, the garbage information (garbage category, location) can be sent to the garbage management platform in time, so as to realize timely and reasonable disposal of garbage according to the garbage category and location.

In some embodiments, the detecting the first image to be detected extracted from the online video stream by using a target garbage detection model to obtain a first detection result further includes: detecting the first image to be detected , determine the target object and the target frame corresponding to the first image to be detected; determine the first intersection ratio according to the target object and the target frame corresponding to the first image to be detected; In the case where the second detection result determines that the target object exists in the second to-be-detected image, determining that the target object is garbage includes: in the case of determining the target object corresponding to the second to-be-detected image, according to the The target frame and the target object corresponding to the second to-be-detected image determine a second intersection ratio; under the condition that the first intersection ratio and the second intersection ratio are greater than the preset intersection ratio threshold at the same time, The target object is determined to be garbage.

In this way, a method of evaluating whether the intersection ratio of target frames on different time frames continuously exceeds a certain threshold is used to determine whether the detected garbage appears in the picture for a long time in a period of time. The daily objects that are similar in shape to garbage but do not belong to the garbage type can be effectively excluded, and the performance of the embodiment of the present application in real scenes can be improved.

In this way, the corresponding terminal or cleaning robot is determined according to the location of the garbage, and then the garbage alarm is sent to the terminal or robot, so that the cleaner holding the terminal can clean up the garbage in time or the cleaning robot can dispose of the garbage in time.

In some embodiments, the target garbage detection model is obtained by adopting the following steps, including: acquiring at least one target image; the target image is inputting the image to be detected intercepted from the video stream to the initial garbage detection model , determined according to the detection result output by the initial garbage detection model; the initial garbage detection model is trained by using a first data set; wherein, the first data set is a data set in which at least some sample images have annotation information ; Obtain the manual labeling result for the at least one target image, and merge the labeled at least one target image into the first data set as a training sample to obtain a second data set; Utilize the second data The initial garbage detection model is trained to obtain the target garbage detection model.

In this way, since the initial garbage detection model is used to determine at least one target image with high value for the training of the initial garbage detection model from the to-be-detected image set including a large number of to-be-detected images, the labeled at least one target image is merged Go to the first data set, obtain the second data set, and use the second data set to train the initial garbage detection model, which not only makes the target garbage detection model obtained after training more accurate when detecting the target, but also can The performance of the target garbage detection model can be effectively improved by using at least one target image after a limited number of annotations, and the computational cost of deep learning can be effectively reduced.

In some embodiments, the acquiring at least one target image includes: inputting the image to be detected into the initial garbage detection model to obtain a posteriori probability of the image to be detected for each frame; When the posterior probability is greater than the first probability threshold and less than the second probability threshold, the image to be detected corresponding to the posterior probability is determined as the target image, wherein the first probability threshold is less than the second probability threshold.

In this way, by using the first probability threshold and the second probability threshold, difficult samples can be selected from the image to be detected. Determining difficult examples as target images can effectively improve the detection accuracy of difficult examples by using the target garbage detection model trained with difficult examples.

In a second aspect, a garbage detection device is provided, comprising: a first acquisition module configured to acquire a first image to be detected; a second acquisition module configured to, when it is determined that a target object exists in the first image to be detected , obtain a second image to be detected, wherein the overlap ratio between the acquisition area of the first image to be detected and the acquisition area of the second image to be detected is greater than a preset threshold, and the first image to be detected is the same as the The collection time of the second to-be-detected image is separated from a preset time interval; the first determination module is configured to determine that the target object is garbage when it is determined that the target object exists in the second to-be-detected image .

In a third aspect, an electronic device is provided, comprising: a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and the processor implements the steps in the above method when the processor executes the computer program .

In a fourth aspect, a computer storage medium is provided, the computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the above method.

In a fifth aspect, the present application also provides a computer program product, the computer program product includes a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is caused to execute the above method. step.

In the embodiment of the present application, the first image to be detected is firstly acquired, then the second image to be detected is acquired when it is determined that there is a target object in the first image to be detected, and finally it is determined that there is a target object in the second image to be detected In the case of , determine that the target object is garbage. In this way, by using multi-frame logic, everyday objects that are similar in shape to garbage but not classified as garbage can be effectively excluded, and objects whose positions have not changed significantly for a long time are determined to be garbage.

Description of drawings

FIG. 1 is a schematic flowchart of a garbage detection method according to an embodiment of the present application;

2A is a schematic flowchart of another garbage detection method provided by an embodiment of the present application;

2B is a schematic diagram of an image to be detected provided by an embodiment of the present application;

2C is a schematic diagram of another to-be-detected image provided by an embodiment of the present application;

3 is a schematic flowchart of another garbage detection method provided by an embodiment of the present application;

4A is a schematic diagram of the architecture of a target detection model provided by an embodiment of the present application;

4B is a schematic flowchart of a garbage detection method provided by an embodiment of the present application;

4C is a schematic flowchart of a garbage detection method provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of the composition and structure of a garbage detection device provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of a hardware entity of an electronic device according to an embodiment of the present application.

Implementation

In order to make the purposes, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the invention will be described in further detail below with reference to the accompanying drawings in the embodiments of the present application. The following examples are used to illustrate the present application, but are not intended to limit the scope of the present application.

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

It should be understood that some embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the technical scope of the present application.

This embodiment proposes a garbage detection method, which is applied to an electronic device. The functions implemented by the method can be realized by calling a program code by a processor in the electronic device. Of course, the program code can be stored in a computer storage medium. It can be seen that the electronic device The device includes at least a processor and a storage medium.

FIG. 1 is a schematic diagram of the implementation flow of a garbage detection method provided by an embodiment of the present application. As shown in FIG. 1 , the method includes:

Step S101, acquiring a first image to be detected;

The first image to be detected may be obtained from an online video stream. In some embodiments, the first image to be detected may be intercepted from the online video stream according to a set duration. The set duration may be a fixed duration, or the set duration may be changed according to user requirements. For example, the set duration may be any duration from 1 second to 30 minutes, for example, the set duration may be 1 second, 5 seconds, 1 minute, 10 minutes, or 30 minutes. In other embodiments, the interception time interval may be determined according to the actual situation (for example, attribute information of the first video stream), and the first image to be detected is intercepted from the online video stream according to the interception time interval.

In some embodiments, the interception time interval may be determined according to attribute parameters of the online video stream. The attribute parameters of the online video stream may be environmental parameters, locations, and may also be the time period in which images of the online video stream are obtained. For example, the environmental parameters and location information of the camera that obtains the online video stream can be used as attribute parameters. The environmental parameters can be shopping malls, kindergartens, or subway stations. In the process of implementation, in one case, the obtained online video stream is a subway station during the rush hour (belonging to the peak flow of people), then the interception time interval can be set to a smaller duration, such as 5 minutes; in another case If the obtained online video stream is a kindergarten after school (belonging to a low peak period), the interception time interval can be set to a longer duration, such as 30 minutes.

Step S102: In the case where it is determined that there is a target object in the first image to be detected, acquire a second image to be detected, wherein a collection area of the first image to be detected and a collection area of the second image to be detected The overlap ratio between them is greater than a preset threshold, and the acquisition times of the first to-be-detected image and the second to-be-detected image are separated by a preset time interval;

In some embodiments, the target object can be determined to be garbage only in some specific scenarios and time periods. For example, a plastic basin that has not been moved for a long time at the door of the house, the mineral water bottle on the court, and the real object that has not changed for a long time on the dining room table. Therefore, a target garbage detection model or a target detection module can be used to determine that a target object exists in the first image to be detected. Here, the determined target object is suspected garbage, that is, it may be an object that has not been moved for a long time.

When it is determined that there is suspected garbage in the first image to be detected, the second image to be detected needs to be acquired again. Since the first to-be-detected image and the second to-be-detected image need to be collected from the same target object, it is necessary to define the acquisition area of the first to-be-detected image and the second to-be-detected image acquisition area, for example, the first to-be-detected image The second to-be-detected image and the second to-be-detected image may be from the same area of the image collected by the same camera device, or from different camera devices, but the overlap ratio between the first to-be-detected image and the second to-be-detected image collection area is greater than a preset threshold, That is, the area of overlapping acquisitions is sufficient to cover the target object.

In some embodiments, the preset time interval may be set according to actual application scenarios. For example, if the food in the cafeteria has not been moved for 30 minutes, it means that the user may have finished the meal and needs to be cleaned up as garbage; if the plastic pot at the door of the house has not been moved for several days, such as 2 days, it may be a plastic pot that the user does not want. In different scenarios, the acquisition time of the first to-be-detected image and the second to-be-detected image can be set to meet the requirements of determining whether the target object is garbage in different scenarios.

Step S103 , in the case that it is determined that the target object exists in the second image to be detected, determine that the target object is garbage.

The definition of garbage is ambiguous and depends on the relationship with the surrounding environment (for example, a small plastic tub placed on the side of the road may be abandoned garbage, or it may be temporarily placed by others), and this relationship with the environment often varies with the environment. changes significantly depending on the location. Another manifestation of this vague definition is the change of position and shape of an item over a period of time, while garbage often does not (for example, a plastic bag in hand will change position and shape over time, placed in The discarded plastic bags on the ground tend to be fixed for a period of time), which leads to the identification of certain garbage that often requires attention to video changes over a period of time.

If both the first to-be-detected image and the second to-be-detected image have the same target object, it is determined to be garbage. Here, the target object may not be in the same position as the first image to be detected and the second image to be detected, but the target object appears within a period of time, that is, in both the images of the first image to be detected and the second image to be detected, or it can be determined for garbage. For example, a plastic bottle that has been moved a certain distance, but still appears in the picture for a long time, can be determined to be garbage.

FIG. 2A is a schematic flowchart of another garbage detection method provided by an embodiment of the present application. As shown in FIG. 2A , the method includes:

Step S201, acquiring a first image to be detected;

Step S202, analyzing scene information in the first image to be detected;

The scene information here may include scene attributes, and the scene information also includes at least one of scene time information and the like. For example: scene attributes can be amusement parks, stations, schools, hospitals, shopping malls, industrial parks, office buildings, canteens, restaurants, etc.; scene time information refers to the time or season corresponding to the current scene, etc. If the scene attribute is canteen, the scene time The information can be the peak and off-peak hours of dining; if the scene attribute is shopping malls, amusement parks, the scene time information can be business hours or non-business hours, etc.; if the scene attribute is school, the scene time information can be holidays or semesters, or It can be during school hours or after school hours.

Step S203, determining the target object according to the scene information in the first to-be-detected image;

Here, the target object is determined according to the scene attribute in the first image to be detected; the target object may also be determined according to the scene attribute and scene time information in the first image to be detected. In the implementation process, a list of target objects may be set according to scene attributes, and the target objects are obtained by identifying the scene information in the first image to be detected, and querying the list according to the scene information.

The scene information can include scene attributes and scene time information. For example, the scene information is a playground where people are exercising during the day or night. There may be basketball, football, badminton, table tennis, clothes, beverage bottles, water glasses, mineral water bottles and other items on the playground. All items are used as the target object, and some of the items such as badminton, table tennis, clothes, beverage bottles, mineral water bottles can also be used as the target object, and the unfinished mineral water bottle or beverage bottle can also be determined as the target object. And treat empty beverage bottles as garbage; if it is a playground where no one is exercising at night, then badminton, table tennis, beverage bottles or mineral water bottles (drinking leftovers or empty bottles or unopened ones) can be used as targets.

The scene information may only include scene attributes. For example, if the scene information is a restaurant, then paper towels, newspapers, and used disposable tableware can all be used as target objects. For another example, when the scene information is a station during peak hours, objects on the ground, such as newspapers and packaging bags, can be determined as the target objects; when the scene information is the dining environment of a canteen, the food on the table can be determined as the target object.

Step S204, in the case of determining that the target object exists in the first image to be detected, obtain a second image to be detected, wherein the acquisition area of the first image to be detected and the area of the second image to be detected are obtained. The overlap ratio between the acquisition areas is greater than a preset threshold, and the acquisition times of the first to-be-detected image and the second to-be-detected image are separated by a preset time interval;

During the implementation process, the corresponding time interval in the playground scene at night can be set to be longer than the corresponding time interval in the daytime playground scene; the corresponding time interval in the campus scene during winter and summer vacations can be set longer than that in class The corresponding time interval of the campus is longer; the corresponding time interval in the non-meal time canteen scene can be set to be longer than the time interval corresponding to the canteen in the dining peak period.

The attribute parameter of the target object can be the time when the target object appears or the location where the target object appears. In the implementation process, in the canteen scene, in the scene where the food tableware and chopsticks are located on the dining table, the time interval can be set to be longer than the corresponding time interval on the ground.

Step S205 , in the case where it is determined that the target object exists in the second image to be detected, determine that the target object is garbage.

In some embodiments, for example, the scene information is a playground where people are exercising during the day. There may be basketball, football, badminton, table tennis, clothes, beverage bottles, water glasses, mineral water bottles and other items on the playground, and the mineral water bottle or beverage bottle is the target object. . In the case where it is determined that there are

mineral water bottles

21, 22 and 23 in the first image to be detected (as shown in FIG. 2B ), if it is determined that there are

mineral water bottles

21 and 22 in the second image to be detected (as shown in FIG. 2C ) For two mineral water bottles placed in similar positions to the

mineral water bottles

21 and 22 in FIG. 2B , it can be determined that the

mineral water bottles

21 and 22 placed in similar positions in FIG. 2B and FIG. 2C respectively are garbage. Likewise, it can be determined that the newly added mineral water bottle 23 in FIG. 2B may be newly placed on the playground by someone.

In the embodiment of the present application, the target object is determined according to the scene information in the first image to be detected, so that the most suspected garbage can be determined according to different scene information settings, and the efficiency of garbage detection is improved; according to the attribute parameters of the first image to be detected and/or Or the attribute parameter of the target object, determine the time interval, in this way, the determined time interval can further improve the efficiency of garbage detection, so that the suspected garbage can be confirmed as soon as possible.

A method for detecting garbage provided by the embodiment of the present application, the method includes:

Step S211, acquiring a first image to be detected;

The first image to be detected is obtained from an online video stream. In some embodiments, the online video stream can be obtained by using a camera device installed in a place where suspected garbage needs to be determined.

In some embodiments, step S211 includes: extracting or intercepting the first image to be detected from the online video stream. In another embodiment, before step S211, the method further includes: intercepting the image to be detected from the online video stream, and storing the image to be detected in an image library or image set, correspondingly, step S211 extracts the image from the image library or image set to obtain the first image to be detected.

Step S212, using a target garbage detection model to detect the first image to be detected to obtain a first detection result;

In some embodiments, the target garbage detection model may be a trained specific target detection model configured to detect suspected garbage in a specific scenario. Using the target garbage detection model, the online video stream can be judged to extract the first image to be detected that may include the target object (suspected garbage), that is, the first detection result is that the first image to be detected includes the target object or does not Include the target object. In some embodiments, step S212 includes: using a target garbage detection model to detect the first image to be detected extracted from the online video stream to obtain a first detection result.

Step S213, in the case of determining that the target object exists in the first image to be detected according to the first detection result, obtain a second image to be detected from a stored video library;

Here, the first image to be detected is acquired from an online video stream, and the acquisition time of the second image to be detected is earlier than the acquisition time of the first image to be detected.

When it is determined by using the target garbage detection model that there is a target object in the first image to be detected, the second detection object needs to be acquired from the stored video library. This is because the currently captured image is detected in the online video stream. If there is suspected garbage in the current image, it is necessary to determine the time when the suspected garbage appears in the same shooting location. For example, if a plastic basin is detected at the door in the online video stream, a second image to be detected needs to be obtained from the stored video library to determine how long the plastic basin has been placed at the door.

Step S214, using the target garbage detection model to detect the second to-be-detected image to obtain a second detection result;

Step S215 , in the case that the target object exists in the second to-be-detected image according to the second detection result, determine that the target object is garbage.

Here, according to the second detection result, it is determined that there is a target object in the second to-be-detected image, and the target object may be at the same position in the two detection images, or may be the same suspected object that meets the distance requirement, and the position restriction may not be considered, as long as It is sufficient that the target object exists in both detection images. In the implementation process, because the beverage bottle or can is kicked for a short distance, or the position of the paper bag or packaging bag is changed by the wind, it is only necessary to confirm that the second image to be detected also exists in the second image to be detected. The target existing in the image to be detected is sufficient.

In the embodiment of the present application, the first image to be detected is obtained from the online video stream, and when it is determined that there is a target object in the first image to be detected, the second image to be detected is obtained from the stored video library, and the second image to be detected is obtained after determining When the target object also exists in the second image to be detected, it is determined that the target object is garbage. In this way, the data mining method is used to mine the online video stream and the stored video database data in the real scene, to quickly determine the difficult samples that are difficult to detect and identify by the target garbage detection model, and solve the problem that the traditional method cannot comprehensively collect data.

Step S301, acquiring a first image to be detected;

The first image to be detected is obtained from an online video stream.

Step S302, using the target garbage detection model to analyze the attribute information of the object in the first to-be-detected image;

Step S303, determining whether the corresponding object belongs to the target object or belongs to garbage according to the attribute information of the object in the first to-be-detected image, and obtains a first detection result;

The attribute information of the object includes at least one of the following: the shape, material, size, and location of the object.

In some embodiments, whether the corresponding object belongs to the target object or the garbage can be directly determined according to the shape of the object in the first image to be detected. For example, if the object is a plastic basin, if the shape of the plastic basin shows that the plastic basin is a deformed plastic basin, the object can be determined to be garbage; the object is a crushed beverage bottle, and the object can be determined to be garbage. In other cases, the object may be determined as the target object. In some embodiments, whether the corresponding object belongs to the target object or the garbage can be determined according to the material of the object in the first image to be detected. For example, if the material of the object is spoiled or dirty food, it can be determined that the object is garbage. In other cases, the object may be determined as the target object. In some embodiments, whether the corresponding object belongs to the target object or the garbage can be determined according to the size of the object in the first image to be detected. For example, if the object is half leftover fruit or incomplete paper products, it can be determined that the object is garbage. In other cases, the object may be determined as the target object. In some embodiments, whether the corresponding object belongs to the target object or the garbage can be determined according to the position of the object in the first image to be detected. For example, if the target object is food, and the food is placed on the ground of the canteen, it can be determined that the object is garbage. In other cases, the object may be determined as the target object.

Step S304, in the case where it is determined according to the first detection result that the target object exists in the first image to be detected, obtain a second image to be detected from a stored video library;

The first image to be detected is acquired from an online video stream, and the acquisition time of the second image to be detected is earlier than the acquisition time of the first image to be detected.

Step S305, using the target garbage detection model to analyze the attribute information of the object in the second to-be-detected image;

Step S306, according to the attribute information of the object in the second to-be-detected image, determine whether the corresponding object belongs to the target object or belongs to garbage, and obtain a second detection result;

Here, the attribute information of the object in the second image to be detected may be the same as the attribute information of the first image to be detected, that is, the attribute information of the object in the second image to be detected may be the shape, material, size of the object , the location. The second to-be-detected image is obtained from the stored video library by analyzing the target garbage detection model. There is also the possibility of garbage in the second to-be-detected image. As shown in step S303, the method for determining garbage is based on the attribute information of the object in the second to-be-detected image. , determine whether the corresponding object belongs to the target object or belongs to garbage, and obtain the second detection result.

Step S307 , in the case that the target object exists in the second to-be-detected image according to the second detection result, determine that the target object is garbage.

In some embodiments, the second detection result indicates that the same target object exists in the second image to be detected as in the first image to be detected, that is, within the same viewing angle range, the same object exists for a period of time, it can be determined that the target object exists in the second image to be detected. Objects are unmanaged, ie, garbage.

In the embodiment of the present application, because the garbage has some specific shapes, materials, sizes, and locations, it can be directly determined whether the target object is garbage according to the attribute information of the object. In this way, the efficiency and accuracy of garbage determination are greatly improved.

Step S311, acquiring a first image to be detected;

The first image to be detected is obtained from an online video stream.

Step S312, using the target garbage detection model to analyze the attribute information of the object in the first image to be detected; determine whether the corresponding object belongs to the target object or garbage according to the attribute information of the object in the first image to be detected, and obtain a first detection result ;

Step S313, when the first detection result is that the object belongs to garbage, determine the garbage category to which the garbage belongs and the location of the garbage according to the attribute information of the object;

Waste categories include recyclables, other waste, kitchen waste and hazardous waste. Among them, recyclables mainly include five categories of waste paper, plastic, glass, metal and cloth; other garbage (dry garbage) includes bricks, ceramics, muck, toilet waste paper, paper towels, etc. waste and dust, food bags (boxes); kitchen waste (wet waste) includes food waste such as leftovers, bones, vegetable roots and leaves, peels; hazardous waste refers to heavy metals that are harmful to human health, toxic Substances or wastes that cause actual or potential harm to the environment. In some embodiments, the garbage category to which the corresponding object belongs may be determined according to the form of the object. For example, if the object is in the form of a battery, it can be determined that the object is hazardous waste. In some embodiments, the garbage category to which the corresponding object belongs may be determined according to the material of the object. For example, if the material of the object is spoiled food, leftovers, bones, vegetable roots or peels, it can be determined that the object is kitchen waste. In some embodiments, the garbage category to which the corresponding object belongs may be determined according to the size and material of the object. For example, if the object is an incomplete paper product, it can be determined that the object is recyclable garbage. The location of the garbage can be determined based on the location of the object.

Step S314: Determine the content corresponding to the garbage alarm according to the garbage category and the location of the garbage;

In some embodiments, the content of the spam alert may include the category of the spam and the location of the spam. For example, the content of the alert can be the discovery of kitchen waste on the canteen floor, or the discovery of recyclable waste on the school playground. In this way, the user can determine the urgency to dispose of the garbage according to the type of garbage and the location of the garbage.

Step S315, sending the content corresponding to the garbage alarm to the garbage management platform;

The garbage management platform is connected with the terminal for managing garbage and the camera device for photographing garbage, and is configured to receive garbage alarms sent by the processing system of the camera device, and send the alarm to the terminal for garbage management. Here, the terminal for garbage management can be used by cleaners The terminal equipment can also be a cleaning robot.

Step S316: In the case where it is determined that there is a target object in the first image to be detected, acquire a second image to be detected, wherein the collection area of the first image to be detected and the collection area of the second image to be detected The overlap ratio between them is greater than a preset threshold, and the acquisition times of the first to-be-detected image and the second to-be-detected image are separated by a preset time interval;

Step S317 , in the case that it is determined that the target object exists in the second image to be detected, determine that the target object is garbage.

In this embodiment of the present application, firstly determine the garbage category to which the garbage belongs and the location of the garbage according to the attribute information of the object; then, determine the content corresponding to the garbage alarm according to the garbage category and the location of the garbage; finally, determine the content corresponding to the garbage alarm; Content sent to spam management platforms. In this way, when it is determined that there is garbage, the garbage information (garbage category, location) can be sent to the garbage management platform in time, so as to realize timely and reasonable disposal of garbage according to the garbage category and location.

Fig. 3 is another garbage detection method provided by the embodiment of the present application. As shown in Fig. 3, the method includes:

Step S321, acquiring a first image to be detected;

Step S322: Detect the first image to be detected, and determine the target object and the target frame corresponding to the first image to be detected; determine the first target object and the target frame corresponding to the first image to be detected. A cross-comparison;

In some embodiments, step S322 can be implemented by a target garbage detection model, and the target garbage detection model can realize the identification of objects in the image, and can also realize the marking of the positions of the objects. The target object and target frame corresponding to the first image to be detected can be determined by using the target garbage detection model. Here, the target object is the recognized object, and the target frame is the marked position of the object. Intersection-over-Union (IoU) is a concept used in object detection, which calculates the ratio of the intersection and union of the predicted frame and the real frame, that is, the ratio of their intersection and union. The ideal situation is complete overlap, i.e. a ratio of 1. Here, the real frame can be the target frame determined after each large detection image is input to the target garbage detection model, and the predicted frame can be the target frame determined after the first image to be detected enters and exits the target garbage detection model. That is to say, the predicted frame marked according to the position where the garbage appears in the first image to be detected.

Step S323, when it is determined that the target object exists in the first image to be detected, obtain a second image to be detected from a stored video library;

Step S324, using the target garbage detection model to analyze the attribute information of the object in the second image to be detected; according to the attribute information of the object in the second image to be detected, determine whether the corresponding object belongs to the target object or belongs to garbage, obtain the second test result;

Step S325, in the case of determining the target object corresponding to the second image to be detected, determine a second intersection ratio according to the target frame and the target object corresponding to the second image to be detected; When the union ratio and the second intersection ratio are greater than the preset intersection ratio threshold at the same time, the target object is determined to be garbage; in some embodiments, because the predicted frame used in the calculation of the second intersection ratio can be The target frame of the first image object to be detected, so the second intersection ratio is the ratio of the intersection and union of the target frame of the second image to be detected and the target frame of the first image to be detected. In the case where the first intersection ratio and the second intersection ratio are greater than the preset intersection ratio threshold, that is, the distance between the position of the target object in the first image to be detected and the position of the target object in the second image to be detected When the threshold requirement is met, it is determined that the target object is garbage.

Step S326, when it is determined that the target object is garbage, determine the location of the garbage according to the first image to be detected and/or the second image to be detected;

Here, since the position of the camera that captures the target object is fixed, the position of the camera can be used as the position where the garbage is located.

Step S327, determining the corresponding terminal or cleaning robot according to the location of the garbage;

Here, because the garbage cleaning in an area can correspond to the terminal or robot that manages the area, during the implementation process, a mapping relationship can be established between the location of the garbage and the terminal or cleaning robot of the location object, according to the mapping relationship and the location of the garbage. The position of the corresponding terminal or cleaning robot is determined.

Step S328: Send the garbage alarm to the terminal or the robot, so that the cleaner holding the terminal cleans the garbage or the cleaning robot processes the garbage.

In the embodiment of the present application, a method of evaluating whether the intersection ratio of target frames on different time frames continuously exceeds a certain threshold is used to determine whether the detected garbage appears in the picture for a long time. In this way, everyday objects that are similar in shape to garbage but do not belong to the type of garbage can be effectively excluded, thereby improving the performance of the embodiment of the present application in a real scene.

In the embodiment of the present application, first, the corresponding terminal or cleaning robot is determined according to the location of the garbage, and then the garbage alarm is sent to the terminal or robot, so that the cleaner who holds the terminal can clean up the garbage in time or Cleaning robots dispose of garbage in a timely manner.

Garbage detection that supports video stream analysis plays a very important role in garbage management. It can support garbage managers to efficiently detect and identify untreated garbage, so that garbage can be quickly removed. It plays an important role in the living standard of citizens. Therefore, garbage detection is an important class of problems.

The difficulties of garbage detection include two aspects:

On the one hand, the forms of garbage itself are very diverse, and the definitions are relatively vague and constantly changing. Garbage comes in a variety of shapes and types (cans, waste plastic bags, waste cartons...), and different types of waste come in different sizes, shapes, colors, and distances from the camera. This results in a huge diversity of size and morphology of target objects in garbage detection tasks, and it is difficult to collect them completely.

On the other hand, the definition of garbage is vague and constantly changing. The types of waste may change with the development of policies, economic construction and civic culture (for example, new types of beverages may generate new types of plastic bottle waste). At the same time, the definition of garbage is ambiguous and depends on the relationship with the surrounding environment (for example, the small plastic basin placed on the roadside may be abandoned garbage, or it may be temporarily placed by others), and this relationship with the environment is often Varies drastically with location. Another manifestation of this vague definition is the change of position and shape of an item over a period of time, while garbage often does not (for example, a plastic bag in hand will change position and shape over time, placed in The discarded plastic bags on the ground tend to be fixed for a period of time), which leads to the identification of certain garbage that often requires attention to video changes over a period of time, and garbage detection that only performs single-frame images may produce false positives.

The difficulties in these two aspects lead to the wide variety of target morphologies that need to be identified for garbage detection. This makes traditional deep learning-based single-frame garbage detection methods, that is, methods that collect large amounts of data and train deep learning networks for single-frame image detection, infeasible. Because the collected data often cannot contain the real data that can be encountered in real scenes, and garbage detection based on a single frame image ignores the changes of objects over a period of time and recognizes normal items as garbage. Therefore, the target detection model obtained by the traditional deep learning method often has the problem of low performance in the real environment.

FIG. 4A is a schematic diagram of the architecture of a target detection model provided by an embodiment of the present application. As shown in FIG. 4A , the target detection model in the embodiment of the present application may be a RetinaNet network model, and the RetinaNet network model may include a backbone network (deep residual Difference network 41 and feature pyramid network 42) and N networks 43 including classification subnetworks and regression subnetworks, wherein the classification subnetworks and regression subnetworks can be referred to as classification and regression subnetworks for short, and N can be taken as shown in Figure 4A. 3. The value can also be set according to actual needs. The backbone network is used to compute and output the convolutional feature maps of the entire input image. The classification subnet classifies the output of the backbone network, and the regression subnet is used to perform the convolutional bounding box regression task on the output of the backbone network.

The Feature Pyramid Net (FPN) 42 is used as the backbone network, built on the standard deep Residual Network (ResNet) 41. FPN extends ResNet through top-down and lateral connections to generate rich multi-scale convolutional feature pyramids. The idea of ResNet is to introduce deep residual to solve the problem of gradient disappearance, that is, let the convolutional network learn the residual mapping. ResNet can have 2 most basic blocks, one of which is an identity block, whose input and output dimensions remain the same, so the structure can be concatenated multiple times; the other basic block is a convolution block (Conv Block), the dimensions of its input and output are not the same, so continuous concatenation cannot be performed. The purpose of the convolution block is to change the dimension of the output feature vector.

A bottom-up path such as ResNet can be used for feature extraction, which computes feature maps at different scales regardless of the size of the input image. Top-down paths can upsample spatially coarser feature maps from higher pyramid levels, and lateral connections merge top-down and bottom-up layers with the same spatial size .

The classification subnet is a small fully convolutional network attached to each layer of the FPN. The regression subnet can be processed in parallel with the classification subnet, and its network structure is almost the same as the classification subnet, but does not share parameters.

The regression subnet can obtain different detection frames in the image, and the classification subnet can obtain the object categories in different detection frames. When the classification subnet determines that the object category in a detection frame is the target object, the detection box as the bounding box of the target object in this image.

FIG. 4B is a schematic flowchart of a garbage detection method provided by an embodiment of the present application. As shown in FIG. 4B , the method includes:

Step S401, cutting the video stream to obtain at least one image;

A small part of garbage detection cold-start data in the management scenario is usually collected by manually taking screenshots and marking in the video stream. The data size is only one thousand pieces, and the video stream can be obtained by using a camera device. In the implementation process of framing the video stream, the video stream usually contains hundreds to thousands of real-time videos of camera points, and it is necessary to perform the framing operation, that is, extract a video from the video stream every T time frame. Usually T is set to 10 minutes, and it can also be set according to actual needs. Using this frame clipping operation, at least one piece of image data can be obtained from the video stream for subsequent mining.

Step S402, inputting the image into the target detection model to obtain the posterior probability of the image;

In some embodiments, the target detection model may be a target detection network as shown in FIG. 4A , the target detection model may be a RetinaNet network model, and the RetinaNet network model may include a backbone network and N sub-networks including classification and regression network of networks. Input the image into the target detection model, and the posterior probability of the image can be obtained. Here, the posterior probability means that each time the target detection model outputs a region of interest box, it also outputs a probability value, which represents the probability that the region of interest box is a target sample.

Step S403, determining the mining data according to the posterior probability of the image;

In the case of mining data, the garbage monitoring model is run on a large number of images obtained and the posterior probability of model judgment is obtained. When the posterior probability is greater than the threshold t ₁ and less than the threshold t ₂ , the image data is deemed to be required. Mining labeled data. For example, in the case of actual use, the threshold t ₁ can be set to 20%, and t ₂ can be set to 80%, then, when the posterior probability is greater than 20% and less than 80%, the image data is identified as requiring mining annotations The data.

Step S404, merging the mining data into the cold start data set;

All data that needs to be mined and labeled are manually labeled and merged into the cold-start dataset. This process will collect more positive sample images of real scenes, and may also generate more negative sample false positives. Adding negative sample false positives to model training can help optimize the model’s suppression of false positives in real scenarios. Positive samples It helps the model to quickly adapt to the garbage shape and type of the current scene.

Step S405, using the cold start data set to train the target detection model.

Every once in a while, after accumulating enough mining and merged images, as shown in Figure 4A, input the cold start data set into the target detection model to train the target detection model. The trained target detection model will be more suitable for the current real scene. Garbage types and shapes, and quickly improve performance.

In the embodiment of the present application, the data mining method is used to mine the online video stream in the real scene, and the difficult samples that are difficult to be detected and identified by the current model are quickly obtained. Solve the problem that traditional methods cannot comprehensively collect data. Introduce manual annotation to label the difficult samples obtained in the previous step and add them to the training set for retraining, so as to correct the errors of the model through human prior knowledge, expand the capability boundary of the model, and make the model adapt to the garbage form in the real environment at this stage. Solve the problem of low performance of traditional method models in real scenes. Using data mining methods, potential high-value samples that are helpful for model improvement can be mined in huge video streams, which can effectively improve model performance in the environment of limited annotation and computing resources, and save a lot of new business costs for deep learning models. manpower and computational cost. Users can use this framework to quickly and iteratively improve the potential garbage detection applications in the garbage management system online under limited labor and computing resources, and quickly meet the performance requirements required by the business with less labor and computing costs. Continue to improve the model performance after that. It solves the problem that the prior art mainly uses a lot of manpower to mark the unlabeled data, which consumes manpower and computing resources.

The embodiment of the present application provides a method for detecting garbage, which is completed by a garbage detection system, and the garbage detection system includes a collection module, a detection module, and an alarm module. FIG. 4C is a schematic flowchart of a garbage detection method provided by an embodiment of the present application. As shown in FIG. 4C , the workflow is described as follows:

Step S410, the acquisition module acquires at least one frame of image of the same camera;

The images acquired by the same camera device may be images captured within the same angle range, or two camera devices that can capture images in the same range may be used. Here, images within the same angle range are mainly acquired, and the imaging device is not limited.

Step S411, the detection module performs garbage detection on the input image;

Inputting at least one frame of images of the same camera device into a target detection model to determine whether there is suspected garbage in each image.

In some embodiments, the object detection model shown in FIG. 4A may be used to determine whether there is suspected garbage in each image.

Step S412, the detection module determines whether suspected garbage is detected;

Here, if suspected garbage is detected using the method of step S412, the flow goes to step S413, and if no suspected garbage is detected, the process returns to step S411 to perform suspected garbage detection again.

Step S413, the detection module determines whether garbage is detected at the same position in the past continuous S frames;

First, it is necessary to acquire S frames of images, wherein the collection time of the S frames of images needs to be earlier than the collection time of the images in which garbage is detected; then, it can be used whether the intersection ratio of the target frame on different time frames continuously exceeds a certain threshold. method to determine whether the detected garbage appears in the picture for a long period of time, that is, to determine whether garbage was detected at the same position in the past S consecutive S frames. In this way, if the intersection ratio between the garbage detection result and the target frame is less than the threshold, it is considered that a new target has been added to the image; if the intersection ratio is greater than the threshold, it is considered that the same target (garbage) appears continuously in the same predicted frame.

Here, if suspected garbage is detected using the method of step S413, it is determined to be garbage, and the flow goes to step S414, and if no garbage is detected, it returns to step S411 to perform suspected garbage detection again.

Step S414, the alarming module sends an external alarm.

In a case where it is determined that garbage exists in the same position of each of the S-frame images, a warning is output.

In the embodiment of the present application, multi-frame logic is used to effectively exclude everyday objects that are similar in shape to garbage but not belonging to the type of garbage, improve the performance of the target garbage detection model in real scenes, and solve the problem that the existing technology only uses single-frame images The limitations of detection, and the problems that cannot be effectively excluded for everyday objects that are similar to the form of garbage and are easily confused. At the same time, multi-frame garbage confirmation logic is added to the real scene detection to exclude easily confused items and further improve the performance of the garbage detection framework. From the user's point of view, the user can use this garbage detection method to quickly and iteratively improve the potential target detection applications in intelligent video analysis or intelligent management online under limited labor and computing resources. The computing cost quickly reaches the performance requirements required by the business, and can continue to improve the model performance after that.

Based on the foregoing embodiments, the embodiments of the present application provide a garbage detection device, the device includes each of the modules included, and can be implemented by a processor in an electronic device; of course, it can also be implemented by a logic circuit.

FIG. 5 is a schematic structural diagram of a garbage detection device provided by an embodiment of the present application. As shown in FIG. 5 , the garbage detection device 500 includes:

a first acquisition module 501, configured to acquire a first image to be detected;

The second acquisition module 502 is configured to acquire a second to-be-detected image when it is determined that a target object exists in the first to-be-detected image, wherein the acquisition area of the first to-be-detected image is the same as the second to-be-detected image. The overlap ratio between the acquisition areas of the detection images is greater than a preset threshold, and the acquisition times of the first to-be-detected image and the second to-be-detected image are separated by a preset time interval;

The first determining module 503 is configured to determine that the target object is garbage when it is determined that the target object exists in the second to-be-detected image.

In some embodiments, the apparatus further includes an analysis module and a second determination module, wherein the analysis module is configured to analyze the scene information in the first image to be detected; the second determination module is configured to The scene information in the first image to be detected determines the target object.

In some embodiments, the apparatus further includes a third determination module configured to determine the time interval according to attribute parameters of the first image to be detected and/or attribute parameters of the target object; wherein the The attribute parameter of the first image to be detected includes at least one of the following: scene information of the first image to be detected, and the time period or season to which the acquisition time of the first image to be detected belongs.

In some embodiments, the first image to be detected is acquired from an online video stream, and the acquisition time of the second image to be detected is earlier than the acquisition time of the first image to be detected.

In some embodiments, the apparatus further includes a first detection module and a second detection module, wherein the first detection module is configured to use a target garbage detection model to detect the first to-be-detected extracted from the online video stream The image is detected to obtain a first detection result; the second acquisition module is further configured to, when it is determined according to the first detection result that the target object exists in the first image to be detected, from the stored video The second to-be-detected image is obtained from a library; the second detection module is further configured to use the target garbage detection model to detect the second to-be-detected image to obtain a second detection result; the first The determining module is further configured to determine that the target object is garbage when it is determined according to the second detection result that the target object exists in the second image to be detected.

In some embodiments, the first detection module is further configured to analyze the attribute information of the object in the first image to be detected by using the target garbage detection model; according to the attribute information of the object in the first image to be detected Determine whether the corresponding object belongs to the target object or belongs to garbage, and obtain a first detection result; the second detection module is configured to analyze the attribute information of the object in the second to-be-detected image by using the target garbage detection model; The attribute information of the object in the second to-be-detected image is used to determine whether the corresponding object belongs to the target object or the garbage, and a second detection result is obtained.

In some embodiments, the attribute information of the object includes at least one of the following: shape, material, size, and location of the object.

In some embodiments, when the first detection result is that the object belongs to garbage, the apparatus further includes a fourth determining module, a fifth determining module, and a first sending module, wherein the fourth determining module a module configured to determine the garbage category to which the garbage belongs and the location of the garbage according to the attribute information of the object; the fifth determination module is configured to determine the garbage category and the location of the garbage according to the garbage category , determine the content corresponding to the garbage alarm; the first sending module is configured to send the content corresponding to the garbage alarm to the garbage management platform.

In some embodiments, the first detection module is further configured to detect the first image to be detected, and determine a target object and a target frame corresponding to the first image to be detected; The target object corresponding to the image and the target frame determine a first intersection ratio; the first determination module is further configured to determine the target object corresponding to the second image to be detected, according to the target frame and the target frame. The target frame corresponding to the first to-be-detected image determines a second intersection ratio; when the first intersection ratio and the second intersection ratio are greater than a preset intersection ratio threshold at the same time, determine the target object for garbage.

In some embodiments, the apparatus further includes a sixth determination module, a seventh determination module, and a second sending module, wherein the sixth determination module is configured to, in the case of determining that the target object is garbage, according to the The first image to be detected and/or the second image to be detected determines the location of the garbage; a seventh determination module is used to determine the corresponding terminal or cleaning robot according to the location of the garbage; the second sending The module is configured to send the garbage alarm to the terminal or the robot, so that the cleaner holding the terminal cleans the garbage or the cleaning robot processes the garbage.

In some embodiments, the apparatus further includes a third acquisition module, a fourth acquisition module and a training module, wherein the third acquisition module is configured to acquire at least one target image; the target image is obtained from the video stream The to-be-detected image intercepted in the 2000 is input to the initial garbage detection model, and is determined according to the detection result output by the initial target garbage detection model; the initial garbage detection model is trained by using the first data set; wherein, the first A dataset is a dataset in which at least some sample images have annotation information; the fourth acquisition module is configured to acquire a manual annotation result for the at least one target image, and use the annotated at least one target image as a The training samples are merged into the first data set to obtain a second data set; the training module is configured to use the second data set to train the initial garbage detection model to obtain the target garbage detection model.

In some embodiments, the third acquisition module includes an input sub-module and a determination sub-module, and the input sub-module is configured to input the at least one frame of the image to be detected into the initial garbage detection model, and obtain the information of each frame. the posterior probability of the image to be detected; the determining sub-module is configured to, when determining that the posterior probability is greater than the first probability threshold and less than the second probability threshold, determine the image to be detected corresponding to the posterior probability The target image is determined, wherein the first probability threshold is smaller than the second probability threshold.

The descriptions of the above apparatus embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects to the method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the descriptions of the method embodiments of the present application for understanding.

It should be noted that, in the embodiments of the present application, if the above garbage detection method is implemented in the form of a software function module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be embodied in the form of software products in essence or the parts that contribute to related technologies. The computer software products are stored in a storage medium and include several instructions to make An electronic device (which may be a mobile phone, a tablet computer, a notebook computer, a desktop computer, a robot, a server, etc.) executes all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read only memory (Read Only Memory, ROM), a magnetic disk or an optical disk and other media that can store program codes. As such, the embodiments of the present application are not limited to any specific combination of hardware and software.

Correspondingly, the embodiments of the present application provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps in the garbage detection method provided in the foregoing embodiments.

Correspondingly, an embodiment of the present application provides an electronic device, and FIG. 6 is a schematic diagram of a hardware entity of an electronic device provided by an embodiment of the present application. As shown in FIG. 6 , the hardware entity of the device 600 includes: a memory 601 and a processing The memory 601 stores a computer program that can be executed on the processor 602, and the processor 602 implements the steps in the methods provided in the above embodiments when the processor 602 executes the program.

The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data to be processed or processed by the processor 602 and various modules in the electronic device 600 (eg, image data, audio data, voice communication data and Video communication data), which can be realized by flash memory (FLASH) or random access memory (Random Access Memory, RAM).

It should be pointed out here that the descriptions of the above storage medium and device embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects to the method embodiments. For technical details not disclosed in the embodiments of the storage medium and device of the present application, please refer to the description of the method embodiments of the present application to understand.

It is to be understood that reference throughout the specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic associated with the embodiment is included in at least one embodiment of the present application. Thus, appearances of "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily necessarily referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in various embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, and should not be dealt with in the embodiments of the present application. implementation constitutes any limitation. The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined, or Can be integrated into another system, or some features can be ignored, or not implemented. In addition, the coupling, or direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical or other forms. of.

The unit described above as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit; it may be located in one place or distributed to multiple network units; Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may all be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above integration The unit can be implemented either in the form of hardware or in the form of hardware plus software functional units.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by program instructions related to hardware, the aforementioned program can be stored in a computer-readable storage medium, and when the program is executed, the execution includes: The steps of the above method embodiments; and the aforementioned storage medium includes: a removable storage device, a read only memory (Read Only Memory, ROM), a magnetic disk or an optical disk and other media that can store program codes.

Alternatively, if the above-mentioned integrated units of the present application are implemented in the form of software function modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be embodied in the form of software products in essence or the parts that contribute to related technologies. The computer software products are stored in a storage medium and include several instructions to make An electronic device (which may be a mobile phone, a tablet computer, a notebook computer, a desktop computer, a robot, a server, etc.) executes all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a removable storage device, a ROM, a magnetic disk, or an optical disk.

The features disclosed in several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

The above is only the embodiment of the present application, but the protection scope of the present application is not limited to this. Covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Industrial Applicability

In the embodiment of the present application, the first image to be detected is firstly acquired, then the second image to be detected is acquired when it is determined that there is a target object in the first image to be detected, and finally it is determined that there is a target object in the second image to be detected In this case, the target object is determined to be garbage. Using multi-frame logic, it is possible to effectively exclude everyday objects that are similar in shape to garbage, but do not belong to the type of garbage, and at the same time determine that objects whose position has not changed significantly for a long time are garbage.

Claims

A garbage detection method, the method is performed by an electronic device, and the method includes:

obtaining the first image to be detected;

In the case where it is determined that there is a target object in the first image to be detected, a second image to be detected is acquired, wherein the distance between the acquisition area of the first image to be detected and the acquisition area of the second image to be detected is The overlap ratio is greater than a preset threshold, and the acquisition times of the first to-be-detected image and the second to-be-detected image are separated by a preset time interval;

When it is determined that the target object exists in the second image to be detected, it is determined that the target object is garbage.
The method of claim 1, wherein the method further comprises:

analyzing scene information in the first image to be detected;

The target object is determined according to scene information in the first image to be detected.
The method of claim 1 or 2, wherein the method further comprises:

determining the time interval according to the attribute parameter of the first image to be detected and/or the attribute parameter of the target object;

Wherein, the attribute parameter of the first image to be detected includes at least one of the following: scene information of the first image to be detected, and the time period or season to which the acquisition time of the first image to be detected belongs.
The method according to any one of claims 1 to 3, wherein the first image to be detected is obtained from an online video stream, and the time of the second image to be detected is earlier than the time of the first image to be detected collection time.
The method of claim 4, wherein the method further comprises: using a target garbage detection model to detect the first image to be detected extracted from the online video stream to obtain a first detection result;

The acquiring a second image to be detected when it is determined that a target object exists in the first image to be detected includes: determining that the target object exists in the first image to be detected according to the first detection result In the case of , obtain the second to-be-detected image from the stored video library;

Correspondingly, the method further includes: using the target garbage detection model to detect the second to-be-detected image to obtain a second detection result;

The determining that the target object is garbage when it is determined that the target object exists in the second image to be detected includes: determining that the target object exists in the second image to be detected according to the second detection result; In the case of the target object, it is determined that the target object is garbage.
The method of claim 5, wherein the detecting the first to-be-detected image extracted from the online video stream by using a target garbage detection model to obtain a first detection result comprises:

Use the target garbage detection model to analyze the attribute information of the object in the first image to be detected; determine whether the corresponding object belongs to the target object or garbage according to the attribute information of the object in the first image to be detected, and obtain the first detection result;

The use of the target garbage detection model to detect the second to-be-detected image to obtain a second detection result includes:

Use the target garbage detection model to analyze the attribute information of the object in the second image to be detected; according to the attribute information of the object in the second image to be detected, determine whether the corresponding object belongs to the target object or belongs to garbage, and obtain the second Test results.
The method of claim 6, wherein the attribute information of the object includes at least one of the following: shape, material, size, and location of the object.
The method according to claim 6 or 7, wherein, when the first detection result is that the object belongs to garbage, the method further comprises:

Determine the garbage category to which the garbage belongs and the location of the garbage according to the attribute information of the object;

Determine the content corresponding to the garbage alarm according to the garbage category and the location of the garbage;

Send the content corresponding to the garbage alarm to the garbage management platform.
The method according to any one of claims 6 to 8, wherein the detecting the first image to be detected extracted from the online video stream by using a target garbage detection model to obtain a first detection result, further comprising: Detecting the first to-be-detected image to determine a target object and a target frame corresponding to the first to-be-detected image; determining a first intersection according to the target object and the target frame corresponding to the first to-be-detected image Compare;

Correspondingly, when it is determined according to the second detection result that the target object exists in the second to-be-detected image, determining that the target object is garbage includes:

In the case of determining the target object corresponding to the second image to be detected, a second intersection ratio is determined according to the target frame and the target object corresponding to the second image to be detected; in the first intersection ratio and When the second intersection ratio is greater than a preset intersection ratio threshold at the same time, it is determined that the target object is garbage.
The method according to any one of claims 5 to 9, wherein the target garbage detection model is obtained by adopting the following steps, including:

Obtain at least one target image; the target image is input to the initial garbage detection model from the image to be detected intercepted from the video stream, and determined according to the detection result output by the initial garbage detection model; the initial garbage detection model is The first data set is used for training; wherein, the first data set is a data set in which at least some of the sample images have annotation information;

Obtain the manual labeling result of the at least one target image, and merge the labelled at least one target image into the first data set as a training sample to obtain a second data set;

The initial garbage detection model is trained by using the second data set to obtain the target garbage detection model.
The method of claim 10, wherein the acquiring at least one target image comprises:

Inputting the to-be-detected image into the initial garbage detection model to obtain the posterior probability of each frame of the to-be-detected image;

When it is determined that the posterior probability is greater than a first probability threshold and less than a second probability threshold, the image to be detected corresponding to the posterior probability is determined as the target image, wherein the first probability threshold is less than the second probability threshold.
A garbage detection device, comprising:

a first acquisition module, configured to acquire a first image to be detected;

The second acquisition module is configured to acquire a second image to be detected when it is determined that there is a target object in the first image to be detected, wherein the acquisition area of the first image to be detected is the same as the second image to be detected The overlap ratio between the image acquisition areas is greater than a preset threshold, and the acquisition times of the first to-be-detected image and the second to-be-detected image are separated by a preset time interval;

The first determination module is configured to determine that the target object is garbage when it is determined that the target object exists in the second to-be-detected image.
The apparatus of claim 12, wherein the apparatus further comprises:

A third determining module, configured to determine the time interval according to the attribute parameters of the first image to be detected and/or the attribute parameters of the target object; wherein the attribute parameters of the first image to be detected at least include the following One: the scene information of the first image to be detected, the time period or season to which the acquisition time of the first image to be detected belongs.
The apparatus of claim 13, wherein the apparatus further comprises:

The first detection module is configured to use the target garbage detection model to detect the first image to be detected extracted from the online video stream to obtain a first detection result;

The second acquisition module is further configured to acquire the second to-be-detected image from a stored video library when it is determined according to the first detection result that the target object exists in the first to-be-detected image ;

The second detection module is further configured to use the target garbage detection model to detect the second to-be-detected image to obtain a second detection result;

The first determining module is further configured to determine that the target object is garbage when it is determined according to the second detection result that the target object exists in the second image to be detected.
The apparatus of claim 14, wherein,

The first detection module is further configured to analyze the attribute information of the object in the first to-be-detected image by using the target garbage detection model; The target object is still garbage, and the first detection result is obtained;

The second detection module is configured to analyze the attribute information of the object in the second to-be-detected image by using the target garbage detection model; according to the attribute information of the object in the second to-be-detected image, determine that the corresponding object belongs to the The target object still belongs to garbage, and a second detection result is obtained.
The apparatus of claim 15, wherein,

The first detection module is further configured to detect the first image to be detected, and to determine a target object and a target frame corresponding to the first image to be detected; the target frame, determine the first cross-union ratio;

The first determining module is further configured to determine a second intersection ratio according to the target frame and the target frame corresponding to the first image to be detected when the target object corresponding to the second image to be detected is determined. ; In the case that the first intersection ratio and the second intersection ratio are greater than a preset intersection ratio threshold at the same time, determine that the target object is garbage.
The apparatus of any one of claims 14 to 16, wherein the apparatus further comprises:

The third acquisition module acquires at least one target image; the target image is input to the initial garbage detection model from the to-be-detected image intercepted from the video stream, and is determined according to the detection result output by the initial garbage detection model; the The initial garbage detection model is trained by using a first data set; wherein, the first data set is a data set in which at least part of the sample images have annotation information;

a fourth acquisition module, configured to acquire a result of manual annotation of the at least one target image, and merge the marked at least one target image into the first data set as a training sample to obtain a second data set;

A training module configured to use the second data set to train the initial garbage detection model to obtain the target garbage detection model.
An electronic device comprising: a memory and a processor,

the memory stores a computer program executable on the processor,

When the processor executes the computer program, the steps in the garbage detection method of any one of claims 1 to 11 are implemented.
A computer storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to realize the garbage according to any one of claims 1 to 11 steps in the detection method.
A computer program product comprising one or more instructions adapted to be loaded by a processor and to perform the steps in the garbage detection method of any one of claims 1 to 11 .