CN114332826B

CN114332826B - Vehicle image recognition method and device, electronic equipment and storage medium

Info

Publication number: CN114332826B
Application number: CN202210228729.9A
Authority: CN
Inventors: 惠强; 熊剑平; 王枫; 任馨怡; 梅少杰; 杨倩茹
Original assignee: Zhejiang Dahua Technology Co Ltd
Current assignee: Zhejiang Dahua Technology Co Ltd
Priority date: 2022-03-10
Filing date: 2022-03-10
Publication date: 2022-07-08
Anticipated expiration: 2042-03-10
Also published as: CN114332826A

Abstract

The application relates to the technical field of image processing, in particular to a vehicle image identification method, a vehicle image identification device, electronic equipment and a storage medium, and aims to improve detection efficiency in a vehicle unloading process. The method comprises the following steps: acquiring a video to be detected, which is acquired aiming at a target unloading vehicle, and carrying out target detection on a carriage of the target unloading vehicle contained in each video frame in the video to be detected; determining a first degree of overlap of the discharging tool and the car detected from each first video frame if it is determined that the car enters the discharging area based on a car detection result of the car of the target discharging vehicle; the first video frame comprises a video frame after a carriage of a determined target unloading vehicle in the video to be detected enters the unloading area; and determining a target video frame from the first video frames according to the determined first overlapping degrees, and detecting the unloading process of the target unloading vehicle based on the target video frame. This application snatchs video image through carriage and the overlap degree of the instrument of unloading, can effectively improve detection effect.

Description

Vehicle image recognition method and device, electronic equipment and storage medium

Technical Field

The present application relates to the field of image processing technologies, and in particular, to a vehicle image recognition method and apparatus, an electronic device, and a storage medium.

Background

In a steel smelting scrap scene, a discharging tool is generally required to be used for discharging the scrap steel in a carriage. Through image processing analysis technique, the image of the interior scrap steel of carriage is snatched to intelligence at the freight train in-process of unloading, makes quality testing personnel can long-rangely supervise and the quality aassessment to the process of unloading, can improve quality testing efficiency simultaneously under the prerequisite of guaranteeing quality testing personnel safety.

In the related art, it is common to analyze the characteristics of the car alone or analyze the motion state of the suction cup alone, and then capture an image of the scrap steel in the car according to the analysis result. Under these two kinds of modes, can only pick the image according to comparatively unilateral information, lead to the image that actually snatchs to be not conform to actual demand easily, lead to detection efficiency lower.

Therefore, how to rapidly capture effective images and improve the detection efficiency in the vehicle discharging process is an urgent need to be solved.

Disclosure of Invention

The embodiment of the application provides a vehicle image identification method and device, electronic equipment and a storage medium, and aims to improve detection efficiency in a vehicle unloading process.

The vehicle image identification method provided by the embodiment of the application comprises the following steps:

acquiring a video to be detected collected by aiming at a target unloading vehicle, and carrying out target detection on a carriage of the target unloading vehicle contained in each video frame in the video to be detected; and

determining a first degree of overlap of the discharging tool and the car detected from each first video frame if it is determined that the car enters the discharging area based on a car detection result of the car of the target discharging vehicle; the first video frame comprises a video frame after the carriage enters the unloading area in the video to be detected; the unloading tool is used for unloading materials in the compartment of the target unloading vehicle;

and determining a target video frame from the first video frames according to the determined first overlapping degrees, and detecting the unloading process of the target unloading vehicle based on the target video frame.

The embodiment of the application provides a vehicle image recognition device, includes:

the analysis unit is used for acquiring a video to be detected collected by aiming at a target unloading vehicle and carrying out target detection on a carriage of the target unloading vehicle contained in each video frame in the video to be detected; and

and the detection unit is used for determining a target video frame from the first video frames according to the determined first overlapping degrees and detecting the unloading process of the target unloading vehicle based on the target video frame.

Optionally, the analysis unit is further configured to:

in the process of sequentially carrying out target detection according to the time sequence of each video frame in the video to be detected, if a second video frame appears in the video to be detected, determining that the carriage enters a discharging area;

the second video frame is a video frame with a corresponding second overlapping degree larger than a first threshold, and the second overlapping degree corresponding to each video frame is as follows: an overlap between the car and the discharge area determined based on the respective car detection results.

Optionally, the analysis unit is further configured to:

carrying out target detection on the unloading tools contained in each video frame in the video to be detected to obtain the respective unloading tool detection results of each video frame;

the analysis unit is specifically configured to:

determining a first overlapping degree of the unloading tool and the compartment in the corresponding video frame based on the unloading tool detection result and the compartment detection result respectively corresponding to each first video frame; or alternatively

And after the carriage unloading state of the target unloading vehicle enters a working state, determining a first overlapping degree of the unloading tool and the carriage in the corresponding video frame based on the average carriage detection result of the carriage of the target unloading vehicle and the unloading tool detection result corresponding to each first video frame.

Optionally, the car detection result includes: a car detection box for identifying a location of a car of the target dump vehicle in a video frame; the discharging tool detection result comprises: a discharge tool detection box for identifying the position of the discharge tool in a video frame;

the analysis unit is specifically configured to:

for each first video frame, the following operations are performed:

and taking the overlapping degree between the unloading tool detection frame and the carriage detection frame corresponding to one first video frame as the first overlapping degree of the one first video frame.

Optionally, the analysis unit is further configured to determine that the car unloading state enters the working state by:

after the compartment of the target unloading vehicle enters an unloading area, if N continuous frames of first video frames with first characteristics are detected, determining that the unloading state of the compartment of the target unloading vehicle enters a working state, wherein N is a positive integer greater than 1;

wherein, in the first video frames of which the consecutive N frames have the first characteristic: the third degree of overlap between each video frame and the car of the adjacent previous frame video frame is greater than the second threshold.

Optionally, the car detection result includes: a car detection box for identifying a location of a car of the target dump vehicle in a video frame; the discharging tool detection result comprises: the discharging tool detection frame is used for identifying the position of the discharging tool in a video frame;

the average car detection result includes: based on the car detection box in the first video frame of which the N continuous frames have the first characteristic, determining an average detection box for the car;

the analysis unit is specifically configured to:

and respectively executing the following operations for each first video frame after the unloading state of the carriage enters the working state:

and taking the overlapping degree between the discharging tool detection frame corresponding to one first video frame and the average detection frame as the first overlapping degree corresponding to the one first video frame.

Optionally, the detection unit is specifically configured to:

determining a candidate video frame set corresponding to each preset time interval; the candidate video frame set comprises video frames, of which the first overlapping degree is smaller than a third threshold value, of first video frames detected in a corresponding preset time interval;

and determining the candidate video frame with the latest acquisition time in each candidate video frame set as the target video frame.

Optionally, the apparatus further comprises:

the ending unit is used for stopping target detection if a first video frame with a second characteristic appears in the video to be detected after the carriage unloading state enters the working state and in the process of sequentially carrying out target detection according to the time sequence of each video frame in the video to be detected;

wherein the first video frame with the second characteristic is: and the fourth overlapping degree of the corresponding compartment detection frame and the average detection frame is less than the first video frame of a fourth threshold value.

Optionally, the detection unit is specifically configured to:

determining whether the material is doped with other substances or not by detecting the material in the carriage in the target video frame; or

And determining the quality of the material by detecting the material in the carriage in the target video frame.

An electronic device provided by an embodiment of the present application includes a processor and a memory, where the memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to execute any one of the steps of the vehicle image recognition method.

An embodiment of the present application provides a computer-readable storage medium, which includes a computer program, when the computer program runs on an electronic device, the computer program is configured to enable the electronic device to execute any one of the steps of the vehicle image recognition method.

Embodiments of the present application provide a computer program product, which includes a computer program, stored in a computer readable storage medium; when the processor of the electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the vehicle image recognition methods described above.

The beneficial effect of this application is as follows:

the embodiment of the application provides a vehicle image identification method and device, electronic equipment and a storage medium. The method comprises the steps of detecting a carriage of a target unloading vehicle contained in each video frame through a target detection method, and obtaining a carriage detection result of each video frame; and under the condition that the compartment of the target unloading vehicle enters the unloading area based on the compartment detection result, determining the first overlapping degree of the unloading tool and the compartment detected from each first video frame, taking the overlapping degree as reference, and capturing video images from the video frames after the compartment enters the unloading area determined from the video to be detected, so as to ensure that the captured images are effective images which are not shielded by the unloading tool.

Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

fig. 1 is a schematic diagram of an application scenario in an embodiment of the present application;

fig. 2 is a schematic flowchart of a vehicle image recognition method in an embodiment of the present application;

FIG. 3A is a schematic view of a video frame of a first video frame according to an embodiment of the present application;

FIG. 3B is a schematic view of a second video frame in the embodiment of the present application;

FIG. 4 is a schematic diagram illustrating a calculation method of the degree of overlap in the embodiment of the present application;

FIG. 5A is a schematic flow chart illustrating another vehicle image recognition method according to an embodiment of the present application;

fig. 5B is a state transition diagram of a car unloading state in the embodiment of the present application;

FIG. 6A is a video frame diagram of a third video frame according to an embodiment of the present application;

FIG. 6B is a video frame diagram of a fourth video frame in the embodiment of the present application;

FIG. 6C is a schematic view of a fifth video frame in the embodiment of the present application;

FIG. 6D is a video frame diagram of a sixth video frame in an embodiment of the present application;

fig. 7 is a schematic flowchart of a vehicle image capturing method according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of a vehicle image recognition apparatus according to an embodiment of the present application;

fig. 9 is a schematic diagram of a hardware component structure of an electronic device to which an embodiment of the present application is applied;

fig. 10 is a schematic diagram of a hardware component structure of another electronic device to which the embodiment of the present application is applied.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments, but not all embodiments, of the technical solutions of the present application. All other embodiments obtained by a person skilled in the art without any inventive step based on the embodiments described in the present application are within the scope of the protection of the present application.

Some concepts related to the embodiments of the present application are described below.

A discharging tool: the tool for unloading the material in the compartment of the target unloading vehicle can be a tool fixed at a certain position outside the unloading vehicle or a movable tool on the unloading vehicle.

A discharging area: the method comprises the steps that a working area, namely an unloading area, which is required when a target unloading vehicle is unloaded and is identified in a video picture is designated, and the target unloading vehicle unloads in the area; in the embodiment of the present application, the unloading area is a preconfigured ROI (Region of Interest), the shape is not specifically limited, and the quadrilateral unloading area is mainly exemplified herein.

The preset time interval is as follows: the method is characterized by comprising the steps of capturing images in a video to be detected, and detecting the unloading process of a target unloading vehicle. When images are captured based on a preset time interval, the images can be captured once at intervals, and the effect of layered capturing of the carriage material images is achieved.

And (3) detection results: the method refers to a detection result of an identified target in the process of performing target detection on each video frame of a video to be detected. In the embodiment of the application, the detection target comprises a carriage and an unloading tool, and correspondingly, the detection result also comprises a carriage detection result and an unloading tool detection result. Specifically, the car detection result of a certain video frame indicates whether a car is detected in the video frame, and if a car is detected, the car detection result may further include position information, boundary information, and the like of the car in the video frame. Similarly, the discharging tool detection result indicates whether the discharging tool is detected in the video frame, and if the discharging tool is detected, the discharging tool detection result may further include position information, boundary information, and the like of the discharging tool in the video frame.

Target video frame: the video frames are screened from the videos to be detected and used for detecting the unloading process of the target unloading vehicle. In the embodiment of the application, the target video frame is screened from the video frames after the target unloading vehicle enters the unloading area, which generally means that the unloading tool does not block the video frame inside the carriage.

The unloading state of the carriage is as follows: and the current state of the compartment of the target unloading vehicle is represented, and the compartment is not always in the unloading state in the whole process, and other states exist. In the embodiment of the application, three carriage unloading states are defined, which are respectively: unknown, START, work. Wherein unknown state is an unknown state, namely an initial state; START is in the START state, and represents the preparation stage before unloading or the stage after unloading; WORKING is the WORKING state and represents the unloading stage.

The following briefly introduces the design concept of the embodiments of the present application:

the scrap steel is one of the main raw materials in steel production, and environmental pollution and resource waste can be reduced by using the scrap steel for steel making. In recent years, the purchasing amount of the scrap steel of each steel plant is larger and larger, but the source consistency of the scrap steel is poor, and manual observation and personal experience are required to be used as the basis of quality evaluation. However, the environment of the waste steel unloading site is poor, a large-scale sucking disc is generally needed to be used for unloading the waste steel in the carriage, quality inspection personnel often need to approach to observe, and safety cannot be guaranteed. Consequently, through the image processing analysis technique in the freight train image of steel scrap in the carriage of intelligent snatching of the in-process of unloading, make quality testing personnel can long-rangely supervise and the quality assessment to the process of unloading, can improve quality testing efficiency simultaneously under the prerequisite of guaranteeing quality testing personnel's safety.

Taking a steel smelting scrap scene as an example, in the related technology, the following two methods for intelligently capturing the scrap steel image in the carriage are provided:

the method comprises the following steps: the unloading state of the carriage is judged through the characteristics and the positions of the region of interest and the carriage, and whether the truck is a scrap steel unloading truck or a scrap steel stacking truck or whether two trucks exist simultaneously is judged.

The method lacks analysis on the unloading sucker, and image grabbing by the technology possibly causes that scrap steel in a carriage is shielded by the sucker in a grabbed image, so that supervision and evaluation of quality inspection personnel are not facilitated.

The method 2 comprises the following steps: the position of the sucker is judged by training a target detection network, and whether the sucker works or not is judged by comparing the position change between the front frame and the rear frame.

The method only analyzes the motion state of the sucker and analyzes whether the area is unloading or not by judging the motion state of the sucker. However, in the scene without a truck, the sucker can move to suck scattered steel scraps on the ground, and the image cannot be directly captured for quality inspection and supervision based on the technology.

In view of this, the embodiment of the present application provides a vehicle image identification method and apparatus, an electronic device, and a storage medium. The method comprises the steps of detecting a carriage of a target unloading vehicle contained in each video frame through a target detection method, and obtaining a carriage detection result of each video frame; and under the condition that the compartment of the target unloading vehicle enters the unloading area based on the compartment detection result, determining the first overlapping degree of the unloading tool and the compartment detected from each first video frame, taking the overlapping degree as reference, and capturing video images from the video frames after the compartment enters the unloading area determined from the video to be detected, so as to ensure that the captured images are effective images which are not shielded by the unloading tool.

The preferred embodiments of the present application will be described below with reference to the accompanying drawings of the specification, it should be understood that the preferred embodiments described herein are merely for illustrating and explaining the present application, and are not intended to limit the present application, and that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.

Fig. 1 is a schematic view of an application scenario in the embodiment of the present application. The application scenario diagram includes two terminal devices 110 and a server 120.

In the embodiment of the present application, the terminal device 110 includes, but is not limited to, a mobile phone, a tablet computer, a notebook computer, a desktop computer, an e-book reader, an intelligent voice interaction device, an intelligent household appliance, a vehicle-mounted terminal, and other devices; the terminal device may be installed with a client related to target detection, vehicle identification, and the like, where the client may be software (e.g., a browser, and the like) or a web page, an applet, and the server 120 is a background server corresponding to the software or the web page, the applet, or a server specially used for target detection, vehicle identification, and the like, and the application is not limited specifically. The server 120 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a web service, cloud communication, a middleware service, a domain name service, a security service, a CDN (Content Delivery Network), a big data and artificial intelligence platform.

It should be noted that the vehicle image recognition method in the embodiments of the present application may be executed by an electronic device, which may be the server 120 or the terminal device 110, that is, the method may be executed by the server 120 or the terminal device 110 alone, or may be executed by both the server 120 and the terminal device 110. For example, when the terminal device 110 and the server 120 are executed together, the terminal device 110 acquires a video to be detected for a target unloading vehicle, and sends the video to the server 120, and the server 120 performs target detection on the video to be detected to obtain detection results of each video frame, which at least include a carriage detection result and may also include an unloading tool detection result; and then, analyzing the first overlapping degree of the unloading tool and the carriage in the corresponding video frame, screening the target video frame according to the determined first overlapping degrees, detecting the unloading process of the target unloading vehicle based on the target video frame, feeding the detection result back to the terminal device 110, and displaying the detection result to a user by the terminal device 110.

In an alternative embodiment, terminal device 110 and server 120 may communicate via a communication network.

In an alternative embodiment, the communication network is a wired network or a wireless network.

It should be noted that fig. 1 is only an example, and the number of the terminal devices and the servers is not limited in practice, and is not specifically limited in the embodiment of the present application.

In the embodiment of the application, when the number of the servers is multiple, the multiple servers can be combined into a block chain, and the servers are nodes on the block chain; according to the vehicle image identification method disclosed by the embodiment of the application, the video data, the video detection result and the like which are related to the vehicle image identification method can be stored on the block chain.

In addition, the embodiment of the application can be applied to various scenes, including but not limited to cloud technology, artificial intelligence, intelligent transportation, driving assistance and the like. For example, when the method is applied to a steel smelting scrap scene, in the whole process from driving to discharging of a target discharging vehicle, a video to be detected is detected through a camera (a camera), the target detection is carried out on the video to be detected, the carriage discharging state of the target discharging vehicle is analyzed according to the detection result, whether the condition of a carriage is detected or not is combined with the carriage discharging state, a target video frame is determined according to the overlapping degree of the carriage, a sucker and the like, and then quality inspection personnel can remotely supervise and evaluate the discharging process based on the target video frame, so that the quality inspection efficiency is improved on the premise of ensuring the safety of the quality inspection personnel. For another example, the process is also similar when applied to a discharging scene of other cargoes, and the process is not particularly limited herein.

The vehicle image recognition method provided by the exemplary embodiment of the present application is described below with reference to the drawings in conjunction with the application scenarios described above, it should be noted that the application scenarios described above are only shown for the convenience of understanding the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect.

Referring to fig. 2, an implementation flowchart of a vehicle image recognition method provided in the embodiment of the present application, taking a server as an execution subject, takes the following steps S21-S24 as an implementation flow of the method:

s21: the method comprises the steps that a server obtains a to-be-detected video collected by a target unloading vehicle, and target detection is carried out on a carriage of the target unloading vehicle contained in each video frame in the to-be-detected video; and determining a first degree of overlap of the discharging tool and the car detected from each of the first video frames if it is determined that the car enters the discharging region based on a car detection result of the car of the target discharging vehicle.

The first video frame includes a video frame after the determined compartment enters the unloading area in the video to be detected, and the unloading area refers to a working area required when the target unloading vehicle identified in the video picture unloads, and is a shaded area in fig. 3A, that is, the unloading area is indicated. The unloading tool is used for unloading materials in the compartment of the target unloading vehicle.

In the embodiment of the application, a steel smelting scrap scene is mainly taken as an example for illustration, in the scene, the carriage of the target unloading vehicle is loaded with the scrap to be unloaded, and the unloading tool can be a sucking disc and is used for sucking the scrap out of the carriage so as to realize unloading (namely unloading).

Fig. 3A is a schematic view of a video frame of a first type of video frame listed in the embodiments of the present application. In the embodiment of the present application, the process of performing target detection on each video frame of a video to be detected can be implemented based on a pre-trained target detection model.

In step S21, in addition to the need to perform target detection on the car of the target discharging vehicle included in each video frame in the video to be detected to obtain a car recognition result, the need to perform target detection on the discharging tool included in the video frame to obtain a discharging tool recognition result is also required. However, in the present application, it is considered that only the first overlap degree needs to be calculated for the first video frame, and thus the video frame that needs to be detected by the discharging tool may be each video frame in the video to be detected, or may also be the video frame that needs to be detected by the discharging tool only for each first video frame, which is exemplified herein by taking each video frame as an example.

Since the target detection model in the embodiment of the present application is mainly used for detecting the car and the unloading tool (e.g. the suction cup), car and suction cup labeling data need to be constructed to train the target detection model.

Specifically, the positions of the car and the sucker in the (historically collected) sample image are marked through a rectangular area, the label of the car is set to 0, the label of the sucker is set to 1, one complete car marking data can be recorded as { 0, x, y, w, h }, and one complete sucker marking data can be recorded as { 1, x, y, w, h }.

Wherein, x and y are the central coordinates of the rectangular marking frame, and w and h are the width and height of the rectangular marking frame.

After a plurality of sample images with labeled data are constructed, the sample images with labeled data can be used to train a deep learning-based target detection model (such as YOLOv 3), the specific training process is an iterative process, and in one iterative process, the specific training process specifically includes the following operations:

inputting the sample image into an initial target detection model, and detecting a carriage and a sucker in the sample image based on the target detection model to obtain a detection result; and comparing the detection result with the labeled data of the sample image, constructing a loss function based on the comparison result, and adjusting the network parameters of the target detection model based on the loss function.

In the embodiment of the present application, the stop condition of the iterative training may be: model convergence, preset upper limit obtained by iteration times and the like, and the text is not specifically limited.

After the training of the target detection model is completed, the target detection can be sequentially performed on each video frame according to the time sequence of each video frame in the video to be detected.

Specifically, in the whole process of driving-unloading-driving-out of the target unloading vehicle, the video picture changes, namely: the position of the carriage of the target dump vehicle can be changed, and therefore the position is different according to the video pictures. Taking the example shown in fig. 3A that the discharging tool is fixed at a certain position, the detection results can be specifically classified into the following categories:

a: the video picture has a sucker and no carriage, and at the moment, the carriage detection result shows that: when the carriage is not detected, the detection result of the sucking disc (namely the detection result of the unloading tool) indicates that the sucking disc is detected, and the position of the sucking disc in the video frame is identified through a sucking disc detection frame;

b: the video picture is provided with a sucker and a carriage, and at the moment, the carriage detection result shows that: and detecting the carriage, identifying the position of the carriage in the video frame through the carriage detection frame, indicating that the sucker is detected according to the sucker detection result, and identifying the position of the sucker in the video frame through the sucker detection frame.

As shown in fig. 3B, which is a schematic view of a video frame of a second type of video frame listed in the embodiment of the present application, that is, an example of the above listed B-type detection result, a car and a suction cup are both present in the video frame, and are respectively marked by rectangular detection frames.

The specific representation mode of the detection frame is the same as the labeled data, namely x and y are the central coordinates of the rectangular detection frame, and w and h are the width and the height of the rectangular detection frame.

S22: and the server determines a target video frame from the first video frames according to the determined first overlapping degrees, and detects the unloading process of the target unloading vehicle based on the target video frame.

In the embodiment of the present application, the target video frame is used for detecting the unloading process of the target unloading vehicle, so in step S21, the overlapping degree is mainly calculated according to the detection result of the video frame after the compartment of the target unloading vehicle enters the unloading area.

For example, a complete video to be detected has 100 frames, the frames 1 to 15 show the process that the target unloading vehicle gradually enters the unloading area, the frames 16 to 25 show the process that the target unloading vehicle enters the unloading area and is stable, the frames 26 to 80 show the unloading process of the target unloading vehicle, and the frames 81 to 100 show the process that the target unloading vehicle exits.

Then, in step S21, the video frames after the 16 th frame can be all represented as the first video frame after the car enters the unloading area, so that the 16 th frame and the following video frames are mainly detected, and the video frames are screened according to the first overlapping degree between the car and the suction cup, so as to ensure that the suction cup in the target video frame does not block the picture inside the car, thereby facilitating the subsequent detection.

An alternative embodiment is: and determining a first overlapping degree of the discharging tool and the compartment in the corresponding video frame based on the discharging tool detection result and the compartment detection result respectively corresponding to each first video frame.

For example, the above-listed video to be detected, for the 16 th frame, the first overlapping degree of the discharging tool and the carriage corresponding to the video frame is: and calculating based on the detection result of the discharging tool in the video frame and the detection result of the carriage in the video frame, and analogizing other video frames.

Wherein, the carriage testing result includes: a car detection box for identifying a position of a car of the target dump vehicle in the video frame; the detection result of the discharging tool comprises: a discharge tool detection box for identifying a position of a discharge tool in a video frame.

On this basis, still taking frame 16 as an example, the first overlap degree can be expressed as an overlap degree between the discharge tool detection box in the video frame of frame 16 and the car detection box in the video frame of frame 16.

Optionally, in order to further improve the effectiveness of the captured image, the captured image is ensured as far as possible to be in the unloading process, the suction cup does not shield the picture in the carriage, the unloading state can be analyzed and the effective image is captured on the basis of target detection, after the unloading state of the carriage of the target unloading vehicle is determined to enter the working state, the corresponding first overlapping degree is calculated, the target video frame is screened according to the first overlapping degree to eliminate the influence of the carriage in the stable stopping stage, and the unloading is not started in the stable stopping process of the carriage actually.

Fig. 4 is a schematic diagram illustrating a calculation method of the overlapping degree in the embodiment of the present application. That is, for two objects, the areas of the two objects are cross-compared as the degree of overlap between the two objects.

Taking the first degree of overlap as an example, the first degree of overlap is the ratio of the area S1 to the area S2, where S1 represents the area of the overlapping portion of the car and the suction cup, and S2 represents the area of the object formed by the car and the suction cup.

In the embodiment of the application, in order to clearly analyze the unloading state of the vehicle, the whole process from entering to unloading is accurately judged, and three carriage unloading states are defined for the target unloading carriage.

An alternative embodiment is: defining the unloading state of the carriage as follows: UNKNOW, START, WORKING. Wherein unknown state, namely initial state; START is in the START state, and represents the preparation stage before unloading or the stage after unloading; WORKING is the WORKING state and represents the unloading stage.

It should be noted that the above listed definitions of several car unloading states are only examples, and actually, other states may be defined to distinguish whether the car is unloaded, and this document is not limited specifically.

Under the condition of combining the unloading states of the carriages, the carriage and the sucker can be simultaneously detected by using a target detection method, the unloading state of the carriage is analyzed based on a detection result, and the images are captured at a proper time by combining the unloading state of the carriage and the position of the sucker, so that the captured images are effective images which are not shielded by an unloading tool.

Referring to fig. 5A, which is a schematic flowchart of another vehicle image recognition method in the embodiment of the present application, the method specifically includes the following steps S51-S55:

step S51: constructing compartment and sucker marking data, and training a target detection model;

the specific implementation of this step can be found in the above embodiments, and the repetition is not described again.

Step S52: configuring a quadrilateral unloading area and a preset time interval;

in the embodiment of the present application, the configured quadrilateral discharging area may be defined as ROI, and the preset time interval is defined as t.

Step S53: detecting the positions of a carriage and a sucker in the video by using a target detection model;

step S54: analyzing the unloading state of the carriage based on the detection result, and capturing effective images based on a preset time interval;

step S55: and the unloading process is supervised through the captured effective image.

In the embodiment of the application, the number of the supervision judgment images is reduced by setting the preset time interval to effectively acquire the images of each layer in the carriage.

The following mainly describes steps S54 and S55:

fig. 5B is a state transition diagram of a car unloading state in the embodiment of the present application.

Firstly, setting an initial carriage unloading state as UNKNOW;

and further, sequentially carrying out target detection on each video frame according to the time sequence of each video frame in the video to be detected. In this process, when a car is detected in the screen, the car discharge state of the target discharge vehicle jumps from un low to START.

When the carriage unloading state of the target unloading vehicle jumps from unknown to START, the preparation stage before the target unloading vehicle enters unloading is indicated, and in the process, the target unloading vehicle gradually enters the unloading area and stops stably.

At this stage, an alternative embodiment is to determine that the compartment of the target dump vehicle enters the dump zone by:

in the process of sequentially carrying out target detection according to the time sequence of each video frame in the video to be detected, if a second video frame is detected to appear in the video to be detected, determining that a compartment of a target unloading vehicle enters an unloading area; wherein the second video frame is: the corresponding video frame with the second overlapping degree of the carriage and the unloading area larger than the first threshold value; the second degree of overlap corresponding to each video frame is: an overlap between the car and the discharge area determined based on the respective car detection results.

Specifically, when the unloading state of the carriage is determined to jump from UNKNOW to START, the carriage detection box R in the current picture is recorded_tThe carriage detection frame R_tCalculating the degree of overlap IOU with a preconfigured region of discharge ROI₁(also called second degree of overlap) (Intersection over Union, representing the degree of overlap of two regions):

equation 1

In equation 1, IOU₁The value is a value between 0 and 1, which indicates the overlapping degree of the two regions, and the closer to 1, the higher the overlapping degree. When IOU is used₁>Thresh₁(typically set to 0.7) indicates that the car has mostly entered the dump region.

For example, still taking the above-listed video to be detected as an example, the video includes 100 frames of video frames, then the target detection is performed on each frame of video frame according to the time sequence, and the unloading state of the carriage is analyzed according to the detection result.

Assuming that a carriage is detected in the frame in the 5 th frame video frame, the carriage unloading state is UNKNOW during the 1 st frame to the 4 th frame video frame, the carriage unloading state jumps from UNKNOW to START in the 5 th frame video frame, and a carriage detection box R in the 5 th frame video frame is recorded_tCalculating IOU based on the above equation 1₁And the calculation result and Thresh are combined₁Make a comparison, assume the IOU₁＜Thresh₁Then continue recording the car detection box R in the 6 th frame video frame_tCalculating IOU based on the above equation 1₁And the calculation result and Thresh are combined₁The comparison … is made and the process is repeated, assuming that when the 16 th frame video frame is detected, the car detection box R in the video frame is_tSecond degree of overlap IOU with region of discharge ROI₁Greater than Thresh₁Showing the carAnd enters a discharge area.

And after the carriage enters the unloading area, a certain time is reserved to enable the target unloading vehicle to be stable, so that the carriage enters a working state, and after the carriage enters the working state, the first overlapping degree is calculated to capture the target video frame.

At this stage, an alternative implementation is:

after a compartment of a target unloading vehicle enters an unloading area, if N continuous frames of first video frames with first characteristics are detected, determining that the compartment unloading state of the target unloading vehicle enters a working state, wherein N is a positive integer greater than 1; wherein, in the first video frames with the first characteristics of the continuous N frames: the third degree of overlap between each video frame and the car of the adjacent previous frame video frame is greater than the second threshold.

In this way, after the carriage unloading state of the target unloading vehicle enters the working state, the first overlapping degree of the unloading tool and the carriage in the corresponding video frame can be determined based on the average carriage detection result of the carriage of the target unloading vehicle and the unloading tool detection result corresponding to each first video frame.

In the mode, the situation that the target video frame captured at the carriage stable stage does not substantially influence the unloading process is considered, based on the process, the first video frame before the unloading state of the carriage of the target unloading vehicle enters the working state can be excluded, only the first video frame when the unloading state of the carriage of the target unloading vehicle enters the working state is calculated, the target effective frame is screened based on the first video frame, and the detection efficiency is improved.

Optionally, the average car detection result includes: based on the car detection box in the first video frame with the first characteristic of the continuous N frames, the average detection box for the car is determined.

In this way, for each first video frame after the carriage unloading state enters the working state, the following operations are respectively performed:

and taking the overlapping degree between the discharging tool detection frame corresponding to the first video frame and the average detection frame as the first overlapping degree corresponding to the first video frame.

Also taking the above-listed video to be detected as an example, the current frame compartment detection frame R is detected starting from the 16 th frame_tAnd the carriage detection frame R of the previous frame_t-1Calculating the third overlap IOU₂And combining the third overlapping degree IOU₂And a second threshold value Thresh₂Comparing; when IOU is used₂>Thresh₂(generally set to 0.85), this indicates that the car has been stabilized in the dump region, the frame number frame is incremented by 1, and the car detection frame R in the 17 th frame is calculated_tAnd the car detection frame R in frame 16_t-1IOU of third degree of overlap therebetween₂…, repeating the above steps, when the frame reaches the set threshold value Count₁(i.e., N), i.e., when the first video frame of the consecutive N frames having the first characteristic is detected, indicating that the vehicle has stopped, the car discharge status jumps from START to WORKING, and the average detection box avg _ rect for the car, which is used for the subsequent calculation of the first overlap, is recorded, which can be determined based on the car detection boxes in the first video frame of the consecutive N frames having the first characteristic.

For example, Count₁Set to 10, indicates that there is no motion of the car for 10 consecutive frames in the dump region. The average detection frame is: and averaging the car detection frames in the 10 th frames from the 16 th frame to the 25 th frame, wherein the 16 th frame to the 25 th frame are the first video frames with the first characteristic. Assume that the car detection box in frame 16 is R₁₆={x₁₆，y₁₆，w₁₆，h₁₆In the 17 th frame, the carriage detection frame is R₁₇={x₁₇，y₁₇，w₁₇，h₁₇…, the carriage detection frame in the 25 th frame is R₂₅={x₂₅，y₂₅，w₂₅，h₂₅The average value avg _ rect of the 10-frame car detection blocks can be expressed as { (x)₁₆+x₁₇+…+x₂₅)/10，(y₁₆+y₁₇+…+y₂₅)/10，(w₁₆+w₁₇+…+w₂₅)/10，(h₁₆+h₁₇+…+h₂₅)/10}。

It should be noted that, in the detection process after the car of the target dump vehicle enters the dump zone, if no car is detected, the state jumps back to unknown.

Optionally, when the first overlap degree is calculated by combining the average detection frame, the specific method is as follows:

and regarding a video frame after the unloading state of the carriage enters the working state, taking the overlapping degree between the unloading tool detection frame in the video frame and the average detection frame as a first overlapping degree corresponding to the video frame.

Still taking the above-listed video to be detected as an example, starting from frame 26, the car is in WORKING state, so that starting from frame 26, the first overlapping degree IOU between the suction cup detection frame corresponding to the suction cup in each video frame and the average detection frame avg _ rect of the car is calculated₃。

Taking the third threshold thresh3=0 as an example, when the IOU is in a state of being started₃When the value is 0, it means that the two are not overlapped, i.e. the suction cup does not block the picture inside the car, the corresponding video frame can be used as the target video frame, and is buffered in valid _ frame.

Considering that discharging is a progressive process, scrap steel needs to be taken out of a carriage little by little, so in order to realize layered grabbing of carriage images and detect different scrap steel, the embodiment of the application sets a preset time interval for effectively acquiring images of each layer in the carriage and reducing the number of supervision judgment images.

Fig. 6A, 6B and 6C are schematic video images of three different video frames in the embodiment of the present application, and the several figures show that as the discharging process proceeds, more and more steel scraps are sucked out of the car, and the steel scraps in the car are reduced little by little, like the steel scraps in the car are reduced layer by layer in height.

An alternative embodiment is: determining a candidate video frame set corresponding to each preset time interval; the candidate video frame set comprises video frames of which the first overlapping degree is smaller than a third threshold value in first video frames detected in a corresponding preset time interval; and determining the candidate video frame with the latest acquisition time in each candidate video frame set as the target video frame.

That is, the video frames with the corresponding first overlapping degree smaller than the third threshold (for example, thresh3= 0) are taken as candidate video frames, and the candidate video frames belonging to the same preset time interval form a candidate video frame set; further, the candidate video frame of the last frame detected in each preset time interval is used as the target video frame according to the time sequence of the video frames. That is, in the case where the preset time interval is set, one candidate video frame may be screened out at intervals (preset time intervals) as a target video frame for subsequent detection.

For example, in the first preset time interval, the detected candidate video frames include: at frame 27, frame 28 and frame 29, when the preset time interval t (generally set to 30s) is met from frame 26, outputting frame 29 as a target video frame; in a second predetermined time interval, the detected candidate video frames have: and the 37 th frame, the 38 th frame and the 39 th frame are output as the target video frame … when the preset time interval is met again, and the like.

It should be noted that, the above is for a simple example, so that the example that the video to be detected includes 100 video frames is taken as an example for illustration, in an actual process, there may be more video frames, and accordingly, the preset time interval may be longer, for example, may be set to 30 s. In the practical application process, the flexible setting is performed according to the practical situation, and the text is not specifically limited.

In the embodiment of the application, after the unloading is finished, the target unloading vehicle can move out of the unloading area, and in this case, the unloading state of the carriage can be switched from the working state to the starting state.

In an optional implementation manner, in the target detection process, if a first video frame with a second characteristic is detected to appear in the video to be detected, the target detection is stopped; wherein the first video frame with the second characteristic is: and the fourth overlapping degree of the corresponding compartment detection frame and the average detection frame is less than the fourth threshold value.

Vision to be examined as also listed aboveFor example, suppose that unloading is completed in 26-80 th frames, and then the target unloading vehicle is to gradually exit the unloading area, in the process, for example, the fourth overlapping degree IOU of the car detection frame and the average detection frame avg _ rect in the video frame is detected in 90 th frame₄<0.7 (fourth threshold Thresh)₄) When the car starts moving, the unloading state of the car jumps from WORKING to STARTING, and the 90 th frame is the first video frame with the second characteristic.

Fig. 6D is a schematic view of a video frame of a sixth video frame in this embodiment. The video picture represents a frame of video picture in the process that the target unloading vehicle is driven out of the unloading area. The shaded portion is a discharge area as exemplified in the embodiments of the present application. It is clear that the degree of overlap of the dump region with the car in the video frame is small, less than the fourth threshold.

Optionally, after the output target video frame valid _ frame is transmitted back to the control room, quality inspection personnel can supervise the unloading process through the target video frame, which is specifically divided into the following two cases:

in the first situation, whether the materials are doped with other substances or not is determined by detecting the materials in the carriage in the target video frame.

If in the steel plant, quality control personnel can remotely monitor whether impurities and dangerous goods exist in the current compartment through the target video frame.

And secondly, detecting the materials in the carriage in the target video frame to determine the quality of the materials.

For example, in a steel plant, quality testing personnel can remotely observe the quality of scrap steel through the target video frame.

Fig. 7 is a schematic flowchart illustrating a vehicle image capturing method according to an embodiment of the present disclosure. Taking a server as an execution subject as an example, the specific implementation flow of the method is as follows:

step S701: the method comprises the steps that a server obtains a video to be detected for a target unloading vehicle, and the unloading state of a compartment of the target unloading vehicle is set to be an unknown state;

step S702: the server carries out target detection on the current video frame of the video to be detected according to the video frame sequence;

step S703: the server judges whether a carriage exists in the current video frame picture according to the detection result, if so, the step S704 is executed, otherwise, the step S702 is returned;

step S704: the server switches the carriage unloading state of the target unloading vehicle into a starting state;

step S705: the server determines a second overlapping degree of a carriage detection frame of the current video frame and a preset unloading area;

step S706: the server judges whether the second overlapping degree is larger than the first threshold value, if so, the step S707 is executed, otherwise, the step S705 is returned;

step S707: the server determines that the target unloading vehicle enters an unloading area;

step S708: the server calculates a third overlapping degree between the carriage detection frame of the current frame and the carriage detection frame of the previous frame;

step S709: the server judges whether the third overlapping degree is larger than a second threshold value, if so, the step S710 is executed; otherwise, returning to the step S708;

step S710: the server counts the frame number frame plus 1 (the initial value is 0), and judges whether the frame reaches the preset value, if yes, the step S711 is executed, otherwise, the step S708 is returned;

step S711: the server switches the carriage unloading state of the target unloading vehicle into a working state and determines an average detection frame of the carriage;

step S712: the server calculates a first overlapping degree of a sucker in a current video frame and a carriage average detection frame;

step S713: the server judges whether the first overlapping degree is smaller than a third threshold value, if so, the step S714 is executed, otherwise, the step S712 is returned to;

step S714: the server determines that the video frame is a target video frame and caches the target video frame of the frame to valid _ frame;

step S715: the server judges whether the current time meets a preset time interval, if so, the step S716 is executed, otherwise, the step S712 is returned to;

step S716: the server outputs a latest cached target video frame in the valid _ frame;

step S717: the server detects the unloading process of the target unloading vehicle based on the output target video frame;

step S718: and when detecting that the fourth overlapping degree of the current carriage detection frame and the average detection frame is smaller than a fourth threshold value, the server switches the carriage unloading state from the working state to the starting state and stops target detection.

Alternatively, in steps S704-S711, if no car is detected, the car discharge state jumps back to the unknown state, not shown in the figure.

In the embodiment, the unloading state of the carriage is judged by analyzing the position change conditions of the unloading area, the carriage and the sucker, and the target video frame is output within the specified preset time interval, so that the image of each layer in the carriage is effectively acquired, the number of the supervision judgment images is reduced, and the detection efficiency is improved.

Based on the same inventive concept, the embodiment of the application also provides a vehicle image recognition device. As shown in fig. 8, which is a schematic structural diagram of a vehicle image recognition device 800, the vehicle image recognition device may include:

the analysis unit 801 is configured to acquire a video to be detected acquired for a target unloading vehicle, and perform target detection on a carriage of the target unloading vehicle included in each video frame in the video to be detected; and

determining a first degree of overlap of the discharging tool and the car detected from each first video frame if it is determined that the car enters the discharging area based on a car detection result of the car of the target discharging vehicle; the first video frame comprises a video frame after a carriage enters a discharging area in a video to be detected; the unloading tool is used for unloading materials in a compartment of the target unloading vehicle;

the detecting unit 802 is configured to determine a target video frame from the first video frames according to the determined first overlapping degrees, and detect a discharging process of the target discharging vehicle based on the target video frame.

Optionally, the analysis unit 801 is further configured to:

the second video frame is a video frame with a corresponding second overlapping degree larger than a first threshold, and the corresponding second overlapping degree of each video frame is as follows: an overlap between the car and the discharge area determined based on the respective car detection results.

Optionally, the analysis unit 801 is further configured to:

carrying out target detection on the unloading tools contained in each video frame in the video to be detected to obtain the detection result of the unloading tool of each video frame;

the analysis unit 801 is specifically configured to:

determining a first overlapping degree of the discharging tool and the compartment in the corresponding video frame based on the discharging tool detection result and the compartment detection result respectively corresponding to each first video frame; or

After the compartment unloading state of the target unloading vehicle enters a working state, determining a first overlapping degree of the unloading tool and the compartment in the corresponding video frame based on an average compartment detection result of the compartment of the target unloading vehicle and the unloading tool detection result corresponding to each first video frame.

Optionally, the car detection result includes: a car detection box for identifying a position of a car of the target dump vehicle in the video frame; the detection result of the discharging tool comprises: a discharge tool detection box for identifying the position of the discharge tool in the video frame;

the analysis unit 801 is specifically configured to:

for each first video frame, the following operations are performed:

and taking the overlapping degree between the unloading tool detection frame and the carriage detection frame corresponding to the first video frame as the first overlapping degree of the first video frame.

Optionally, the analysis unit 801 is further configured to determine that the car unloading state enters the working state by:

after a compartment of a target unloading vehicle enters an unloading area, if N continuous frames of first video frames with first characteristics are detected, determining that the unloading state of the compartment of the target unloading vehicle enters a working state, wherein N is a positive integer greater than 1;

wherein, in the first video frames with the first characteristic of the continuous N frames: the third degree of overlap between each video frame and the car of the adjacent previous frame video frame is greater than the second threshold.

Optionally, the car detection result includes: a car detection box for identifying a position of a car of the target dump vehicle in the video frame; the detection result of the discharging tool comprises: the discharging tool detection frame is used for identifying the position of the discharging tool in the video frame;

the average car detection result includes: based on the carriage detection frame in the first video frame with the first characteristic of the continuous N frames, determining an average detection frame for the carriage;

the analysis unit 801 is specifically configured to:

for each first video frame after the unloading state of the carriage enters the working state, respectively executing the following operations:

Optionally, the detecting unit 802 is specifically configured to:

determining a candidate video frame set corresponding to each preset time interval; the candidate video frame set comprises video frames of which the first overlapping degree is smaller than a third threshold value in the first video frames detected in the corresponding preset time interval;

Optionally, the apparatus further comprises:

an ending unit 803, configured to, after the carriage unloading state enters the working state, perform target detection in sequence according to the time sequence of each video frame in the video to be detected, and stop target detection if a first video frame with a second characteristic appears in the video to be detected;

wherein the first video frame with the second characteristic is: and the fourth overlapping degree of the corresponding compartment detection frame and the average detection frame is less than the first video frame of the fourth threshold value.

Optionally, the detecting unit 802 is specifically configured to:

determining whether the materials are doped with other substances or not by detecting the materials in the carriage in the target video frame; or

The quality of the material is determined by detecting the material in the carriage in the target video frame.

The method comprises the steps of detecting a carriage of a target unloading vehicle contained in each video frame through a target detection method, and obtaining a carriage detection result of each video frame; and under the condition that the compartment of the target unloading vehicle enters the unloading area is determined based on the compartment detection result, determining the first overlapping degree of the unloading tool and the compartment detected from each first video frame, capturing the video image from the video frame after the compartment enters the unloading area is determined from the video to be detected by taking the overlapping degree as reference, and ensuring that the captured image is an effective image which is not shielded by the unloading tool.

For convenience of description, the above parts are separately described as modules (or units) according to functional division. Of course, the functionality of the various modules (or units) may be implemented in the same one or more pieces of software or hardware when implementing the present application.

Having described the vehicle image recognition method and apparatus according to an exemplary embodiment of the present application, an electronic device according to another exemplary embodiment of the present application will be described next.

As will be appreciated by one skilled in the art, aspects of the present application may be embodied as a system, method or program product. Accordingly, various aspects of the present application may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.

The electronic equipment is based on the same inventive concept as the method embodiment, and the embodiment of the application also provides the electronic equipment. In one embodiment, the electronic device may be a server, such as server 120 shown in FIG. 1. In this embodiment, the electronic device may be configured as shown in fig. 9, and include a memory 901, a communication module 903, and one or more processors 902.

A memory 901 for storing computer programs executed by the processor 902. The memory 901 may mainly include a program storage area and a data storage area, where the program storage area may store an operating system, a program required for running an instant messaging function, and the like; the storage data area can store various instant messaging information, operation instruction sets and the like.

Memory 901 may be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory 901 may also be a non-volatile memory (non-volatile memory), such as a read-only memory (rom), a flash memory (flash memory), a hard disk (HDD) or a solid-state drive (SSD); or the memory 901 is any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and that can be accessed by a computer, but is not limited to such. The memory 901 may be a combination of the above memories.

The processor 902 may include one or more Central Processing Units (CPUs), a digital processing unit, and the like. A processor 902 for implementing the above-described vehicle image recognition method when calling the computer program stored in the memory 901.

The communication module 903 is used for communicating with terminal equipment and other servers.

The embodiment of the present application does not limit the specific connection medium among the memory 901, the communication module 903, and the processor 902. In the embodiment of the present application, the memory 901 and the processor 902 are connected through the bus 904 in fig. 9, the bus 904 is depicted by a thick line in fig. 9, and the connection manner between other components is merely illustrative and is not limited. The bus 904 may be divided into an address bus, a data bus, a control bus, and the like. For ease of description, only one thick line is depicted in fig. 9, but not only one bus or one type of bus.

The memory 901 stores a computer storage medium, in which computer-executable instructions are stored, and the computer-executable instructions are used for implementing the vehicle image identification method according to the embodiment of the present application. The processor 902 is configured to execute the vehicle image recognition method described above, as shown in fig. 2.

In another embodiment, the electronic device may also be other electronic devices, such as the terminal device 110 shown in fig. 1. In this embodiment, the structure of the electronic device may be as shown in fig. 10, including: a communications component 1010, a memory 1020, a display unit 1030, a camera 1040, a sensor 1050, audio circuitry 1060, a bluetooth module 1070, a processor 1080, and the like.

The communication component 1010 is configured to communicate with a server. In some embodiments, a Wireless Fidelity (WiFi) module may be included, the WiFi module being a short-range Wireless transmission technology, through which the electronic device may help the user to transmit and receive information.

Memory 1020 may be used to store software programs and data. Processor 1080 performs various functions of terminal device 110 and data processing by executing software programs or data stored in memory 1020. The memory 1020 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. The memory 1020 stores an operating system that enables the terminal device 110 to operate. The memory 1020 may store an operating system and various application programs, and may also store a computer program for executing the vehicle image recognition method according to the embodiment of the present application.

The display unit 1030 may also be used to display information input by the user or information provided to the user and a Graphical User Interface (GUI) of various menus of the terminal device 110. Specifically, the display unit 1030 may include a display screen 1032 disposed on the front surface of the terminal device 110. The display 1032 may be configured in the form of a liquid crystal display, a light emitting diode, or the like. The display unit 1030 can be used for displaying the video to be detected, the car detection result, the discharging tool detection result, and the like in the embodiment of the present application.

The display unit 1030 may also be configured to receive input numeric or character information and generate signal input related to user settings and function control of the terminal device 110, and specifically, the display unit 1030 may include a touch screen 1031 disposed on the front surface of the terminal device 110 and configured to collect touch operations by a user thereon or nearby, such as clicking a button, dragging a scroll box, and the like.

The touch screen 1031 may cover the display screen 1032, or the touch screen 1031 and the display screen 1032 may be integrated to implement the input and output functions of the terminal device 110, and the integrated function may be referred to as a touch display screen for short. In the present application, the display unit 1030 may display the application program and the corresponding operation steps.

The camera 1040 may be used to capture still images, and the user may post comments on the images captured by the camera 1040 through the application. The number of the cameras 1040 may be one or plural. The object generates an optical image through the lens and projects the optical image to the photosensitive element. The photosensitive element may be a Charge Coupled Device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The light sensing elements convert the light signals into electrical signals which are then passed to a processor 1080 for conversion into digital image signals.

The terminal device may further comprise at least one sensor 1050, such as an acceleration sensor 1051, a distance sensor 1052, a fingerprint sensor 1053, a temperature sensor 1054. The terminal device may also be configured with other sensors such as a gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, motion sensor, and the like.

Audio circuitry 1060, speaker 1061, microphone 1062 may provide an audio interface between a user and terminal device 110. The audio circuit 1060 may transmit the electrical signal converted from the received audio data to the speaker 1061, and convert the electrical signal into a sound signal by the speaker 1061 and output the sound signal. Terminal device 110 may also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1062 converts the collected sound signals into electrical signals, which are received by the audio circuit 1060 and converted into audio data, which is then output to the communication module 1010 for transmission to, for example, another terminal device 110, or to the memory 1020 for further processing.

The bluetooth module 1070 is used for exchanging information with other bluetooth devices having a bluetooth module through a bluetooth protocol. For example, the terminal device may establish a bluetooth connection with a wearable electronic device (e.g., a smart watch) also equipped with a bluetooth module through the bluetooth module 1070, so as to perform data interaction.

The processor 1080 is a control center of the terminal device, connects various parts of the entire terminal device using various interfaces and lines, and performs various functions of the terminal device and processes data by running or executing software programs stored in the memory 1020 and calling data stored in the memory 1020. In some embodiments, processor 1080 may include one or more processing units; processor 1080 may also integrate an application processor, which primarily handles operating systems, user interfaces, applications, etc., and a baseband processor, which primarily handles wireless communications. It is to be appreciated that the baseband processor described above may not be integrated into processor 1080. In the present application, the processor 1080 may run an operating system, an application program, a user interface display, a touch response, and the vehicle image recognition method according to the embodiment of the present application. Further, processor 1080 is coupled to a display unit 1030.

In some possible embodiments, the various aspects of the vehicle image recognition method provided by the present application may also be implemented in the form of a program product including a computer program for causing an electronic device to perform the steps in the vehicle image recognition method according to various exemplary embodiments of the present application described above in this specification when the program product is run on the electronic device, for example, the electronic device may perform the steps as shown in fig. 2.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The program product of embodiments of the present application may employ a portable compact disc read only memory (CD-ROM) and include a computer program, and may be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with a command execution system, apparatus, or device.

A readable signal medium may include a propagated data signal with a readable computer program embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with a command execution system, apparatus, or device.

The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer programs for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer program may execute entirely on the user computing device, partly on the user computing device, as a stand-alone software package, partly on the user computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. Indeed, the features and functions of two or more units described above may be embodied in one unit, according to embodiments of the application. Conversely, the features and functions of one unit described above may be further divided into embodiments by a plurality of units.

Further, while the operations of the methods of the present application are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having a computer-usable computer program embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the present application.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims

1. A vehicle image recognition method, characterized in that the method comprises:

acquiring a video to be detected, which is acquired aiming at a target unloading vehicle, and carrying out target detection on a carriage of the target unloading vehicle contained in each video frame in the video to be detected; and

determining a first degree of overlap of the discharging tool and the car detected from each first video frame if it is determined that the car enters the discharging area based on a car detection result of the car of the target discharging vehicle; the first video frame comprises a video frame after the carriage enters the unloading area in the video to be detected; the unloading tool is used for unloading materials in a compartment of the target unloading vehicle;

and determining a target video frame from the first video frames according to the determined first overlapping degrees, and detecting the unloading process of the target unloading vehicle based on the target video frame, wherein the target video frame is a video frame of which the corresponding first overlapping degree is smaller than a third threshold value in the first video frames.

2. The method of claim 1, wherein determining that the car enters the discharge region based on car detection results for the car of the target discharge vehicle comprises:

the second video frame is a video frame with a corresponding second degree of overlap greater than a first threshold, and the second degree of overlap corresponding to each video frame is: an overlap between the car and the discharge area determined based on the respective car detection results.

3. The method of claim 1, wherein the step of obtaining a video to be detected captured for a target discharge vehicle further comprises:

the determining a first degree of overlap of the discharging tool and the car detected from each first video frame comprises:

4. The method of claim 3, wherein the car detection result comprises: a car detection box for identifying a location of a car of the target dump vehicle in a video frame; the discharging tool detection result comprises: a discharge tool detection box for identifying the position of the discharge tool in a video frame;

the determining the first overlapping degree of the discharging tool and the compartment in the corresponding video frame based on the discharging tool detection result and the compartment detection result respectively corresponding to each first video frame comprises:

for each first video frame, the following operations are performed:

5. The method of claim 3, wherein the car dump state is determined to enter the active state by:

after the compartment of the target unloading vehicle enters an unloading area, if N continuous frames of first video frames with first characteristics are detected, determining that the compartment unloading state of the target unloading vehicle enters a working state, wherein N is a positive integer greater than 1;

6. The method of claim 5, wherein the car detection result comprises: a car detection box for identifying a location of a car of the target dump vehicle in a video frame; the discharging tool detection result comprises: a discharge tool detection box for identifying the position of the discharge tool in a video frame; the average car detection result includes: based on the car detection box in the first video frame of which the N continuous frames have the first characteristic, determining an average detection box for the car;

after the carriage unloading state of the target unloading vehicle enters a working state, determining a first overlapping degree of the unloading tool and the carriage in the corresponding video frame based on an average carriage detection result of the carriage of the target unloading vehicle and the unloading tool detection result corresponding to each first video frame respectively, including:

7. The method of claim 1, wherein said determining a target video frame from said first video frames based on said determined first respective degrees of overlap comprises:

8. The method of claim 6, wherein the method further comprises:

after the carriage unloading state enters a working state, sequentially performing target detection according to the time sequence of each video frame in the video to be detected, and stopping target detection if a first video frame with a second characteristic appears in the video to be detected;

9. The method according to any one of claims 1 to 8, wherein the detecting of the unloading process of the target unloading vehicle based on the target video frame comprises:

10. A vehicle image recognition apparatus characterized by comprising:

the analysis unit is used for acquiring a video to be detected, which is acquired aiming at a target unloading vehicle, and carrying out target detection on a carriage of the target unloading vehicle contained in each video frame in the video to be detected; and

and the detection unit is used for determining a target video frame from the first video frames according to the determined first overlapping degrees and detecting the unloading process of the target unloading vehicle based on the target video frame, wherein the target video frame is a video frame of which the corresponding first overlapping degree is smaller than a third threshold value in the first video frames.

11. An electronic device, comprising a processor and a memory, wherein the memory stores a computer program which, when executed by the processor, causes the processor to carry out the steps of the method of any of claims 1 to 9.

12. A computer-readable storage medium, characterized in that it comprises a computer program for causing an electronic device to carry out the steps of the method according to any one of claims 1 to 9, when said computer program is run on said electronic device.