CN114092842A

CN114092842A - Package abnormity detection method, device, control equipment and storage medium

Info

Publication number: CN114092842A
Application number: CN202010631688.9A
Authority: CN
Inventors: 曾创
Original assignee: SF Technology Co Ltd
Current assignee: SF Technology Co Ltd
Priority date: 2020-07-03
Filing date: 2020-07-03
Publication date: 2022-02-25

Abstract

The application provides a parcel anomaly detection method, a device, control equipment and a storage medium, wherein the parcel anomaly detection method comprises the following steps: acquiring parcel delivery images of a plurality of delivery areas acquired by a plurality of camera devices, wherein the delivery areas are delivery areas of the plurality of delivery devices in a preset logistics area; respectively carrying out parcel detection on the parcel delivery images of the delivery areas to determine the position information and the state information of parcels in the delivery areas; generating abnormal state integrated information of the packages on the plurality of conveying devices based on the position information of the packages and the state information of the packages. The commodity circulation regional parcel abnormal overall detection system has the advantages that the visual images of a plurality of conveying regions can be collected to carry out abnormal overall detection on the parcel of one commodity circulation region, the problem that the parcel is not integral enough for single conveying device or single conveying region is avoided, the abnormal parcel of the whole commodity circulation region can be automatically detected, the abnormal state integrated information of the whole commodity circulation region is generated, the sorting efficiency is greatly improved, and the maintenance cost is reduced.

Description

Package abnormity detection method, device, control equipment and storage medium

Technical Field

The application relates to the technical field of parcel detection, in particular to a parcel anomaly detection method and device, control equipment and a storage medium.

Background

In recent years, machine vision algorithms are more and more widely applied in the field of logistics sorting, and the most basic characteristic of a machine vision system is to improve the flexibility and the automation degree of production. In some dangerous working environments which are not suitable for manual operation or occasions where manual vision is difficult to meet the requirements, machine vision is often used to replace the manual vision. Meanwhile, in the process of mass repetitive industrial production, the machine vision detection method can greatly improve the production efficiency and the automation degree.

At present, in the field of logistics sorting, most of the objects are sensed by means of photoelectric sensors. For regional delivery, a number of cells are typically arranged in a queue to form a triggering edge to sense the arrival of an object in the region. Especially for wide-area sorting, a large number of photoelectric sensors are needed to effectively sense that the objects reach the designated conveying or sorting area, the installation is complex, and the mode cannot sense the state information of the objects in the conveying process in real time.

Disclosure of Invention

The application provides a parcel anomaly detection method, a parcel anomaly detection device, control equipment and a storage medium, which are used for collecting visual images of a plurality of conveying areas to carry out parcel anomaly overall detection of a logistics area, avoiding the problem that the single conveying device or the single conveying area is not integrally detected, automatically detecting parcel anomaly of the whole logistics area, generating abnormal state integrated information of the whole logistics area, greatly improving sorting efficiency and reducing maintenance cost.

In one aspect, the present application provides a package anomaly detection method, including:

acquiring parcel conveying images of a plurality of conveying areas acquired by a plurality of camera devices, wherein the plurality of conveying areas are conveying areas of the plurality of conveying devices in a preset logistics area, the vision coverage areas of the plurality of camera devices cover the plurality of conveying areas, and each camera device corresponds to at least one conveying area in the plurality of conveying areas;

respectively carrying out parcel detection on the parcel delivery images of the delivery areas so as to determine position information and state information of parcels in the delivery areas;

generating abnormal state integrated information of the packages on the plurality of conveying devices based on the position information of the packages and the state information of the packages.

In some embodiments of the present application, each delivery area corresponds to a predetermined plurality of delivery grid cells, and prior to said capturing, by the plurality of cameras, package delivery images for a plurality of delivery areas of the plurality of delivery devices, the method further comprises:

and performing image visual field division and grid positioning identification on a plurality of conveying areas on the plurality of conveying devices through the plurality of camera devices to obtain grid units corresponding to each conveying area and visual field coverage of each camera device in the plurality of conveying areas.

In some embodiments of the present application, the performing, by the plurality of image capturing devices, grid positioning recognition and image view division on a plurality of conveying areas on the plurality of conveying devices to obtain a grid unit corresponding to each conveying area, and a view coverage of each image capturing device in the plurality of conveying areas includes:

after the plurality of camera devices are fixed in position according to a preset installation strategy, respectively sending control instructions to the plurality of camera devices, wherein the control instructions are used for instructing each camera device to adjust the camera angle of the camera device so as to determine the view coverage of each camera device in the plurality of conveying areas;

acquiring first registration images corresponding to the plurality of transmission areas respectively acquired by the plurality of camera devices;

respectively taking each transmission area as a target transmission area, and identifying grid boundary points of a target registration image corresponding to the target transmission area according to preset grid size information to determine the grid boundary points in the target transmission area;

and determining the grid unit corresponding to the target transmission area according to the grid boundary point in the target transmission area.

In some embodiments of the present application, the method further comprises:

acquiring an adjustment request for adjusting the grid size of a divided transmission area by a user, wherein the adjustment request comprises new grid size information;

acquiring second registration images corresponding to the plurality of transmission areas respectively acquired by the plurality of camera devices;

respectively identifying the grid boundary points of the plurality of transmission areas again according to the new grid size information, and determining the grid boundary points in the plurality of transmission areas;

and determining the grid units corresponding to the plurality of transmission areas according to the grid boundary points in the plurality of transmission areas.

In some embodiments of the present application, the performing package detection on the package delivery images of the delivery areas to determine the position information and the status information of the packages in the delivery areas comprises:

respectively taking the parcel delivery images of the delivery areas as target parcel images, carrying out parcel detection on the target parcel images, and determining the current parcel placement area in the target parcel images;

acquiring coordinate information corresponding to the current parcel placement area based on a pre-established coordinate system;

acquiring position information of the parcel in the corresponding delivery area in the target parcel image according to the coordinate information corresponding to the current parcel placement area;

and detecting the state of the packages in the target package image to obtain the state information of the packages in the target package image.

In some embodiments of the present application, the generating abnormal status information for a parcel in a plurality of delivery devices based on the location information for the parcel and the status information for the parcel comprises:

and respectively filling the position information of each parcel and the state information of each parcel in the position information and the state information of each parcel into a preset parcel abnormal state information table to obtain the abnormal state integrated information of the parcels in the plurality of conveying devices.

In some embodiments of the present application, after generating abnormal status information for a parcel in a plurality of delivery devices based on location information for the parcel and status information for the parcel, the method further comprises:

determining a transfer region where an abnormality exists based on the abnormality state integration information;

and sending prompt information to the user equipment corresponding to the abnormal transmission area, wherein the prompt information comprises the abnormal state integrated information.

In another aspect, the present application provides a package anomaly detection device, the device comprising:

the system comprises an image acquisition module, a display module and a display module, wherein the image acquisition module is used for acquiring parcel conveying images of a plurality of conveying areas acquired by a plurality of camera devices, the plurality of conveying areas are conveying areas of the plurality of conveying devices in a preset logistics area, the vision coverage areas of the plurality of camera devices cover the plurality of conveying areas, and each camera device corresponds to at least one conveying area in the plurality of conveying areas;

the parcel detection module is used for respectively carrying out parcel detection on the parcel delivery images of the delivery areas so as to determine the position information and the state information of parcels in the delivery areas;

and the information generating module is used for generating the abnormal state integrated information of the packages on the plurality of conveying devices based on the position information of the packages and the state information of the packages.

In some embodiments of the present application, each transmission region corresponds to a predetermined plurality of transmission grid cells, the apparatus further comprising a region dividing module:

the area division module is used for carrying out image visual field division and grid positioning identification on the plurality of conveying areas on the plurality of conveying devices through the plurality of camera devices before the plurality of camera devices acquire parcel conveying images of the plurality of conveying areas in the plurality of conveying devices, so that grid units corresponding to each conveying area and the visual field coverage range of each camera device in the plurality of conveying areas are obtained.

In some embodiments of the present application, the region dividing module is specifically configured to:

In some embodiments of the present application, the region dividing module is further configured to:

In some embodiments of the present application, the package detection module is specifically configured to:

In some embodiments of the present application, the information generating module is specifically configured to:

In some embodiments of the present application, the apparatus further comprises a prompting module configured to:

determining a delivery area where an abnormality exists based on the abnormality state integration information after generating abnormality state information of parcels in a plurality of delivery devices based on the position information of the parcels and the state information of the parcels;

In another aspect, the present application also provides a control apparatus, including:

one or more processors;

a memory; and

one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the package anomaly detection method.

In another aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is loaded by a processor to execute the steps in the package anomaly detection method.

This application is complicated at a large amount of photoelectric sensor installations of prior art, and can not perceive the status information basis in the object transfer process in real time, on the one hand cover a plurality of conveying regions on at least one conveyer through a plurality of camera devices, gather the visual image of a plurality of conveying regions and carry out the regional parcel of a commodity circulation unusual overall detection, avoid detecting single conveyer or single conveying region not holistic problem inadequately, on the other hand, through the parcel anomaly detection to a plurality of conveying regions, can the regional parcel of automated inspection whole commodity circulation unusual, generate the regional abnormal state integrated information of whole commodity circulation, make things convenient for managers to carry out whole the accuse to the state in the commodity circulation letter sorting process, guarantee the normal operating of object conveying letter sorting, greatly improved letter sorting efficiency, and maintenance cost is reduced.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic view of a scenario of a package anomaly detection system provided in an embodiment of the present application;

FIG. 2 is a view showing a plurality of cameras according to an embodiment of the present disclosure;

FIG. 3 is a schematic flow chart diagram illustrating one embodiment of a package anomaly detection method provided in embodiments of the present application;

FIG. 4 is a schematic flow chart diagram illustrating one embodiment of determining grid cells in a target delivery area as provided in an embodiment of the present application;

FIG. 5 is a schematic diagram of one embodiment of a scenario of grid cells in a delivery area provided in an embodiment of the present application;

FIG. 6 is a schematic flow chart diagram illustrating one embodiment of a re-gridding process provided in embodiments of the present application;

FIG. 7 is a flowchart illustrating an embodiment of step 302 provided in the embodiments of the present application

FIG. 8 is an abnormal status display diagram of package information and delivery areas provided in an embodiment of the present application;

FIG. 9 is a schematic diagram of an embodiment of a state table of a gridded transmission area provided in an embodiment of the present application;

FIG. 10 is a schematic structural diagram of one embodiment of a package anomaly detection device provided in embodiments of the present application;

fig. 11 is a schematic structural diagram of an embodiment of the control device provided in the embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

In the description of the present application, it is to be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like indicate orientations or positional relationships based on those shown in the drawings, and are used merely for convenience of description and for simplicity of description, and do not indicate or imply that the referenced device or element must have a particular orientation, be constructed in a particular orientation, and be operated, and thus should not be considered as limiting the present application. Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", may explicitly or implicitly include one or more of the described features. In the description of the present application, "a plurality" means two or more unless specifically limited otherwise.

In this application, the word "exemplary" is used to mean "serving as an example, instance, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. The following description is presented to enable any person skilled in the art to make and use the application. In the following description, details are set forth for the purpose of explanation. It will be apparent to one of ordinary skill in the art that the present application may be practiced without these specific details. In other instances, well-known structures and processes are not set forth in detail in order to avoid obscuring the description of the present application with unnecessary detail. Thus, the present application is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

Embodiments of the present application provide a package anomaly detection method, a package anomaly detection device, a server, and a storage medium, which are described in detail below.

Referring to fig. 1, fig. 1 is a schematic view of a scene of a package anomaly detection system according to an embodiment of the present application, where the package anomaly detection system may include a control device 100 and a plurality of image capturing apparatuses 200 connected to the control device via a network, and the control device 100 is integrated with a package anomaly detection apparatus, such as the control device in fig. 1.

In the embodiment of the present application, the control apparatus 100 is mainly configured to obtain parcel delivery images of multiple delivery areas acquired by multiple cameras, where the multiple delivery areas are delivery areas of multiple delivery devices in a preset logistics area, the vision coverage areas of the multiple cameras cover the multiple delivery areas, and each camera corresponds to at least one delivery area in the multiple delivery areas; respectively carrying out parcel detection on the parcel delivery images of the delivery areas so as to determine position information and state information of parcels in the delivery areas; generating abnormal state integrated information of the packages on the plurality of conveying devices based on the position information of the packages and the state information of the packages.

Among the plurality of image capturing devices, the image capturing device may be an industrial camera, such as a standard industrial camera, specifically, for example, a focal length is 5mm, a frame rate is 30FPS (Frames Per Second), pixels are 200 ten thousand pixels, and a field coverage can cover an area of 2mx2 m.

A plurality of cameras may be disposed corresponding to the at least one conveyor, one camera may correspond to a conveying area of one conveyor, and may also correspond to a plurality of conveying areas of a plurality of conveyors, as shown in fig. 2, a specific scene is set for the cameras, one camera corresponds to one conveying area of one conveyor, such as conveying area 1 corresponding to camera 1, and when the coverage of the camera's field of view is large enough, a plurality of conveying areas may be covered, for example, in fig. 2, when the coverage of the camera's 1 field of view is large enough, conveying area 1, conveying area 2, conveying area 3, conveying area 4, etc. may be covered.

It should be noted that, in the embodiment of the present application, the image capturing device needs to be set to be aligned with at least one conveying area of the conveying device, so as to ensure that the coverage area of the field of view of the image capturing device covers the at least one conveying area of the conveying device, and thus the image capturing device can play a practical role.

In addition, the area of the logistics area and the number of the cameras can be set in a direct ratio, namely the larger the area of the logistics area is, the more the number of the cameras can be set, so that the transmission areas of all the transmission devices in the logistics area can be covered by the cameras, and the transmission areas can be determined according to actual conditions. For example, assuming a logistics area of a site 35 meters long and 26 meters wide, a single camera view size can cover an area of 2mx2m, thus 30 cameras are used for view coverage.

In this embodiment, the control device 100 may be a device, or a cluster of multiple devices, for example, may be a control device, or may be a cluster including a control device and multiple industrial personal computers, for example, in a specific embodiment, one industrial personal computer may connect 4 image capturing devices, and is responsible for processing the trigger states of 4 visual field areas, when the number of image capturing devices in the multiple image capturing devices exceeds 4, the entire control device 100 may include 1 control device and multiple industrial personal computers, the 1 control device is connected to the multiple industrial personal computers, the multiple industrial personal computers may be connected to the control device through a Position Location Server (PLS), and the control device may be a control terminal.

Specifically, the control device 100 may be an independent server, or may be a server network or a server cluster composed of servers, for example, the control device 100 described in this embodiment includes, but is not limited to, a computer, a network host, a single network server, multiple network server sets, or a cloud server composed of multiple servers. Among them, the Cloud server is constituted by a large number of computers or web servers based on Cloud Computing (Cloud Computing).

It is to be understood that the control device 100 used in the embodiments of the present application may also be a device that includes both receiving and transmitting hardware, i.e., a device having receiving and transmitting hardware capable of performing two-way communication over a two-way communication link. Such a device may include: a cellular or other communication device having a single line display or a multi-line display or a cellular or other communication device without a multi-line display. The specific control device 100 may be a desktop terminal or a mobile terminal, and may be one of a mobile phone, a tablet computer, a notebook computer, and the like.

Those skilled in the art will understand that the application environment shown in fig. 1 is only one application scenario related to the present application, and does not constitute a limitation on the application scenario of the present application, and that other application environments may further include more or less control devices than those shown in fig. 1, for example, only 1 control device and 2 cameras are shown in fig. 1, and it is understood that the package anomaly detection system may further include one or more other cameras, which is not limited herein.

In addition, as shown in fig. 1, the package abnormality detection system may further include a memory 300 for storing package data, such as image data acquired by the camera and package abnormality state data.

It should be noted that the scenario diagram of the package anomaly detection system shown in fig. 1 is merely an example, and the package anomaly detection system and the scenario described in the embodiment of the present application are for more clearly illustrating the technical solution of the embodiment of the present application, and do not form a limitation on the technical solution provided in the embodiment of the present application.

As shown in fig. 3, which is a schematic flow chart of an embodiment of a package anomaly detection method in the embodiment of the present application, the package anomaly detection method includes the following steps 301 to 303:

301. and acquiring parcel delivery images of a plurality of delivery areas acquired by a plurality of camera devices.

The plurality of conveying areas are conveying areas of a plurality of conveying devices in a preset logistics area, each camera device has a view coverage, the view coverage of each camera device in the plurality of camera devices can be the same or different, and is not limited specifically, only the view coverage of the plurality of camera devices is required to cover the plurality of conveying areas, and each camera device corresponds to at least one conveying area in the plurality of conveying areas.

In addition, the logistics area described in the embodiment of the present application may be all or a part of the logistics branch, such as a certain sorting area of the logistics branch; the conveying device can be various conveying devices with plane conveying areas, such as a conveyor belt or a conveying plate; the conveying area may be the whole area of the conveying device (when the conveying area of the conveying device is small, for example, a conveying plate), or may be a part of the conveying device (for example, a part of a conveyor belt), and is not limited herein.

302. And respectively carrying out package detection on the package delivery images of the delivery areas so as to determine the position information and the state information of the packages in the delivery areas.

303. Generating abnormal state integrated information of the packages on the plurality of conveying devices based on the position information of the packages and the state information of the packages.

In the embodiment shown in fig. 3, each delivery area may correspond to a plurality of predetermined delivery grid cells, that is, each delivery area may correspond to a plurality of predetermined delivery grid cells, and therefore, in some embodiments of the present application, before the package delivery images of a plurality of delivery areas of a plurality of delivery devices are captured by a plurality of cameras, it is required to determine each delivery area to correspond to a plurality of delivery grid cells in advance, at this time, the package anomaly detection method may further include: and performing image visual field division and grid positioning identification on a plurality of conveying areas on the plurality of conveying devices through the plurality of camera devices to obtain grid units corresponding to each conveying area and visual field coverage of each camera device in the plurality of conveying areas.

Specifically, as shown in fig. 4, the grid positioning recognition and image view division are performed on a plurality of conveying areas on the plurality of conveying devices by the plurality of image capturing devices to obtain a grid unit corresponding to each conveying area and a view coverage of each image capturing device in the plurality of conveying areas, and the method may include the following steps 401 to 404:

401. and after the plurality of camera devices are fixed in position according to a preset installation strategy, respectively sending control instructions to the plurality of camera devices.

The control instruction is used for instructing each camera to adjust the camera angle thereof so as to determine the view coverage of each camera in the plurality of conveying areas.

402. And acquiring first registration images corresponding to the plurality of transmission areas respectively acquired by the plurality of camera devices.

Specifically, the first registration image is used for image gridding in the subsequent step, and identification and registration of the transmission region are performed.

403. And respectively taking each transmission area as a target transmission area, and carrying out grid boundary point identification on a target registration image corresponding to the target transmission area according to preset grid size information to determine grid boundary points in the target transmission area.

In order to complete gridding of the image, specifically, grid boundary point identification may be performed on the target registration image corresponding to the target transmission region according to preset grid size information, so as to determine the grid boundary points in the target transmission region. The grid size information is size information of a subsequent divided grid unit, and the grid size information may be determined according to a size of a package delivered by a current delivery device, for example, the grid size information is 20cm by 20cm, and the like, which is not limited herein.

It should be noted that, because the sizes of the packages transmitted by different transmission devices may be different, the grid size information of different transmission devices may be different, that is, the preset grid size information may include multiple kinds of grid size information, and the multiple kinds of grid size information respectively correspond to different transmission devices of the multiple transmission devices, for example, the multiple transmission devices include a transmission device a and a transmission device B, the transmission device a transmits a large package, and the transmission device B transmits a small package, the preset grid size information includes grid size information 1(20cm × 20cm), grid size information 2(10cm × 10cm), the transmission device a corresponds to the grid size information 1, and the transmission device B corresponds to the grid size information 2, etc.

404. And determining the grid unit corresponding to the target transmission area according to the grid boundary point in the target transmission area.

Since the mesh boundary points in the target transmission region have been determined in step 403, the mesh cells corresponding to the target transmission region may be directly formed, for example, 2x2 or 3x3 or other free mesh cells may be formed, based on the plurality of mesh boundary points.

As shown in fig. 5, in the object conveying process, the image pickup device and the conveying area may be arranged according to actual conditions, and the mesh view of 2x2 is used for sorting small pieces and the mesh view of 3x3 is used for sorting large pieces. One image pickup device is a conveying area corresponding to a plurality of conveying areas, such as conveying areas corresponding to 9 conveying plates, and the conveying area of each conveying plate can be divided into a plurality of grid units; the division of the size of the grid cells in the conveying plate can be divided according to the size of the package, the grid cells with a little less number are divided by the large package, the grid cells with a little more number are divided by the small package, and the method is not limited in detail here.

Because one conveying device cannot convey only one specification of packages all the time, the gridding of the conveying area is not constant, and the gridding treatment of the conveying area can be adjusted at any time according to the conveying requirements of different package sizes to obtain grid units with different specifications. Specifically, the package anomaly detection method may further include a step of performing gridding processing again, and specifically may include steps 601 to 604 in fig. 6:

601. and acquiring an adjustment request for adjusting the grid size of the divided transmission area by a user, wherein the adjustment request comprises new grid size information.

602. And acquiring second registration images corresponding to the plurality of transmission areas respectively acquired by the plurality of camera devices.

603. And respectively carrying out grid boundary point identification on the plurality of transmission areas again according to the new grid size information, and determining grid boundary points in the plurality of transmission areas.

604. And determining the grid units corresponding to the plurality of transmission areas according to the grid boundary points in the plurality of transmission areas.

The processes in steps 601 to 604 are similar to the related steps of the transmission area gridding in steps 401 to 404, and reference may be made to the above embodiments specifically, which are not described herein again.

The image resolution of the imaging device is generally 1080p, 720p, 480p, and the like. Different resolutions of images are combined with different focal lengths and different visual angle areas formed by the lens, and in the environment of a sorting area of logistics network points, the fact that one camera device has a visual field corresponding to one transmission plate is obviously quite a waste matter. In the embodiment of the application, a large-field camera with a unified specification (for example, a 1080p large-field camera) can be used as a unified standard, different field coverage ranges can be obtained only by adjusting the height of the camera, and the transmission area can be registered by optionally performing gridding in the field.

As shown in fig. 7, the step 302 of performing package detection on the package delivery images of the delivery areas to determine the position information and the status information of the packages in the delivery areas may further include:

701. and respectively taking the parcel delivery images of the delivery areas as target parcel images, carrying out parcel detection on the target parcel images, and determining the current placement area of parcels in the target parcel images.

The target package image is subjected to package detection, and an existing target detection algorithm, such as a V-J detection algorithm, a YOLO algorithm, and the like, can be adopted for determining a current placement area of a package in the target package image.

702. And acquiring coordinate information corresponding to the current parcel placement area based on a pre-established coordinate system.

After the current placement area of the parcel in the target parcel image is determined in step 701, the coordinate information corresponding to the current placement area of the parcel can be directly acquired in a pre-established coordinate system.

It should be noted that, in the embodiment of the present application, when the plurality of conveyors are arranged, the plurality of conveyor areas are regularly arranged, for example, arranged side by side, and the pre-established coordinate system may be established by the plurality of conveyor areas, for example, when the plurality of conveyors are arranged side by side, a coordinate system is established by using one corner endpoint of the plurality of conveyors as an origin, a transverse boundary of the conveyor corresponding to an origin in the plurality of conveyors as an abscissa, and a longitudinal boundary as an ordinate, and as shown in fig. 8, a coordinate system is established by using an upper left endpoint of the leftmost conveyor 1 as an origin, a transverse boundary of the conveyor 1 as an abscissa, and a longitudinal boundary as an ordinate.

703. And acquiring the position information of the parcel in the corresponding delivery area in the target parcel image according to the coordinate information corresponding to the current parcel placement area.

Specifically, after the coordinate information corresponding to the currently placed area of the parcel is obtained in step 702, the position information of the parcel in the corresponding delivery area in the image of the target parcel of the cargo in the pre-established coordinate system may be obtained according to the coordinate information corresponding to the currently placed area of the parcel.

704. And detecting the state of the packages in the target package image to obtain the state information of the packages in the target package image.

In this embodiment of the application, the state of the package in the target package image is detected in step 704, and the obtained state information of the package in the target package image can be detected by using a pre-trained neural network model, specifically, the target package image is input into the pre-trained neural network model, and the state information of the package in the target package image is output.

The pre-trained Neural Network model may be a Convolutional Neural Network (CNN) model, or a Recurrent Neural Network (RNN) model, and the like, which is not limited herein.

In the embodiment of the present application, the abnormal state of the package may include the following:

the stacking state: the state of overlapping and bundling packages occurs;

and (3) a damaged state: breakage of the package;

non-standard logistics: wrapping a standard logistics piece;

the clamping piece state: the stay time in the conveying area is continuously detected, if the stay time exceeds a threshold value t set by a system.

The initial neural network model can be trained by collecting a large number of images of the type of the stacked state, the damaged state, the non-standard logistics piece and the like, so that the pre-trained neural network model is obtained.

In the embodiment of the application, the state information of the packages in the target package image can be obtained by detecting the state of the packages in the target package image, and further the state information of the packages in the plurality of conveying areas, such as the information of the above abnormal states of the packages and the normal state information of the packages, can be obtained. It should be noted that the status information of the package may be a text, for example, as shown in fig. 8, the "stack" indicates a "stack status", the "clip" indicates a "clip status", and the "normal" indicates a "normal status", etc.; it can also be preset with various different marks (which can be letters, numbers or their combination), such as "1" for "stack state", "2" for "clip state", "3" for "normal state", etc.

After determining the status information of the parcels in the plurality of delivery areas, it may be determined which areas of the plurality of delivery areas have parcels with status abnormal and which parcels with status normal.

In an embodiment of the present application, the generating abnormal status information of the parcels in the plurality of delivery devices based on the location information of the parcels and the status information of the parcels in step 303 may further include:

Specifically, a parcel status vector may be generated according to the location information of the parcel and the location information and the status information of each parcel in the status information of the parcel; and filling the package state vector into a preset package abnormal state information table to obtain the integrated information of the abnormal state of the packages in the plurality of conveying devices.

It should be noted that, in the foregoing embodiments, the state information of the packages in the multiple delivery areas is all filled in the preset package abnormal state information table, and it is understood that, in some other embodiments of the present application, in order to reduce the data volume, valid information is retained, and only the package abnormal state information in the state information of the packages in the multiple delivery areas may be filled in the preset package abnormal state information table (that is, the normal state information of the packages is ignored).

In the package abnormal state information table, each cell corresponds to one of the plurality of conveying areas, the cells can be filled with state information corresponding to packages, if the packages in a certain conveying area are in a normal state, the cells corresponding to the package abnormal state information table are not filled with information, of course, identifiers representing the normal state can be filled, and specific details are not limited here.

As shown in fig. 8, a diagram showing the relationship between the information of the packages and the abnormal state of the delivery area is shown, in fig. 8, box represents the packages, each grid represents a grid unit, a plurality of packages in one grid unit represent stacks, and the packages are always represented in one grid unit within a preset time period to represent package clamps.

The package state vector can be represented by p ═ p1, p2, p3.. ], p1 represents the package state in grid unit 1, p2 represents the package state in grid unit 2, p3 represents the package state in grid unit 3, and so on, the package state vector is filled into a preset package abnormal state information table, and finally a visual gridding abnormal state integrated information table is formed. The abnormal state integrated information table can be sent to the user terminal corresponding to the administrator in real time for sorting or transferring. Fig. 9 is a schematic diagram of the abnormal state integrated information table after 2 × 2 and 3 × 3 gridding.

After the abnormal state integrated information table is obtained, the control device may send the abnormal state integrated information table to a user device of an administrator to remind the administrator to perform corresponding processing, so that the abnormality can be solved in time, and therefore after the abnormal state information of the packages in the plurality of conveying devices is generated based on the position information of the packages and the state information of the packages, the method in the embodiment of the present application may further include: determining a transfer region where an abnormality exists based on the abnormality state integration information; and sending prompt information to the user equipment corresponding to the abnormal transmission area, wherein the prompt information comprises the abnormal state integrated information.

Based on the principle mechanism, the original single-view parcel abnormal state information is converted into a multi-region object abnormal state information table, and the information contains the related state that the parcel is at a certain position of the conveying plate at the moment. The manager, having obtained this information, can then know the actual status of the package at a particular delivery area based on the status information. If the packages are found to be in an abnormal operation state, the operation of the nearby conveying device can be immediately stopped, other packages can be enabled to bypass the problem area, and the subsequent sorting task can be continuously carried out. Relevant managers can immediately deal with the relevant problems of the packages in the problem conveying area, and meanwhile, subsequent sorting production is not influenced, and sorting efficiency and sorting quality are improved.

In order to better implement the package anomaly detection method in the embodiment of the present application, on the basis of the package anomaly detection method, an embodiment of the present application further provides a package anomaly detection device, as shown in fig. 10, where the package anomaly detection device 1000 includes:

the system comprises an image acquisition module 1001, an image processing module and a display module, wherein the image acquisition module is used for acquiring parcel conveying images of a plurality of conveying areas acquired by a plurality of camera devices, the plurality of conveying areas are conveying areas of the plurality of conveying devices in a preset logistics area, the vision coverage areas of the plurality of camera devices cover the plurality of conveying areas, and each camera device corresponds to at least one conveying area in the plurality of conveying areas;

a parcel detection module 1002, configured to perform parcel detection on the parcel delivery images in the multiple delivery areas respectively, so as to determine position information and state information of parcels in the multiple delivery areas;

an information generating module 1003, configured to generate abnormal state integrated information of the packages on the multiple conveying devices based on the location information of the packages and the state information of the packages.

In some embodiments of the present application, the package detection module 1002 is specifically configured to:

In some embodiments of the present application, the information generating module 1003 is specifically configured to:

The embodiment of the present invention further provides a control device, which integrates any package anomaly detection apparatus provided by the embodiment of the present invention, and the control device includes:

one or more processors;

a memory; and

one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor for performing the steps of the package anomaly detection method in any of the above package anomaly detection method embodiments.

The embodiment of the invention also provides control equipment which integrates any package abnormity detection device provided by the embodiment of the invention. As shown in fig. 11, a schematic structural diagram of a control device according to an embodiment of the present invention is shown, specifically:

the control device may include components such as a processor 1101 of one or more processing cores, memory 1102 of one or more computer-readable storage media, a power supply 1103, and an input unit 1104. Those skilled in the art will appreciate that the control device configuration shown in fig. 11 does not constitute a limitation of the control device and may include more or fewer components than shown, or some components may be combined, or a different arrangement of components. Wherein:

the processor 1101 is a control center of the control apparatus, connects various parts of the entire control apparatus by various interfaces and lines, and performs various functions of the control apparatus and processes data by running or executing software programs and/or modules stored in the memory 1102 and calling data stored in the memory 1102, thereby performing overall monitoring of the control apparatus. Optionally, processor 1101 may include one or more processing cores; preferably, the processor 1101 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 1101.

The memory 1102 may be used to store software programs and modules, and the processor 1101 executes various functional applications and data processing by operating the software programs and modules stored in the memory 1102. The memory 1102 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to the use of the control apparatus, and the like. Further, the memory 1102 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 1102 may also include a memory controller to provide the processor 1101 with access to the memory 1102.

The control device further includes a power supply 1103 for supplying power to the various components, and preferably, the power supply 1103 is logically connected to the processor 1101 via a power management system, so that functions of managing charging, discharging, and power consumption are implemented via the power management system. The power supply 1103 may also include any component, such as one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like.

The control device may further include an input unit 1104, and the input unit 1104 may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

Although not shown, the control apparatus may further include a display unit and the like, which are not described in detail herein. Specifically, in this embodiment, the processor 1101 in the control device loads an executable file corresponding to a process of one or more application programs into the memory 1102 according to the following instructions, and the processor 1101 runs the application programs stored in the memory 1102, so as to implement various functions as follows:

It will be understood by those skilled in the art that all or part of the steps of the methods of the above embodiments may be performed by instructions or by associated hardware controlled by the instructions, which may be stored in a computer readable storage medium and loaded and executed by a processor.

To this end, an embodiment of the present invention provides a computer-readable storage medium, which may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like. The package anomaly detection method comprises a computer program stored on the computer program, and the computer program is loaded by a processor to execute the steps of any package anomaly detection method provided by the embodiment of the invention. For example, the computer program may be loaded by a processor to perform the steps of:

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and parts that are not described in detail in a certain embodiment may refer to the above detailed descriptions of other embodiments, and are not described herein again. In a specific implementation, each unit or structure may be implemented as an independent entity, or may be combined arbitrarily to be implemented as one or several entities, and the specific implementation of each unit or structure may refer to the foregoing method embodiment, which is not described herein again.

The package anomaly detection method, the package anomaly detection device, the control equipment and the storage medium provided by the embodiment of the application are introduced in detail, a specific example is applied in the description to explain the principle and the implementation mode of the application, and the description of the embodiment is only used for helping to understand the method and the core idea of the application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. A package anomaly detection method, the method comprising:

2. The package anomaly detection method according to claim 1, wherein each delivery area corresponds to a predetermined plurality of delivery grid cells, and before said capturing package delivery images of a plurality of delivery areas of a plurality of delivery devices by a plurality of cameras, the method further comprises:

3. The parcel anomaly detection method according to claim 2, wherein said grid positioning recognition and image visual field division are performed on a plurality of conveying areas on said plurality of conveying devices by said plurality of cameras to obtain grid cells corresponding to each conveying area and a visual field coverage of each camera in said plurality of conveying areas, comprising:

4. The package anomaly detection method according to claim 3, further comprising:

5. The package anomaly detection method according to any one of claims 1 to 3, wherein the package detection for the package delivery images of the plurality of delivery areas to determine the position information and the state information of the package in the plurality of delivery areas comprises:

6. The package abnormality detection method according to claim 1, wherein the generating of the abnormal state information of the package in the plurality of conveyor devices based on the position information of the package and the state information of the package includes:

7. The package abnormality detection method according to claim 1, after generating abnormal state information of a package in a plurality of conveying devices based on the position information of the package and the state information of the package, the method further comprising:

8. A package anomaly detection device, said device comprising:

9. A control apparatus, characterized in that the control apparatus comprises:

one or more processors;

a memory; and

one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the package anomaly detection method of any one of claims 1-7.

10. A computer-readable storage medium, having stored thereon a computer program which is loaded by a processor to perform the steps of the package anomaly detection method according to any one of claims 1 to 7.