CN111027418A - Factory material management method and device and electronic equipment - Google Patents
Factory material management method and device and electronic equipment Download PDFInfo
- Publication number
- CN111027418A CN111027418A CN201911150133.6A CN201911150133A CN111027418A CN 111027418 A CN111027418 A CN 111027418A CN 201911150133 A CN201911150133 A CN 201911150133A CN 111027418 A CN111027418 A CN 111027418A
- Authority
- CN
- China
- Prior art keywords
- image
- area
- materials
- determining
- trained
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the invention provides a factory material management method, a device and electronic equipment, and relates to the field of factory management, wherein in the factory material management method, a ground area which is not allowed to be placed with materials is marked in a monitoring area in advance through a marking line, and the method comprises the steps of obtaining an image of the monitoring area; applying a pre-trained image semantic segmentation model, segmenting the image of the monitoring area based on the marking line, and determining the image area where the material is not allowed to be placed; applying a pre-trained material identification model to identify the materials in the image of the monitoring area and determining the image area of material stacking; if the ratio of the intersection area between the image area of the material stack and the image area which does not allow the material to be placed to the image area of the material stack is larger than a first threshold value, determining that the condition of the material misplacement exists; the method solves the problem of low accuracy of the identification result in material management in the prior art, and can improve the accuracy of the identification result.
Description
Technical Field
The invention relates to the field of factory management, in particular to a factory material management method and device and electronic equipment.
Background
The 6S management is a modern enterprise management model, wherein the 6S may include consolidation (SEIRI), consolidation (SEITON), sweep (SEISO), clean (SEIKETSU), literacy (SHITSUKE), SECURITY (SECURITY), and the like. The 6S management is an important method for modern factory management, and comprises various links such as human, machine, material, method and ring.
At present, in the existing 6S management, the management of materials is mainly realized by a material manager manually patrolling the materials, for example, whether the materials in a plant are randomly placed is judged. However, the material management method depends on the subjective judgment of a material manager by means of manual management, and the accuracy of the identification result is low.
Disclosure of Invention
In view of the above, the present invention provides a method, an apparatus, an electronic device and a computer-readable storage medium for managing factory materials.
In order to achieve the above purpose, the embodiment of the present invention adopts the following technical solutions:
in a first aspect, an embodiment of the present invention provides a factory material management method, in which a ground area where materials are not allowed to be placed is previously divided by a marking line in a monitored area, the method including the following steps:
acquiring an image of a monitoring area;
applying a pre-trained image semantic segmentation model, segmenting the image of the monitoring area based on the marking line, and determining an image area where the material is not allowed to be placed;
applying a pre-trained material identification model to identify the materials in the image of the monitoring area and determining an image area for stacking the materials;
and if the ratio of the intersection area between the image area of the material stack and the image area which does not allow the material to be placed relative to the image area of the material stack is larger than a first threshold value, determining that the condition of the material misplacement exists.
In an alternative embodiment, the image semantic segmentation model is a network model based on ResNet; the material identification model is an object detection model based on Mask R-CNN.
In an alternative embodiment, before the step of acquiring an image of a monitored area, the method further comprises:
acquiring a training sample, wherein the training sample comprises multi-angle and multi-scene image samples of the monitoring area, and the image samples comprise pixel level marks of image areas which are predetermined and not allowed to place materials;
and applying the training sample to train the initial image semantic segmentation model to obtain the trained image semantic segmentation model.
In an optional embodiment, the identifying the material in the image of the monitoring area by using a pre-trained material identification model, and the determining the image area of the material stack includes:
identifying the materials in the images of the monitoring area by applying a pre-trained material identification model to obtain the image area to be confirmed of the materials;
applying a pre-trained operator identification model, identifying operators in the image of the monitoring area, and determining the positions of the operators, wherein the operators comprise operators and/or working tools;
and if the distance between the image area to be confirmed and the operator is greater than a second threshold value, determining that the image area to be confirmed is the image area of the material stack.
In an alternative embodiment, the method further comprises:
and if the distance between the image area to be confirmed and the operator is smaller than a second threshold value, determining that the materials in the image area to be confirmed are temporarily stacked.
In an alternative embodiment, the method further comprises:
judging whether the current time is the non-manual operation time;
and if the current time is the non-manual operation time, executing the application of the pre-trained material identification model, identifying the material in the image of the monitoring area, and determining the image area of the material stack.
In a second aspect, an embodiment of the present invention provides a factory material management method, where a ground area where a material is allowed to be placed is marked in a monitored area in advance by a second marking line; the method comprises the following steps:
acquiring an image of a monitoring area;
applying a pre-trained second image semantic segmentation model, segmenting the image of the monitoring area based on the second marking line, and determining an image area allowing to place a material;
applying a pre-trained material identification model to identify the materials in the image of the monitoring area and determining an image area for stacking the materials;
and if the ratio of the intersection area between the image area for stacking the materials and the image area for allowing the materials to be placed relative to the image area for stacking the materials is smaller than a preset threshold value, determining that the condition of the messy placement of the materials exists.
In a third aspect, an embodiment of the present invention provides a plant material management apparatus, in which a ground area where a material is not allowed to be placed is previously divided by a marking line in a monitored area, the apparatus including:
the acquisition module is used for acquiring an image of a monitoring area;
the segmentation module is used for applying a pre-trained image semantic segmentation model, segmenting the image of the monitoring area based on the marking line and determining an image area where materials are not allowed to be placed;
the identification module is used for applying a pre-trained material identification model to identify the materials in the image of the monitoring area and determine the image area of material stacking;
and the determining module is used for determining that the situation of the misplaced materials exists if the ratio of the intersection area between the image area of the stacked materials and the image area of the non-allowed materials to the image area of the stacked materials is larger than a first threshold value.
In a fourth aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, where the memory stores machine executable instructions capable of being executed by the processor, and the processor can execute the machine executable instructions to implement the method described in any one of the foregoing embodiments.
In a fifth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the method according to any one of the foregoing embodiments.
The embodiment of the invention provides a factory material management method, a device, electronic equipment and a computer readable storage medium, wherein in the factory material management method, a ground area which is not allowed to be placed with materials is marked in a monitoring area in advance through a marking line; then, a pre-trained image semantic segmentation model is applied, the image of the monitoring area is segmented based on the marking line, and the image area where the material is not allowed to be placed is determined; applying a pre-trained material identification model to identify the materials in the image of the monitoring area and determine an image area for stacking the materials; and if the ratio of the intersection area between the image area of the material stack and the image area which does not allow the material to be placed relative to the image area of the material stack is larger than a first threshold value, determining that the condition of the material misplacement exists. Therefore, according to the technical scheme provided by the embodiment of the invention, the ground area is labeled in the monitoring area in advance, then the pre-trained image semantic segmentation model is applied to identify the ground area where the material is not allowed to be placed, and the pre-trained material identification model pair is applied to identify the material area of the material opposite side; finally, judging whether the situation of the disordered placement of the materials occurs or not by calculating the percentage of the material area in the ground area; the method solves the problem of low accuracy of the identification result in material management in the prior art, and can improve the accuracy of the identification result. In addition, the method is an intelligent factory building interior material management method based on ground semantic segmentation and material detection, materials and the like in a factory building can be intelligently identified, whether the materials are placed in disorder or not is analyzed, and real-time online management of 6S data is facilitated.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
FIG. 1 is a flow chart illustrating a method for managing plant materials according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a method for obtaining an image semantic segmentation model according to an embodiment of the present invention;
FIG. 3 shows a flowchart of step S106 in FIG. 1;
FIG. 4 is a flow diagram illustrating another method for plant material management provided by an embodiment of the present invention;
FIG. 5 is a schematic diagram of a plant material management apparatus according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of another plant material management apparatus provided by an embodiment of the present invention;
fig. 7 shows a schematic diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.
It is noted that relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
At present, in the existing 6S management, the management of materials is mainly realized by a material manager manually patrolling the materials, for example, whether the materials in a plant are randomly placed is judged. However, the material management method depends on the subjective judgment of a material manager by means of manual management, and the accuracy of the identification result is low.
Based on this, the factory material management method, the device, the electronic device and the computer-readable storage medium provided by the embodiment of the invention can alleviate the problem of low accuracy of the identification result in material management in the prior art, and can improve the accuracy of the identification result.
The present disclosure is further described with reference to the following figures and detailed description.
Referring to fig. 1, an embodiment of the present invention provides a method for managing factory materials, which is applied to the field of factory material management, and is applied to an electronic device, before the method is executed, a ground area where materials are not allowed to be placed is marked out in a monitoring area in advance through a marking line, where the monitoring area may be an entire area of a factory building or the like of the factory building or a local area in the factory building, the marking line may be marked with a preset color, such as red, and the marking line may also be marked with a line type, such as a solid line.
Specifically, the method comprises the following steps:
step S102, acquiring an image of a monitoring area;
step S104, a pre-trained image semantic segmentation model is applied, based on the marking line, the image of the monitoring area is segmented, and the image area where the material is not allowed to be placed is determined;
step S106, applying a pre-trained material identification model to identify the material in the image of the monitoring area and determine the image area of material stacking;
step S108, if the ratio of the intersection area between the image area of the material stack and the image area not allowing the material to be placed relative to the image area of the material stack is larger than a first threshold value, determining that the condition of the material misplacing exists.
In step S102, an image of the monitoring area may be acquired by setting an acquisition device such as a camera.
In step S104, the image semantic segmentation model is trained in advance, and the image of the monitoring area can be segmented according to the marked line, so as to determine the image area where the material is not allowed to be placed; namely, the trained image semantic segmentation model can automatically identify the ground area where the material is not allowed to be placed.
In step S106, the material identification model is also trained in advance, and can identify the material in the image of the monitoring area, and determine the image area where the material is stacked, that is, the trained material identification model can detect the area where the material is stacked, and identify the position of the material and the information of the external frame.
In step 108, the material misplacement behavior is logically judged according to the image area where the material is not allowed to be placed and the image area where the material is stacked, so that whether the material misplacement behavior exists or not is identified.
Specifically, whether the situation of the random placement of the materials occurs is judged by calculating the ratio of the intersection area between the image area where the materials are stacked and the image area where the materials are not allowed to be placed relative to the image area where the materials are stacked.
And if the ratio of the intersection area between the image area of the material stack and the image area which does not allow the material to be placed relative to the image area of the material stack is larger than a first threshold value, determining that the condition of the material misplacement exists.
And if the ratio of the intersection area between the image area of the material stack and the image area allowing the material to be placed relative to the image area of the material stack is not more than (less than or equal to) a first threshold value, determining that no material misplacement condition exists.
The intersection region refers to an intersection region or an overlapping region; the first threshold may be set according to actual needs, for example, set to 0.5; if the material is larger than 0.5, the material is determined to be randomly placed, otherwise, the material is determined not to be randomly placed.
In an optional embodiment, the image semantic segmentation model is a semantic segmentation algorithm based on deep learning to segment the monitoring region; in other words, the image semantic segmentation model is a network model of a deep learning-based semantic segmentation algorithm.
In an alternative embodiment, the image semantic segmentation model is a network model based on ResNet;
specifically, the image semantic segmentation model is a UNET network model based on ResNet 18.
In an alternative embodiment, before the step of acquiring the image of the monitored area, the method further includes a step of acquiring an image semantic segmentation model, which, referring to fig. 2, includes:
step S202, obtaining a training sample, wherein the training sample comprises multi-angle and multi-scene image samples of the monitoring area, and the image samples comprise pixel level labels of image areas which are predetermined and not allowed to place materials;
for example, a large amount of training data required for identifying a ground area (here, an image area where materials are not allowed to be placed) can be obtained by collecting image data of a large number of different perspectives and different plants as image samples and performing pixel-level labeling on the ground area (here, an image area where materials are not allowed to be placed) in the image data.
And step S204, applying the training sample to train the initial image semantic segmentation model to obtain the trained image semantic segmentation model.
After training data of a ground area (an image area where materials are not allowed to be placed) is obtained, an image semantic segmentation model of the ground area is trained by adopting a semantic segmentation algorithm based on deep learning, specifically, a UNet (U-Net: connected networks for biological image segmentation) algorithm which is popular in the field of image semantic segmentation is adopted for ground segmentation, and ResNet18 is adopted as a basic network model, so that the trained model can achieve very high ground segmentation accuracy.
In an alternative embodiment, the material identification model is an optical flow algorithm-based object detection model for identifying a moving object in a video stream, in other words, the material identification model is an optical flow algorithm and deep learning-based object detection model.
In an optional embodiment, the material identification model is a Mask R-CNN-based object detection model.
In order to obtain the positions of the materials, in an optional implementation manner, a material detection model based on deep learning is selected in the embodiment.
In order to obtain training data required by material detection, the embodiment adopts an optical flow algorithm (optical flow method) -based method to identify moving objects in a video stream, extracts common materials except people and vehicles, and can greatly improve the efficiency of training data preparation.
Specifically, the material identification model adopted in the embodiment is an object detection model based on an optical flow algorithm and Mask R-CNN, and the training of the target detection model is performed on the material which is prone to being placed in disorder, so that the position and the external frame information of the material can be accurately identified.
In the factory material management process, the problem of identification errors can be caused due to the fact that operators participate in material transportation.
In an alternative embodiment, referring to fig. 3, step S106 includes the following steps:
step S302, a pre-trained material identification model is applied to identify the material in the image of the monitoring area, and an image area to be confirmed of the material is obtained;
step S304, applying a pre-trained operator identification model, identifying an operator in the image of the monitoring area, and determining the position of the operator, wherein the operator comprises an operator and/or a working tool;
the operating personnel comprise material transportation personnel, logistics sorting personnel and the like, and the operating tools can be transport tools such as a forklift and the like and can also be sorting tools such as a material sorting machine and the like.
It should be noted that similarly, the operator identification model may also identify the operator and/or the work tool in the video stream based on an optical flow algorithm.
In other embodiments, other existing target detection algorithms (e.g., face detection algorithms) may be used to identify the operator; similarly, the work tool may be identified, which will not be described in detail.
And S306, if the distance between the image area to be confirmed and the operator is greater than a second threshold value, determining that the image area to be confirmed is the image area for stacking the materials.
And S308, if the distance between the image area to be confirmed and the operator is smaller than a second threshold value, determining that the materials in the image area to be confirmed are temporarily stacked.
The method can eliminate the influence caused by manual operation, for example, the situation that materials are transported manually and materials are transported by a forklift when people start to participate in operation, at the moment, the materials are often considered to be placed properly, so that the materials are not identified as having a random placing behavior, namely, the method can be more suitable for a factory logistics management scene, the adaptability of the model to the factory logistics management scene is improved, and the problem of identification deviation is avoided.
In view of the fact that in actual operation, when people work manually, such as manually transporting materials and people drive a forklift to transport materials, the materials are often considered to be placed properly. Therefore, in order to avoid the problem of recognizing the above situation as the presence of recognition errors caused by the play-out behavior, in an alternative embodiment, the method further comprises:
(1) judging whether the current time is the non-manual operation time or not;
specifically, whether the current time is in a non-manual operation time interval is judged, and if yes, the current time is determined to be non-working time; otherwise, the current time is determined not to be the non-manual work time, or the current time is the human work time.
(2) And if the current time is the non-manual operation time, executing the pre-trained material identification model, identifying the material in the image of the monitoring area, and determining the image area of material stacking.
(3) And (4) if the current time is not the non-manual work time, namely the current time is the human work time, executing the step.
(4) Determining the position of an operator in the image of the monitoring area, wherein the operator comprises an operator and/or a working tool; and logically judging the material random placement behavior based on the position of the operator and the image area of the material stacking.
Specifically, whether the physical distribution disorder behavior exists is judged according to whether the distance between the position of an operator in the image and the image area for stacking the materials is smaller than a distance threshold value;
if the distance between the position of the operator in the image and the image area for stacking the materials is smaller than a distance threshold value, temporary storage is judged, namely, no physical distribution disorder is existed; and if the distance between the position of the operator in the image and the image area for stacking the materials is not less than the distance threshold value, judging that the physical distribution disorder behavior exists.
The influence of operators in the operation process can be eliminated through the steps (1) - (4), the adaptability of the factory material management method in an actual logistics management scene is improved, the identification result is more accurate, in addition, people (operators), vehicles (operation tools), materials and the like in a factory building can be intelligently identified by the method, the real-time detection of the materials, the people, the vehicles and the like in the factory building is realized, the ground area can be divided and identified in real time, and whether the materials are placed in disorder or not is analyzed, the method can monitor the placement state of the materials in the factory in real time, and the real-time online management of 6S data is facilitated.
Referring to fig. 4, the present embodiment further provides another factory material management method, which is applied in the field of factory material management, and is applied to an electronic device, before the method is executed, a ground area where materials are allowed to be placed is pre-divided in a monitoring area by a mark line, where the monitoring area may be an entire area of a factory building or the like of the factory building or a local area in the factory building, a second mark line may be marked with a preset color, such as yellow, and the mark line may also be marked with a line type, such as a dotted line, in the present embodiment, the mark line is a yellow dotted line to mark the ground area where materials are allowed to be placed, and the rest ground areas are not allowed to place materials.
Specifically, the method comprises the following steps:
step S402, acquiring an image of a monitoring area;
s404, applying a pre-trained second image semantic segmentation model, segmenting the image of the monitoring area based on the second marking line, and determining an image area allowing to place a material;
step S406, a pre-trained material identification model is applied to identify the materials in the image of the monitoring area, and the image area of material stacking is determined;
step S408, if the ratio of the intersection area between the image area for stacking the materials and the image area for allowing the materials to be placed relative to the image area for stacking the materials is smaller than a preset threshold value, determining that the situation of the materials being randomly placed exists.
Before the factory material management method is executed, marking a ground area allowing to place materials in a monitored area in advance through a second marking line; when in execution, firstly, an image of a monitoring area is obtained; secondly, applying a pre-trained second image semantic segmentation model, segmenting the image of the monitoring area based on the second marking line, and determining an image area allowing to place a material; applying a pre-trained material identification model to identify the materials in the image of the monitoring area and determine an image area for stacking the materials; and if the ratio of the intersection area between the image area for stacking the materials and the image area for allowing the materials to be placed relative to the image area for stacking the materials is smaller than a preset threshold value, determining that the condition of the messy placement of the materials exists. Therefore, the method relieves the problem of low accuracy of the identification result in the prior art, and improves the identification accuracy.
In an alternative embodiment, the second image semantic segmentation model is a ResNet-based network model; the material identification model is an object detection model based on Mask R-CNN.
It will be appreciated that the second image segmentation model is trained on image samples of marked ground areas where material placement is allowed, whereas the image segmentation model described above is trained on image samples of marked ground areas where material placement is not allowed.
In an alternative embodiment, before the step of acquiring an image of a monitored area, the method further comprises:
acquiring a training sample, wherein the training sample comprises multi-angle and multi-scene image samples of the monitoring area, and the image samples comprise pixel level marks of image areas allowing materials to be placed;
and applying the training sample to train the initial image semantic segmentation model to obtain a trained second image semantic segmentation model.
Referring to fig. 5, the present embodiment further provides a factory material management apparatus 500, which is configured to previously mark a ground area in a monitoring area, where the ground area is not allowed to be placed with a material, by using a mark line, and includes:
an obtaining module 501, configured to obtain an image of a monitoring area;
a segmentation module 502, configured to apply a pre-trained image semantic segmentation model, segment the image of the monitoring area based on the indication line, and determine an image area where a material is not allowed to be placed;
the identification module 503 is configured to apply a pre-trained material identification model to identify a material in the image of the monitored area, and determine an image area where the material is stacked;
a determining module 504, configured to determine that a material misplacing condition exists if a ratio of a crossing area between the image area of the material stack and the image area where the material is not allowed to be placed with respect to the image area of the material stack is greater than a first threshold.
In an alternative embodiment, the image semantic segmentation model is a network model based on ResNet; the material identification model is an object detection model based on Mask R-CNN.
In an optional implementation manner, the apparatus further includes a training module, configured to obtain a training sample, where the training sample includes multi-angle and multi-scene image samples of the monitoring area, and the image samples include predetermined pixel-level labels of image areas where the material is not allowed to be placed; and applying the training sample to train the initial image semantic segmentation model to obtain the trained image semantic segmentation model.
In an optional embodiment, the identification module 503 is configured to apply a pre-trained material identification model to identify a material in an image of the monitored area and determine an image area where the material is stacked, and is configured to apply the pre-trained material identification model to identify the material in the image of the monitored area to obtain an image area to be confirmed of the material; applying a pre-trained operator identification model, identifying operators in the image of the monitoring area, and determining the positions of the operators, wherein the operators comprise operators and/or working tools; and if the distance between the image area to be confirmed and the operator is greater than a second threshold value, determining that the image area to be confirmed is the image area of the material stack.
In an alternative embodiment, the identification module 503 is configured to determine that the material in the image area to be confirmed is temporarily stacked if the distance between the image area to be confirmed and the operator is smaller than a second threshold.
In an optional embodiment, the apparatus may further include a time determination module, configured to determine whether the current time is a non-manual operation time; and if the current time is the non-manual operation time, executing the pre-trained material identification model by using an identification module 503, identifying the material in the image of the monitored area, and determining the image area of the material stack.
Referring to fig. 6, the present embodiment further provides another factory material management apparatus 600, wherein a ground area where materials are allowed to be placed is marked in a monitored area in advance by a second marking line; the device comprises:
a second obtaining module 601, configured to obtain an image of a monitoring area;
a second segmentation module 602, configured to apply a pre-trained second image semantic segmentation model, segment the image of the monitoring area based on the second marking line, and determine an image area where a material is allowed to be placed;
a second identification module 603, configured to apply a pre-trained material identification model to identify a material in an image of the monitored area, and determine an image area where the material is stacked;
a second determining module 604, configured to determine that a material misplacing condition exists if a ratio of a crossing area between the image area where the material is stacked and the image area where the material is allowed to be placed with respect to the image area where the material is stacked is smaller than a preset threshold.
In an alternative embodiment, the second image semantic segmentation model is a ResNet-based network model; the material identification model is an object detection model based on Mask R-CNN.
It should be understood that the second image segmentation model is trained based on image samples of the marked ground area where the material is allowed to be placed, whereas the image segmentation model is trained based on image samples of the marked ground area where the material is not allowed to be placed.
In an optional implementation manner, the apparatus further includes a second training module, configured to obtain a training sample, where the training sample includes multi-angle and multi-scene image samples of the monitoring area, and the image samples include a predetermined pixel-level label of an image area where the material is allowed to be placed; and applying the training sample to train the initial image semantic segmentation model to obtain a trained second image semantic segmentation model.
Referring to fig. 7, based on the same inventive concept, an embodiment of the invention further provides an electronic device 100, including: a processor 40, a memory 41, a bus 42 and a communication interface 43, wherein the processor 40, the communication interface 43 and the memory 41 are connected through the bus 42; the processor 40 is arranged to execute executable modules, such as computer programs, stored in the memory 41.
The memory 41 may include a high-speed Random Access Memory (RAM) and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 43 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, etc. may be used.
The bus 42 may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 7, but this does not indicate only one bus or one type of bus.
The memory 41 is used for storing a program, the processor 40 executes the program after receiving an execution instruction, and the method executed by the apparatus defined by the flow process disclosed in any of the foregoing embodiments of the present invention may be applied to the processor 40, or implemented by the processor 40.
The processor 40 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 40. The Processor 40 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory 41, and the processor 40 reads the information in the memory 41 and completes the steps of the method in combination with the hardware thereof.
The embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the plant material management method provided in the foregoing embodiment are executed.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative and, for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, each functional module or unit in each embodiment of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a smart phone, a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application.
Claims (10)
1. A factory material management method characterized in that a ground area where materials are not allowed to be placed is previously divided within a monitored area by a marking line, said method comprising the steps of:
acquiring an image of a monitoring area;
applying a pre-trained image semantic segmentation model, segmenting the image of the monitoring area based on the marking line, and determining an image area where the material is not allowed to be placed;
applying a pre-trained material identification model to identify the materials in the image of the monitoring area and determining an image area for stacking the materials;
and if the ratio of the intersection area between the image area of the material stack and the image area which does not allow the material to be placed relative to the image area of the material stack is larger than a first threshold value, determining that the condition of the material misplacement exists.
2. The method according to claim 1, wherein the image semantic segmentation model is a ResNet based network model; the material identification model is an object detection model based on Mask R-CNN.
3. The method of claim 1, wherein prior to the step of acquiring an image of a monitored area, the method further comprises:
acquiring a training sample, wherein the training sample comprises multi-angle and multi-scene image samples of the monitoring area, and the image samples comprise pixel level marks of image areas which are predetermined and not allowed to place materials;
and applying the training sample to train the initial image semantic segmentation model to obtain the trained image semantic segmentation model.
4. The method of claim 1, wherein the step of identifying the material in the image of the monitored area using a pre-trained material identification model, the step of determining the image area of the material stack comprising:
identifying the materials in the images of the monitoring area by applying a pre-trained material identification model to obtain the image area to be confirmed of the materials;
applying a pre-trained operator identification model, identifying operators in the image of the monitoring area, and determining the positions of the operators, wherein the operators comprise operators and/or working tools;
and if the distance between the image area to be confirmed and the operator is greater than a second threshold value, determining that the image area to be confirmed is the image area of the material stack.
5. The method of claim 4, further comprising:
and if the distance between the image area to be confirmed and the operator is smaller than a second threshold value, determining that the materials in the image area to be confirmed are temporarily stacked.
6. The method of claim 1, further comprising:
judging whether the current time is the non-manual operation time;
and if the current time is the non-manual operation time, executing the application of the pre-trained material identification model, identifying the material in the image of the monitoring area, and determining the image area of the material stack.
7. A factory material management method is characterized in that a ground area allowing materials to be placed is marked in a monitored area in advance through a second marking line; the method comprises the following steps:
acquiring an image of a monitoring area;
applying a pre-trained second image semantic segmentation model, segmenting the image of the monitoring area based on the second marking line, and determining an image area allowing to place a material;
applying a pre-trained material identification model to identify the materials in the image of the monitoring area and determining an image area for stacking the materials;
and if the ratio of the intersection area between the image area for stacking the materials and the image area for allowing the materials to be placed relative to the image area for stacking the materials is smaller than a preset threshold value, determining that the condition of the messy placement of the materials exists.
8. A plant material management apparatus characterized in that a ground area where material is not allowed to be placed is previously divided by a marking line in a monitored area, said apparatus comprising:
the acquisition module is used for acquiring an image of a monitoring area;
the segmentation module is used for applying a pre-trained image semantic segmentation model, segmenting the image of the monitoring area based on the marking line and determining an image area where materials are not allowed to be placed;
the identification module is used for applying a pre-trained material identification model to identify the materials in the image of the monitoring area and determine the image area of material stacking;
and the determining module is used for determining that the situation of the misplaced materials exists if the ratio of the intersection area between the image area of the stacked materials and the image area of the non-allowed materials to the image area of the stacked materials is larger than a first threshold value.
9. An electronic device comprising a processor and a memory, the memory storing machine executable instructions executable by the processor to implement the method of any one of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911150133.6A CN111027418B (en) | 2019-11-21 | 2019-11-21 | Factory material management method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911150133.6A CN111027418B (en) | 2019-11-21 | 2019-11-21 | Factory material management method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111027418A true CN111027418A (en) | 2020-04-17 |
| CN111027418B CN111027418B (en) | 2023-04-18 |
Family
ID=70206404
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911150133.6A Active CN111027418B (en) | 2019-11-21 | 2019-11-21 | Factory material management method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111027418B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111783689A (en) * | 2020-07-03 | 2020-10-16 | 三一重工股份有限公司 | Material line pressing identification method and device |
| CN113554001A (en) * | 2021-09-18 | 2021-10-26 | 三一汽车制造有限公司 | Mixer truck flash identification method and system, electronic equipment and mixing station |
| CN114927001A (en) * | 2022-04-19 | 2022-08-19 | 天津二建建筑工程有限公司 | Video monitoring method, system, intelligent terminal and storage medium |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011150460A (en) * | 2010-01-20 | 2011-08-04 | Hitachi Information & Control Solutions Ltd | System and method for managing warehousing/shipping |
| DE102013225476A1 (en) * | 2013-12-10 | 2015-06-11 | Deutsche Telekom Ag | A method and system for improving an intelligent goods management relating to a goods management area with a spatial area within which a plurality of storage communication elements can be placed, the goods management area having a communication interface module respectively locally connected to the storage communication elements has, goods management area, computer program and computer program product |
| US9330474B1 (en) * | 2014-12-23 | 2016-05-03 | Ricoh Co., Ltd. | Distinguishing between stock keeping units using a physical dimension of a region depicted in an image |
| US20170372159A1 (en) * | 2016-06-22 | 2017-12-28 | United States Postal Service | Item tracking using a dynamic region of interest |
| CN107545271A (en) * | 2016-06-29 | 2018-01-05 | 阿里巴巴集团控股有限公司 | Image-recognizing method, device and system |
| CN107729908A (en) * | 2016-08-10 | 2018-02-23 | 阿里巴巴集团控股有限公司 | A kind of method for building up, the apparatus and system of machine learning classification model |
| CN108833857A (en) * | 2018-07-10 | 2018-11-16 | 京东方科技集团股份有限公司 | A kind of shelf monitor method, system and shelf |
| CN108985214A (en) * | 2018-07-09 | 2018-12-11 | 上海斐讯数据通信技术有限公司 | The mask method and device of image data |
| WO2019036931A1 (en) * | 2017-08-23 | 2019-02-28 | 深圳蓝胖子机器人有限公司 | Method, device, and system for placing goods, electronic device and readable storage medium |
| CN109559331A (en) * | 2017-09-27 | 2019-04-02 | 九阳股份有限公司 | A kind of food management method based on video image |
| WO2019136095A1 (en) * | 2018-01-02 | 2019-07-11 | Intelligent Agricultural Solutions Llc | Material flow monitoring system and method |
| US20190228519A1 (en) * | 2018-01-25 | 2019-07-25 | International Business Machines Corporation | Anomaly detection using image-based physical characterization |
| WO2019145145A1 (en) * | 2018-01-26 | 2019-08-01 | Credé Vermoegensverwaltungs-Gmbh + Co. Kg | Device and method for welding hard material bodies to teeth of a saw blade |
| US20190279426A1 (en) * | 2018-03-09 | 2019-09-12 | Staples, Inc. | Dynamic Item Placement Using 3-Dimensional Optimization of Space |
| JP2019159818A (en) * | 2018-03-13 | 2019-09-19 | 矢崎総業株式会社 | Work evaluation device and work evaluation method |
-
2019
- 2019-11-21 CN CN201911150133.6A patent/CN111027418B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011150460A (en) * | 2010-01-20 | 2011-08-04 | Hitachi Information & Control Solutions Ltd | System and method for managing warehousing/shipping |
| DE102013225476A1 (en) * | 2013-12-10 | 2015-06-11 | Deutsche Telekom Ag | A method and system for improving an intelligent goods management relating to a goods management area with a spatial area within which a plurality of storage communication elements can be placed, the goods management area having a communication interface module respectively locally connected to the storage communication elements has, goods management area, computer program and computer program product |
| US9330474B1 (en) * | 2014-12-23 | 2016-05-03 | Ricoh Co., Ltd. | Distinguishing between stock keeping units using a physical dimension of a region depicted in an image |
| US20170372159A1 (en) * | 2016-06-22 | 2017-12-28 | United States Postal Service | Item tracking using a dynamic region of interest |
| CN107545271A (en) * | 2016-06-29 | 2018-01-05 | 阿里巴巴集团控股有限公司 | Image-recognizing method, device and system |
| CN107729908A (en) * | 2016-08-10 | 2018-02-23 | 阿里巴巴集团控股有限公司 | A kind of method for building up, the apparatus and system of machine learning classification model |
| WO2019036931A1 (en) * | 2017-08-23 | 2019-02-28 | 深圳蓝胖子机器人有限公司 | Method, device, and system for placing goods, electronic device and readable storage medium |
| CN109559331A (en) * | 2017-09-27 | 2019-04-02 | 九阳股份有限公司 | A kind of food management method based on video image |
| WO2019136095A1 (en) * | 2018-01-02 | 2019-07-11 | Intelligent Agricultural Solutions Llc | Material flow monitoring system and method |
| US20190228519A1 (en) * | 2018-01-25 | 2019-07-25 | International Business Machines Corporation | Anomaly detection using image-based physical characterization |
| WO2019145145A1 (en) * | 2018-01-26 | 2019-08-01 | Credé Vermoegensverwaltungs-Gmbh + Co. Kg | Device and method for welding hard material bodies to teeth of a saw blade |
| US20190279426A1 (en) * | 2018-03-09 | 2019-09-12 | Staples, Inc. | Dynamic Item Placement Using 3-Dimensional Optimization of Space |
| JP2019159818A (en) * | 2018-03-13 | 2019-09-19 | 矢崎総業株式会社 | Work evaluation device and work evaluation method |
| CN108985214A (en) * | 2018-07-09 | 2018-12-11 | 上海斐讯数据通信技术有限公司 | The mask method and device of image data |
| CN108833857A (en) * | 2018-07-10 | 2018-11-16 | 京东方科技集团股份有限公司 | A kind of shelf monitor method, system and shelf |
Non-Patent Citations (1)
| Title |
|---|
| MARCO SEREGNI,等: "The impact of IoT Technologies on product-oriented PSS" * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111783689A (en) * | 2020-07-03 | 2020-10-16 | 三一重工股份有限公司 | Material line pressing identification method and device |
| CN111783689B (en) * | 2020-07-03 | 2024-02-20 | 盛景智能科技(嘉兴)有限公司 | Material line pressing identification method and device |
| CN113554001A (en) * | 2021-09-18 | 2021-10-26 | 三一汽车制造有限公司 | Mixer truck flash identification method and system, electronic equipment and mixing station |
| CN114927001A (en) * | 2022-04-19 | 2022-08-19 | 天津二建建筑工程有限公司 | Video monitoring method, system, intelligent terminal and storage medium |
| CN114927001B (en) * | 2022-04-19 | 2023-09-26 | 天津二建建筑工程有限公司 | Video monitoring method, system, intelligent terminal and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111027418B (en) | 2023-04-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111027418B (en) | Factory material management method and device and electronic equipment | |
| US11455805B2 (en) | Method and apparatus for detecting parking space usage condition, electronic device, and storage medium | |
| CN111310645B (en) | Method, device, equipment and storage medium for warning overflow bin of goods accumulation | |
| WO2021051885A1 (en) | Target labeling method and apparatus | |
| CN109815912B (en) | Highway safety inspection system based on artificial intelligence | |
| CN113822385B (en) | Coal conveying abnormity monitoring method, device and equipment based on image and storage medium | |
| CN112084822A (en) | Lane detection device and method and electronic equipment | |
| US20160307050A1 (en) | Method and system for ground truth determination in lane departure warning | |
| CN114419038A (en) | Method and device for identifying surface defects of hub, storage medium and electronic equipment | |
| CN111191557B (en) | Mark identification positioning method, mark identification positioning device and intelligent equipment | |
| CN110602446A (en) | Garbage recovery reminding method and system and storage medium | |
| CN114022810A (en) | Method, system, medium and terminal for detecting working state of climbing frame protective net in construction site | |
| CN109887303B (en) | Lane-changing behavior early warning system and method | |
| CN109615610B (en) | Medical band-aid flaw detection method based on YOLO v2-tiny | |
| CN112784639A (en) | Intersection detection, neural network training and intelligent driving method, device and equipment | |
| CN108647570B (en) | Zebra crossing detection method and device and computer readable storage medium | |
| CN106297301A (en) | A kind of vehicle fake-license inspection method and the vehicle fake-license detecting system of use the method | |
| CN112001336A (en) | Pedestrian boundary crossing alarm method, device, equipment and system | |
| CN111860040A (en) | Station signal equipment state acquisition method and device and computer equipment | |
| CN116012939A (en) | Method and device for determining illegal behaviors, storage medium and electronic device | |
| CN110866441A (en) | Vehicle identification and continuation tracking method and device and road side system | |
| CN115063739A (en) | Abnormal behavior detection method, device, equipment and computer storage medium | |
| CN116543327A (en) | Method, device, computer equipment and storage medium for identifying work types of operators | |
| CN110795974B (en) | Image processing method, device, medium and equipment | |
| CN116664582B (en) | Road surface detection method and device based on neural vision network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230106 Address after: 314506 room 116, building 4, No. 288, development avenue, Tongxiang Economic Development Zone, Tongxiang City, Jiaxing City, Zhejiang Province Applicant after: Shengjing Intelligent Technology (Jiaxing) Co.,Ltd. Address before: 102200 5th floor, building 6, No.8 Beiqing Road, Changping District, Beijing Applicant before: SANY HEAVY INDUSTRY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |