WO2023279846A1

WO2023279846A1 - Method and apparatus for generating traceable production data, and device, medium and program

Info

Publication number: WO2023279846A1
Application number: PCT/CN2022/092410
Authority: WO
Inventors: 王飞; 王磊; 林君仪; 周嘉明; 白登峰
Original assignee: 上海商汤智能科技有限公司
Priority date: 2021-07-06
Filing date: 2022-05-12
Publication date: 2023-01-12
Also published as: CN113450125A

Abstract

Provided in the present disclosure are a method and apparatus for generating traceable production data, and an electronic device, a computer-readable storage medium and a computer program product. The method comprises: acquiring video data, which corresponds to an operation area and is collected by a camera device; obtaining abnormal production data on the basis of the video data, wherein the abnormal production data is used for reflecting an abnormal action, which is executed by a target object in the operation area for a target product; and associating a quality detection result of the target product with the abnormal production data, so as to obtain traceable production data for product quality tracing. By means of the present disclosure, abnormal production data can be determined by means of video analysis, product quality tracing can be performed on the basis of the abnormal production data, and in particular, a targeted solution can be provided by means of the abnormal production data when a product has a defect, such that the product qualification ratio is improved.

Description

Method, device, equipment, medium and program for generating traceable production data

Cross References to Related Applications

This disclosure is based on a Chinese patent application with application number 202110761942.1 and application date of July 06, 2021. The applicant is Beijing SenseTime Technology Development Co., Ltd., and the application name is "Method, device, and electronic equipment for generating traceable production data." and storage medium” and claim the priority of the Chinese patent application, the entire content of the Chinese patent application is hereby incorporated by reference into this disclosure.

technical field

The present disclosure relates to the field of computer technology, and relates to but not limited to a generation method, device, electronic equipment, computer readable storage medium and computer program product of traceable production data.

Background technique

Whether in the process of product production or after the completion of product production, it is often necessary to conduct quality inspections on products or semi-finished products to determine qualified products and unqualified products.

Most of the current product inspections start with the product itself, that is, inspectors inspect the composition and appearance of the product one by one to determine whether the product is qualified. However, there is a lack of a corresponding detection plan for what causes the product to fail.

Contents of the invention

Embodiments of the present disclosure at least provide a method, device, electronic device, computer-readable storage medium, and computer program product for generating traceable production data, so as to trace data related to product quality problems and improve product qualification rate.

An embodiment of the present disclosure provides a method for generating traceable production data, the method including:

Obtain the video data corresponding to the operation area collected by the camera equipment;

Obtaining abnormal production data based on the video data, the abnormal production data is used to reflect the abnormal actions performed by the target object in the operation area on the target product;

Associating the quality inspection results of the target product with the abnormal production data to obtain traceable production data for product quality traceability.

Using the method for generating traceable production data described above, the abnormal production data made by the target object in the operation area for the target product can be determined based on the acquired video data. The abnormal production data here can include abnormal actions and make abnormal actions. In this way, in the case of associating the quality inspection results of the target product with abnormal production data, the traceable production data of product quality traceability can be determined. It can be seen that the embodiment of the present disclosure can determine abnormal production data through video analysis, and based on this abnormal production data, product quality can be traced, especially in the case of product defects, targeted solutions can be provided through abnormal production data , in order to improve the product qualification rate.

In some embodiments of the present disclosure, the obtaining abnormal production data based on the video data includes:

identifying an execution action sequence corresponding to the target object in the video data;

Obtaining a reference action sequence corresponding to the operation area;

By comparing the execution action sequence with the reference action sequence, the abnormal actions in the execution action sequence and the problem types of the abnormal actions are obtained.

Here, the abnormal actions in the execution action sequence and the problem type of the abnormal action can be determined based on the comparison results between the execution action sequence of the target object and the reference action sequence, which provides effective data support for subsequent product quality traceability.

In some embodiments of the present disclosure, by comparing the execution action sequence with the reference action sequence, the abnormal actions in the execution action sequence and the problem types of the abnormal actions are obtained, including:

It is determined that the abnormal actions include at least one of actions added, missing actions in the execution action sequence, and actions with the same execution order and action content similarity less than a threshold value compared with the reference action sequence, The problem types that describe abnormal actions include abnormal actions.

It is determined that the abnormal action includes an action whose execution sequence is wrong in the execution action sequence compared with the reference action sequence, and the problem type of the abnormal action includes a step exception.

In some embodiments of the present disclosure, compared with the reference action sequence, the actions in the executed action sequence that are executed in the wrong sequence are determined according to the following steps:

taking each action in the execution action sequence as a first target action, determining the first order of the first target action in the execution action sequence, and determining the first target action from the reference action sequence The first reference action corresponding to the sequence;

In a case where the first target action does not match the first reference action, it is determined that the first target action is an incorrectly executed action in the execution action sequence.

Acquiring a second target action and a second reference action in a second order from the execution action sequence and the reference action sequence according to the arrangement order of the actions in the execution action sequence;

In a case where the second target action does not match the second reference action, determining that the second target action is an incorrectly executed action in the execution action sequence;

In the case that the second order is the first step in the execution action sequence, take the next step in the second order as the second order and repeat the above steps until the steps in the execution action sequence are traversed. All actions; or, in the case where the second order is the last step in the execution action sequence, take the previous step in the second order as the second order and repeat the above steps until the execution is traversed All actions in an action sequence.

Here, the consistency between the action sequences of the two action sequences can be verified from multiple dimensions, that is, the consistency comparison can not only be carried out according to the individual comparison method, but also the consistency comparison can be carried out in combination with the action sequence, thus taking into account more The quality inspection requirements of many application scenarios.

In some embodiments of the present disclosure, after obtaining the abnormal action in the execution action sequence and the problem type of the abnormal action by comparing the execution action sequence with the reference action sequence, the method further includes:

sending to the target device a first result containing the abnormal action in the execution action sequence, and a second result containing the question type of the abnormal action, so that the target device outputs the first result and the second result At least one of the .

Here, the target device can be used to output the first result containing the abnormal action in the action sequence and the second result containing the problem type of the abnormal action, so that the target device can take corresponding actions on the above results. Responses.

In some embodiments of the present disclosure, the method also includes:

In the case of determining that there are multiple abnormal actions in the execution action sequence, determining the importance level of each abnormal action;

The sending the first result containing the abnormal action in the execution action sequence to the target device, and the second result containing the question type of the abnormal action includes:

sending to the target device a first result containing each abnormal action in the execution action sequence, a second result containing the question type of each abnormal action, and the importance level of each abnormal action, so that the target device Display and/or broadcast the abnormal actions and the problem types of the abnormal actions according to the importance levels.

Here, considering that different execution actions may have different importance levels, the corresponding output results can also have different reminder strengths, so it can be displayed and/or broadcast in combination with the importance level, so that when problems occur in different execution actions , for adaptive solutions.

In some embodiments of the present disclosure, the sending to the target device the first result including the abnormal action in the execution action sequence, and the second result including the question type of the abnormal action include:

Obtain the equipment identification of the production equipment corresponding to the operation area;

Sending the first result including the abnormal action in the execution action sequence, the second result including the question type of the abnormal action, and the device identification to the target device, so that the target device can display and/or broadcast.

In some embodiments of the present disclosure, the method also includes:

In response to receiving the product quality traceability request, acquire the traceable production data associated with the target product indicated by the product quality traceability request;

Based on the target detection result in the quality detection results of the target product, the target abnormal production data matching the target detection result is determined from the traceable production data.

An embodiment of the present disclosure also provides a device for generating traceable production data, the device comprising:

The obtaining module is configured to obtain video data corresponding to the operation area collected by the camera equipment;

The determination module is configured to obtain abnormal production data based on the video data, and the abnormal production data is used to reflect the abnormal actions performed by the target object in the operation area on the target product;

The generation module is configured to correlate the quality inspection results of the target product with the abnormal production data to obtain traceable production data for product quality traceability.

An embodiment of the present disclosure also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the The memory communicates with each other through a bus, and when the machine-readable instructions are executed by the processor, the method for generating traceable production data described in any one of the above-mentioned items is executed.

An embodiment of the present disclosure also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor as described in any one of the first aspect and its various implementation modes The generation method of traceable production data.

An embodiment of the present disclosure also provides a computer program product, the computer program product carries a program code, and the instructions included in the program code can be used to execute the method for generating traceable production data described in any one of the above items.

For the effect description of the above-mentioned generating device for traceable production data, electronic equipment, and computer-readable storage medium, please refer to the description of the above-mentioned method for generating traceable production data, which will not be repeated here.

In order to make the above-mentioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments will be described in detail below together with the accompanying drawings.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present disclosure more clearly, the following will briefly introduce the accompanying drawings used in the embodiments. The accompanying drawings here are incorporated into the specification and constitute a part of the specification. The drawings show the embodiments consistent with the present disclosure, and are used together with the description to explain the technical solutions of the present disclosure. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as limiting the scope. For those skilled in the art, they can also make From these drawings other related drawings are obtained.

FIG. 1A is a flowchart of a method for generating traceable production data provided by an embodiment of the present disclosure;

Figure 1B is a schematic flow diagram of determining abnormal production data based on traceable production data provided by the embodiment of the present application;

Fig. 1C is a schematic flow diagram of determining abnormal production data provided by the embodiment of the present application;

FIG. 1D is a schematic flow diagram of determining an action in the execution sequence that is executed in the wrong order provided by the embodiment of the present application;

FIG. 1E is another schematic flow diagram for determining actions performed in the wrong order in the action sequence provided by the embodiment of the present application;

FIG. 2 is a schematic diagram of a generation device for traceable production data provided by an embodiment of the present disclosure;

Fig. 3 is a schematic diagram of an electronic device provided by an embodiment of the present disclosure.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only It is a part of the embodiments of the present disclosure, but not all of them. The components of the disclosed embodiments generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort shall fall within the protection scope of the present disclosure.

It should be noted that like numerals and letters denote similar items in the following figures, therefore, once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

The term "and/or" in this article only describes an association relationship, which means that there can be three kinds of relationships, for example, A and/or B can mean: there is A alone, A and B exist at the same time, and B exists alone. situation. In addition, the term "at least one" herein means any one of a variety or any combination of at least two of the more, for example, including at least one of A, B, and C, which may mean including from A, Any one or more elements selected from the set formed by B and C.

After research, it is found that most of the current product inspections start with the product itself, that is, inspectors inspect the composition and appearance of the product one by one to judge whether the product is qualified. However, there is a lack of a corresponding detection plan for what causes the product to fail.

During production activities, production line workers need to follow certain action specifications, but in the actual production process, there may be situations such as missed or wrong execution of actions, which will lead to unqualified quality of the produced products . At present, factories can set up specific posts to carry out random inspections of execution actions. This method not only consumes manpower and material resources, but also makes it difficult to control the entire production process.

Based on the above research, the embodiments of the present disclosure provide a method, device, electronic device, computer readable storage medium, and computer program product for generating traceable production data, so as to trace the relevant data of product quality problems and assist in finding The root cause of quality problems, and provide targeted adjustment strategies or solutions from the root cause, so as to improve the product qualification rate.

In order to facilitate the understanding of the embodiments of the present disclosure, a method for generating traceable production data provided by the embodiments of the present disclosure is firstly introduced in detail. The execution subject of the method for generating traceable production data provided by the embodiments of the present disclosure is generally a Capable electronic equipment, the electronic equipment includes, for example: terminal equipment, server or other processing equipment; wherein, the terminal equipment can be user equipment (User Equipment, UE), mobile equipment, user terminal, cellular phone, cordless phone, personal digital assistant (Personal Digital Assistant, PDA), handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc. In some embodiments of the present disclosure, the method for generating traceable production data may be implemented by a processor invoking computer-readable instructions stored in a memory.

Referring to FIG. 1A , it is a flow chart of a method for generating traceable production data provided by an embodiment of the present disclosure. The method includes steps S101 to S103, wherein:

Step S101: Obtain video data corresponding to the work area collected by the camera equipment;

Step S102: Obtain abnormal production data based on the video data, and the abnormal production data is used to reflect the abnormal actions performed by the target object in the operation area for the target product;

Step S103: Associating the quality inspection results of the target product with abnormal production data to obtain traceable production data for product quality traceability.

Here, in order to facilitate the understanding of the method for generating traceable production data provided by the embodiments of the present disclosure, the application scenarios of the generation method may be introduced in detail below. The generation method in the embodiments of the present disclosure can be mainly applied to various related fields that require product quality inspection, and in practical applications in these related fields, the capture of video data during the product production process is used as a reference basis instead of Based on the product obtained as a reference.

For product inspection starting from the product itself, to a large extent, it can only verify whether the product itself is a qualified product or a non-qualified product. For non-qualified products, recycling, scrapping or reprocessing can be used for transformation; however, due to It is impossible to know the specific reasons for unqualified products, which leads to the scrapping of unqualified products to a large extent; it can be seen that clarifying the specific reasons for unqualified products is of great significance for product quality inspection and subsequent adjustment of production operations.

Based on this, an embodiment of the present disclosure provides a method for generating traceable production data. The method uses the analysis results of video data to determine the traceable production data corresponding to the quality inspection results, and then can provide targeted information based on the traceable production data. Solutions to improve product qualification rate.

Wherein, the video data may be video clips collected by camera equipment deployed in the operation area, for example, it may be deployed at the production line personnel's position to ensure that the operation area where the personnel are located can be photographed. The camera equipment here can turn on the screen capture function when production is started in the work area, and turn on the power saving mode when it is idle.

For the collected video data, it is possible to determine the abnormal production data made by the target object in the video data for the target product; the abnormal production data here can be the data relative to the normal production data, and the normal production data can include normal production activities In-progress production data, such as reference actions pointed to by normative operations at the time of execution of actions, may also be referred to as standard actions.

Correspondingly, abnormal production data can be abnormal actions that do not match the standard actions, or details of abnormal actions such as whether the abnormal action is slow or fast by how many seconds, the degree of abnormality of the abnormal action, and the person or person corresponding to the abnormal action. Equipment identification and other information, or other abnormal production data.

Among them, the target object can be a production line worker in the operation area. The worker here can be a real person or a robot. At this time, the video data needs to ensure that the worker's operation action can be captured; in addition, the target object can also be It is an operation device that can work automatically in the work area. At this time, the video data needs to ensure that the operation actions of the operation device can be captured.

In the case of associating the quality inspection results of the target product with abnormal production data, the traceable production data that caused the quality failure can be determined from the abnormal production data, which largely clarifies the cause of the quality failure . For example, if it is determined that the cause of the unqualified quality comes from the irregularity of a worker in a certain operation, then targeted guidance can be provided to the worker based on the irregularity of the action.

In addition, in the embodiments of the present disclosure, statistical analysis of data can also be performed based on traceable production data to determine the main cause of unqualified quality, so as to provide more popular solutions and further improve the pass rate of subsequent products.

In the embodiments of the present disclosure, the relevant quality detection results may be determined manually, or based on the trained quality detection network detection, or determined by other methods, and different methods may be used in specific applications , no specific restrictions are made here.

Here, considering that the target product may have multiple quality problems, the generation method of traceable production data provided by the embodiment of the present disclosure can also trace the source of each quality problem, which can be realized through the steps shown in Figure 1B:

Step S1031: In response to receiving the product quality traceability request, obtain traceable production data associated with the target product indicated by the product quality traceability request;

Step S1032: Based on the target detection result in the quality detection results of the target product, determine the target abnormal production data matching the target detection result from the traceable production data.

The target detection result here may be one or several results of the quality detection results. In the case of determining traceable production data, matching abnormal production data can be determined for each quality inspection result. For example, when it is determined that the weight of the target product is not up to the standard, it can be judged whether there are fewer pieces due to missing actions. The result is that the dustproof parts are too loose.

Wherein, the above-mentioned product quality traceability request may be initiated on the target device, for example, it may be triggered by the product producer by triggering a relevant button of an application program (Application, APP) on the handheld user terminal. In this case, the embodiments of the present disclosure can search the data associated with the target product from various traceable production data, so as to realize the acquisition of targeted abnormal production data for different quality problems.

Considering the key impact of the standardization of execution actions on the quality pass rate, the abnormal production data here mainly includes the abnormal actions performed by the target object in the operation area for the target product and the problem types of the abnormal actions, which can be obtained through The steps shown to identify the above abnormal production data:

Step S1021: identifying the execution action sequence corresponding to the target object in the video data; and acquiring a reference action sequence corresponding to the work area;

Step S1022: By comparing the execution action sequence with the reference action sequence, obtain the abnormal actions in the execution action sequence and the problem types of the abnormal actions.

The execution action sequence recognition here can be obtained based on the trained action recognition network recognition, that is, the acquired video data is input into the trained action recognition network, and the execution action made by the target object in the video data can be obtained sequence.

Among them, in the process of training the action recognition network, a plurality of video samples and the action annotation information marked for each video sample in the plurality of video samples can be obtained first, where the video sample corresponds to the action annotation corresponding to an action, or by An action label of a complete action composed of multiple actions. The action label information here can be the operation action of pointing to the target object in the video sample to operate the target product. For example, it can be to unscrew the screw, install the accessory, and then screw on the screw this series of operations.

Then, the video sample can be input into the action recognition network to be trained, the output result of the action recognition network can be determined, and the output result can be compared with the action labeling information marked on the video sample. If the comparison is inconsistent, the current The network parameter value cannot meet the training requirements. At this time, the network parameter value of the action recognition network can be adjusted, and the video sample is input into the adjusted action recognition network, and the network training is performed again until the determined output result is consistent with the action marked by the video sample. The trained action recognition network can be obtained by matching the marked information.

The methods for generating traceable production data provided by the embodiments of the present disclosure can be determined based on other action recognition methods in addition to using the above-mentioned action recognition network to determine the sequence of execution actions made by the target object, which is not specifically limited here.

In the case of determining the execution action sequence of the target object, it is possible to first determine the reference action sequence that matches the operation area corresponding to the current camera equipment from the reference action sequences contained in the template library that match multiple operation areas, and then based on The comparison result between the execution action sequence of the target object and the reference action sequence matching the operation area determines the abnormal actions with problems and the corresponding problem types.

Among them, different reference action sequences can be set for different operation areas. The operation area here can be pre-bound with the product to be operated, the operation time, and the production line workers who operate the product; in the case of determining the operation area, It can be determined that the target product will be operated in the operation area, and there is a reference action sequence corresponding to the target product.

In addition, the abnormal action in the embodiments of the present disclosure may point to a problem in the execution of the action itself, a problem in the sequence of executing the action, or other problems. The problem type of abnormal action can point to the specific action abnormality, whether it is missing an action, or doing too much action, etc., and it can also point to the abnormal step.

In the case that the execution action sequence of the target object includes an execution action, directly compare the execution action with the corresponding reference execution action in the template library; if the actions are consistent, there is no problem with the execution action; if the actions are inconsistent, it means There was a problem executing the action.

In the case that the execution action sequence of the target object includes multiple execution actions, it can be determined whether the execution action sequence of the target object is the same as the reference action sequence based on the execution action sequence of the target object and the reference action sequence matching the work area; There is a problem in determining the executed action of the target object when the executed action sequence of the target object is different from the reference action sequence.

Here, the difference between the execution action sequence of the target object and the reference action sequence may be that the execution action in the execution action sequence of the target object is different from the execution action of the reference action sequence, or it may be that the execution action sequence of the target object is different from the reference action sequence The execution sequence of the execution actions is different, that is, it is necessary to comprehensively evaluate whether there is a problem in combination with the number of execution actions, specific execution actions, and the order of each execution action.

As far as the execution actions in the execution action sequence of the target object are different from the execution actions of the reference action sequence, if the execution action sequence of the target object contains additional execution actions that are not in the reference action sequence, it can be regarded as The target object has done an action that should not be done, that is, it is determined that the target object has the problem of additional execution actions (that is, doing more); when the execution action sequence of the target object does not include all the execution actions in the reference action sequence, It can be determined that the target object has the problem of missing execution actions (that is, omission); in the execution action sequence of the target object, the execution action corresponding to the reference execution action included in the reference action sequence, and the included execution action is the same as the reference execution action In the case where the similarity between the content of the target object is less than the threshold value, that is, when it is determined that the reference execution action has actually been performed, but the actual operation action is not standardized, it can be determined that the target object has made a wrong action (that is, a wrong action); In the execution action sequence of the target object, if the execution order of each execution action is different from that of each reference action in the reference action sequence, it can be determined that the target object has performed a wrong action (that is, performed a wrong action).

In practical applications, it is possible to first judge whether the number of execution actions contained in the execution action sequence of the target object is consistent with the number of execution actions contained in the reference action sequence matching the work area; in the case of inconsistency, it can be directly determined that there is a problem with the execution action ; In the case of consistency, it can be further determined whether the execution actions match.

For the case where the execution sequence of the target object is wrong with the execution sequence of the reference action sequence, it can be determined that the target object has the problem of executing the wrong sequence of actions, which can be described by the following two aspects:

The first aspect: the steps shown in Figure 1D can be followed to determine the actions performed in the wrong order in the action sequence compared to the reference action sequence:

Step S10221: Take each action in the execution action sequence as the first target action, determine the first order of the first target action in the execution action sequence, and determine the first reference corresponding to the first order from the reference action sequence action;

Step S10222: In the case that the first target action does not match the first reference action, determine that the first target action is an incorrectly executed action in the execution action sequence.

Here, for each action of the target object, firstly, the corresponding reference action can be searched from the reference action sequence based on the order of this action in the entire execution action sequence (any step), and judge whether the two actions match. In the case of , it can indicate that there is an incorrectly performed action.

In specific applications, the sequence of execution actions can be determined based on item-by-item comparison, that is, regardless of the action sequence, each action is compared independently of each other; for example, the template library specifies that the first item is action A, and the second item is action B , then when performing detection, associate the detected execution actions of the target object with the action sequence to obtain the actual first item is action C, and the actual second item is action D. By comparing action C with action A, action D Compare with action B to determine whether the order is wrong. Whether action C does not match action A or action D does not match action B, it means that the order is wrong.

The second aspect: the steps shown in Figure 1E can be followed to determine the actions performed in the wrong order in the action sequence compared to the reference action sequence:

Step S10223: Obtain a second target action and a second reference action in the second order from the execution action sequence and the reference action sequence respectively according to the arrangement order of the actions in the execution action sequence;

Step S10224: In the case that the second target action does not match the second reference action, determine that the second target action is an incorrectly executed action in the execution action sequence.

It can be understood that steps S10221-S10222 and steps S10223-S10224 represent implementations of determining actions in the execution sequence that are executed in the wrong order compared with the reference action sequence from two different aspects.

Wherein, in the case that the second order is the first step in the action sequence, take the next step of the second order as the second order and repeat the above steps until all actions in the action sequence are traversed; or, in the second If the order is the last step in the action sequence, take the previous step of the second order as the second order and repeat the above steps until all actions in the action sequence are traversed.

The second order here may be the initial state of the action comparison, for example, the first step or the last step. Taking the first step as the second order, in the case that two actions (ie, the second target action and the second reference action) are obtained for the first step, directly compare whether the two actions match. Regardless of whether it matches or not, here you can select the next two actions for comparison in positive order, and so on, until the last action is traversed.

In a specific application, contextual comparison can be used to determine the sequence of actions to be executed, and to check sequentially, that is, to check in a forward or reverse order, that is, each action is related and not completely independent; for example, the first item in the template library is an action A, the next item adjacent to action A, that is, the second item is action B, then the adjacent relationship needs to be considered when detecting, for example, the first item actually detected is action C, and the action is executed after action C D, then in the case that action C is the same as action A, compare action D after action C with action B after action A to determine whether the order is wrong.

In order to be able to promptly remind the product manufacturer to discover production problems, the method for generating traceable production data provided by the embodiments of the present disclosure can also send the first result including the abnormal action in the execution action sequence to the relevant target equipment, as well as the problem including the abnormal action The second result of type; thus, the product producer can use the target device to view the above results.

Wherein, the target device may output one or more of the above results, and the specific output form may be display, broadcast, or other forms.

Considering that different execution actions may affect different products in different degrees, for example, the simple execution action of screwing has little effect on some products, but for some precision instruments, it may cause the entire instrument to fail. Scrap; therefore, in the case of determining that there is a problem with the execution action of the target object, based on the importance level of each execution action contained in the template library, the importance level of the abnormal action that has the problem can be determined, and then combined with the importance level. The above-mentioned first result and second result are output. For example, it is possible to give priority to execution action issues with a higher importance level and broadcast them in a strong reminder manner. Another example is to broadcast important ones and display non-important ones. Another example is to use different colors according to the importance level. Displayed to differentiate grades.

In addition, the generation method provided by the embodiment of the present disclosure can also obtain the equipment identification (IDentity, ID) of the production equipment corresponding to the operation area, and then use the target equipment to combine the equipment identification to output the above results.

In a specific application, when the detection results show wrong, missed, over done, or wrong sequence, a production error alarm event can be generated, which includes the wrong action problem type (wrongly done, missed, over done, wrong sequence), action Importance level and corresponding device ID.

Among them, the target device in the embodiment of the present disclosure can be a user terminal, a front-end display interface (large screen), an audible and visual alarm, etc., so that product managers can find production problems in time, and it can also be an industrial system interface, such as data collection and monitoring Control (Supervisory Control And Data Acquisition, SCADA) system or Programmable Logic Controller (Programmable Logic Controller, PLC) to perform linkage operations based on adverse events, etc., can also be other associated devices, and will not be described here.

Those skilled in the art can understand that in the above method of specific implementation, the writing order of each step does not mean a strict execution order and constitutes any limitation on the implementation process. The specific execution order of each step should be based on its function and possible The inner logic is OK.

Based on the same inventive concept, the embodiments of the present disclosure also provide a traceable production data generation device corresponding to the generation method of the traceable production data. Since the problem-solving principle of the device in the embodiments of the present disclosure is the same as that described above in the embodiments of the present disclosure, The generation method of traceable production data is similar, so the implementation of the device can refer to the implementation of the method, and the repetition will not be repeated.

Referring to FIG. 2, it is a schematic diagram of a generation device for traceable production data provided by an embodiment of the present disclosure. The device includes: an acquisition module 201, a determination module 202, and a generation module 203; wherein,

The obtaining module 201 is configured to obtain video data corresponding to the work area collected by the camera equipment;

The determination module 202 is configured to obtain abnormal production data based on the video data, and the abnormal production data is used to reflect the abnormal actions performed by the target object in the operation area for the target product;

The generation module 203 is configured to correlate the quality inspection results and abnormal production data of the target product to obtain traceable production data for product quality traceability.

Using the above-mentioned generation device for traceable production data, the abnormal production data made by the target object in the operation area for the target product can be determined based on the acquired video data. The abnormal production data here can include abnormal actions and abnormal actions. In this way, in the case of associating the quality inspection results of the target product with abnormal production data, the traceable production data of product quality traceability can be determined. It can be seen that the present disclosure can determine abnormal production data through video analysis, and based on this abnormal production data, product quality can be traced. Especially in the case of product defects, targeted solutions can be provided through abnormal production data to improve Rate of qualified products.

In some embodiments of the present disclosure, the determining module 202 is configured to obtain abnormal production data based on video data according to the following steps:

Identify the sequence of execution actions corresponding to the target object in the video data;

Obtain the reference action sequence corresponding to the operation area;

In some embodiments of the present disclosure, the determination module 202 is configured to obtain the abnormal action in the execution action sequence and the problem type of the abnormal action by comparing the execution action sequence with the reference action sequence according to the following steps:

Determining that the abnormal action includes at least one of the actions added in the action sequence, the missing action, and the action in the same order and the similarity of the action content is less than the threshold compared with the reference action sequence. The problem type of the abnormal action includes action abnormal.

Determining the abnormal actions includes actions in the wrong execution sequence in the execution action sequence compared with the reference action sequence, and the problem type of the abnormal actions includes step exceptions.

In some embodiments of the present disclosure, the determining module 202 is configured to perform the actions in the action sequence that are performed in the wrong order compared with the reference action sequence according to the following steps:

Taking each action in the execution action sequence as the first target action, determining the first order of the first target action in the execution action sequence, and determining the first reference action corresponding to the first order from the reference action sequence;

In the case that the first target action does not match the first reference action, it is determined that the first target action is an incorrectly executed action in the execution action sequence.

In some embodiments of the present disclosure, the determination module 202 is configured to perform the actions in the action sequence that are performed in the wrong order compared with the reference action sequence according to the following steps:

Obtaining a second target action and a second reference action in a second order from the execution action sequence and the reference action sequence respectively according to the arrangement order of the actions in the execution action sequence;

In the case that the second target action does not match the second reference action, determining that the second target action is an incorrectly executed action in the execution action sequence;

In the case that the second order is the first step in the action sequence, take the next step in the second order as the second order and repeat the above steps until all actions in the action sequence are traversed; or, in the case where the second order is In the case of executing the last step in the action sequence, take the previous step of the second order as the second order and repeat the above steps until all actions in the action sequence are traversed.

In some embodiments of the present disclosure, the above-mentioned device also includes:

The sending module 204 is configured to send the first result including the abnormal action in the execution action sequence to the target device after obtaining the abnormal action in the execution action sequence and the problem type of the abnormal action by comparing the execution action sequence with the reference action sequence, and a second result of the question type containing abnormal actions, causing the target device to output at least one of the first result and the second result.

In some embodiments of the present disclosure, the sending module 204 is configured to send to the target device the first result containing the abnormal action in the action sequence and the second result containing the question type of the abnormal action to the target device according to the following steps:

In the case of determining that there are multiple abnormal actions in the execution action sequence, determine the importance level of each abnormal action;

Send to the target device the first result containing each abnormal action in the action sequence, the second result including the problem type of each abnormal action, and the importance level of each abnormal action, so that the target device will act on each abnormal action according to the importance level and the problem type of each abnormal action to display and/or broadcast.

Send the first result containing the abnormal action in the action sequence, the second result containing the question type of the abnormal action, and the device identification to the target device, so that the target device can display and/or broadcast.

The response module is configured to, in response to receiving the product quality traceability request, obtain traceable production data associated with the target product indicated by the product quality traceability request; Identify target anomalous production data that matches target detection results.

For the description of the processing flow of each module in the device and the interaction flow between the modules, please refer to the relevant description in the above method embodiment, and will not be described in detail here.

An embodiment of the present disclosure also provides an electronic device, as shown in FIG. 3 , which is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure, including: a processor 301 , a memory 302 , and a bus 303 . The memory 302 stores machine-readable instructions executable by the processor 301 (for example, execution instructions corresponding to the acquisition module 201, the determination module 202, and the generation module 203 in the device in FIG. 2 ), and when the electronic device is running, the processor 301 Communicate with the memory 302 through the bus 303, and when the machine-readable instructions are executed by the processor 301, the following processing is performed:

The abnormal production data is obtained based on the video data, and the abnormal production data is used to reflect the abnormal actions performed by the target object in the operation area for the target product;

Correlate the quality inspection results and abnormal production data of the target product to obtain traceable production data for product quality traceability.

An embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, the method for generating traceable production data described in the above-mentioned method embodiments is executed. . Wherein, the storage medium may be a volatile or non-volatile computer-readable storage medium.

An embodiment of the present disclosure also provides a computer program product, the computer program product carries a program code, and the instructions included in the program code can be used to execute the method for generating traceable production data described in the method embodiment above. For details, please refer to The foregoing method embodiments are not described in detail here.

Wherein, the above-mentioned computer program product may be specifically implemented by means of hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) etc. Wait.

Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-described system and device can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make an electronic device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

Finally, it should be noted that: the above-mentioned embodiments are only specific implementations of the present disclosure, and are used to illustrate the technical solutions of the present disclosure, rather than limit them, and the protection scope of the present disclosure is not limited thereto, although referring to the aforementioned The embodiments have described the present disclosure in detail, and those skilled in the art should understand that any person familiar with the technical field can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present disclosure Changes can be easily imagined, or equivalent replacements can be made to some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in this disclosure. within the scope of protection. Therefore, the protection scope of the present disclosure should be defined by the protection scope of the claims.

Industrial Applicability

The present disclosure relates to a method, device, electronic device, computer readable storage medium and computer program product for generating traceable production data. The method includes: acquiring the video data corresponding to the work area collected by the camera equipment; based on the video data Obtaining abnormal production data, the abnormal production data is used to reflect the abnormal actions performed by the target object in the operation area for the target product; associating the quality inspection results of the target product with the abnormal production data to obtain the Traceable production data for quality traceability. Therefore, the present disclosure can determine abnormal production data through video analysis, and based on this abnormal production data, product quality can be traced. Especially in the case of product defects, targeted solutions can be provided through abnormal production data, so as to Improve product qualification rate.

Claims

A method for generating traceable production data, the method comprising:

Obtain the video data corresponding to the operation area collected by the camera equipment;

Obtaining abnormal production data based on the video data, the abnormal production data is used to reflect the abnormal actions performed by the target object in the operation area on the target product;

Associating the quality inspection results of the target product with the abnormal production data to obtain traceable production data for product quality traceability.
The generating method according to claim 1, wherein said obtaining abnormal production data based on said video data comprises:

identifying an execution action sequence corresponding to the target object in the video data;

Obtaining a reference action sequence corresponding to the operation area;

By comparing the execution action sequence with the reference action sequence, the abnormal actions in the execution action sequence and the problem types of the abnormal actions are obtained.
The generation method according to claim 1 or 2, wherein, by comparing the execution action sequence with the reference action sequence, the abnormal actions in the execution action sequence and the problem types of the abnormal actions are obtained, include:

It is determined that the abnormal actions include at least one of actions added, missing actions in the execution action sequence, and actions with the same execution order and action content similarity less than a threshold value compared with the reference action sequence, The problem types that describe abnormal actions include abnormal actions.
The generation method according to any one of claims 1 to 3, wherein, by comparing the execution action sequence with the reference action sequence, the abnormal action in the execution action sequence and the abnormal action are obtained types of questions, including:

It is determined that the abnormal action includes an action whose execution sequence is wrong in the execution action sequence compared with the reference action sequence, and the problem type of the abnormal action includes a step exception.
The generation method according to claim 4, wherein, compared with the reference action sequence, the actions in the executed action sequence that are performed in the wrong order are determined according to the following steps:

taking each action in the execution action sequence as a first target action, determining the first order of the first target action in the execution action sequence, and determining the first target action from the reference action sequence The first reference action corresponding to the sequence;

In a case where the first target action does not match the first reference action, it is determined that the first target action is an incorrectly executed action in the execution action sequence.
The generation method according to claim 4 or 5, wherein, compared with the reference action sequence, the actions in the execution action sequence that are performed in the wrong order are determined according to the following steps:

Obtaining a second target action and a second reference action in a second order from the execution action sequence and the reference action sequence according to the arrangement order of the actions in the execution action sequence;

In a case where the second target action does not match the second reference action, determining that the second target action is an incorrectly executed action in the execution action sequence;

In the case that the second order is the first step in the execution action sequence, take the next step in the second order as the second order and repeat the above steps until the steps in the execution action sequence are traversed. All actions; or, in the case where the second order is the last step in the execution action sequence, take the previous step in the second order as the second order and repeat the above steps until the execution is traversed All actions in an action sequence.
The generation method according to any one of claims 2 to 6, wherein by comparing the execution action sequence with the reference action sequence, the abnormal actions in the execution action sequence and the problems of the abnormal actions are obtained After the type, the method also includes:

sending to the target device a first result containing the abnormal action in the execution action sequence, and a second result containing the question type of the abnormal action, so that the target device outputs the first result and the second result At least one of the .
The generating method according to claim 7, wherein the method further comprises:

In the case of determining that there are multiple abnormal actions in the execution action sequence, determining the importance level of each abnormal action;

The sending the first result containing the abnormal action in the execution action sequence to the target device, and the second result containing the question type of the abnormal action includes:

sending to the target device a first result containing each abnormal action in the execution action sequence, a second result containing the question type of each abnormal action, and the importance level of each abnormal action, so that the target device Display and/or broadcast the abnormal actions and the problem types of the abnormal actions according to the importance levels.
The generating method according to claim 7 or 8, wherein the sending to the target device the first result containing the abnormal action in the execution action sequence, and the second result containing the question type of the abnormal action, comprising :

Obtain the equipment identification of the production equipment corresponding to the operation area;

Sending the first result including the abnormal action in the execution action sequence, the second result including the question type of the abnormal action, and the device identification to the target device, so that the target device can display and/or broadcast.
The generation method according to any one of claims 1 to 9, wherein the method further comprises:

In response to receiving the product quality traceability request, acquire the traceable production data associated with the target product indicated by the product quality traceability request;

Based on the target detection result in the quality detection results of the target product, the target abnormal production data matching the target detection result is determined from the traceable production data.
A device for generating traceable production data, the device comprising:

The obtaining module is configured to obtain video data corresponding to the operation area collected by the camera equipment;

The determination module is configured to obtain abnormal production data based on the video data, and the abnormal production data is used to reflect the abnormal actions performed by the target object in the operation area on the target product;

The generation module is configured to correlate the quality inspection results of the target product with the abnormal production data to obtain traceable production data for product quality traceability.
The generating device according to claim 11, wherein the determination module 202 is configured to obtain abnormal production data based on video data according to the following steps:

Identify the sequence of execution actions corresponding to the target object in the video data;

Obtain the reference action sequence corresponding to the operation area;

By comparing the execution action sequence with the reference action sequence, the abnormal actions in the execution action sequence and the problem types of the abnormal actions are obtained.
The generating device according to claim 11 or 12, wherein the determination module is configured to obtain the abnormal actions in the execution action sequence and the problem types of the abnormal actions by comparing the execution action sequence with the reference action sequence according to the following steps:

Determining that the abnormal action includes at least one of the actions added in the action sequence, the missing action, and the action in the same order and the similarity of the action content is less than the threshold compared with the reference action sequence. The problem type of the abnormal action includes action abnormal.
The generating device according to any one of claims 11 to 13, wherein the determination module is configured to obtain the abnormal action in the execution action sequence and the information of the abnormal action by comparing the execution action sequence with the reference action sequence according to the following steps: question type:

Determining the abnormal actions includes actions in the wrong execution sequence in the execution action sequence compared with the reference action sequence, and the problem type of the abnormal actions includes step exceptions.
The generating device according to claim 14, wherein the determination module is configured to perform actions in the action sequence that are performed in the wrong order compared with the reference action sequence according to the following steps:

Taking each action in the execution action sequence as the first target action, determining the first order of the first target action in the execution action sequence, and determining the first reference action corresponding to the first order from the reference action sequence;

In the case that the first target action does not match the first reference action, it is determined that the first target action is an incorrectly executed action in the execution action sequence.
The generating device according to claim 14 or 15, wherein the determination module is configured to perform actions in the action sequence that are performed in the wrong order compared with the reference action sequence according to the following steps:

Obtaining a second target action and a second reference action in a second order from the execution action sequence and the reference action sequence respectively according to the arrangement order of the actions in the execution action sequence;

In the case that the second target action does not match the second reference action, determining that the second target action is an incorrectly executed action in the execution action sequence;

In the case that the second order is the first step in the action sequence, take the next step in the second order as the second order and repeat the above steps until all actions in the action sequence are traversed; or, in the case where the second order is In the case of executing the last step in the action sequence, take the previous step of the second order as the second order and repeat the above steps until all actions in the action sequence are traversed.
The generating device according to any one of claims 12 to 16, wherein the device further comprises: a sending module;

The sending module is configured to send the first result including the abnormal action in the execution action sequence to the target device after obtaining the abnormal action in the execution action sequence and the problem type of the abnormal action by comparing the execution action sequence with the reference action sequence , and the second result of the question type containing the abnormal action, causing the target device to output at least one of the first result and the second result.
The generating device according to claim 17, wherein the sending module is configured to send the first result including the abnormal action in the action sequence to the target device and the second result including the question type of the abnormal action to the target device according to the following steps:

In the case of determining that there are multiple abnormal actions in the execution action sequence, determine the importance level of each abnormal action;

sending to the target device the first result containing each abnormal action in the action sequence, the second result containing the question type of each abnormal action, and the importance level of each abnormal action, so that the target device evaluates each abnormal action according to the importance level The abnormal actions and the problem types of each abnormal action are displayed and/or broadcast.
The generating device according to claim 17 or 18, wherein the sending module is configured to send to the target device the first result including the abnormal action in the execution action sequence and the second result including the question type of the abnormal action to the target device according to the following steps :

Obtain the equipment identification of the production equipment corresponding to the operation area;

Send the first result containing the abnormal action in the action sequence, the second result containing the question type of the abnormal action, and the device identification to the target device, so that the target device can display and/or broadcast.
The generating device according to any one of claims 11 to 19, wherein the device further comprises: a response module;

The response module is configured to, in response to receiving the product quality traceability request, obtain traceable production data associated with the target product indicated by the product quality traceability request; The abnormal production data of the target that matches the target detection result is determined in the data.
An electronic device, comprising: a processor, a memory, and a bus, the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory through the bus The method for generating traceable production data according to any one of claims 1 to 10 is executed when the machine-readable instructions are executed by the processor.
A computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, the method for generating traceable production data according to any one of claims 1 to 10 is executed.
A computer program product, comprising computer readable code, when the computer readable code is run in an electronic device, a processor in the electronic device executes the traceable source code according to any one of claims 1 to 10 The method of production data generation.