TW202219671A - Product line monitoring method and monitoring system thereof - Google Patents

Abstract

The present disclosure provides a product line monitoring method and a monitoring system thereof. The monitoring system is configured to: obtain a plurality of images of an operator; based on an image recognition model, determine a motion type of the operation in the images; determine a time of occurrence and a motion period of the operator motion; and record the time of occurrence and the motion period of the operator motion.

Description

Production line monitoring method and monitoring system

The present invention relates to a production line monitoring method and its monitoring system, more particularly, to a production line monitoring method and monitoring system based on machine learning technology.

In the production process of traditional industrial products, the assembly of parts of various devices still requires human assistance. Specifically, a single device usually needs to be equipped with many parts, and the assembly of different parts on the device is usually done manually by operators on each platform of the production line of the factory.

However, errors in manual operation or delays caused by various factors often cause bottlenecks in the output of the production line. Therefore, a monitoring device is required on the production line to record and determine the cause of the output bottleneck of the production line, so as to facilitate the improvement of subsequent efficiency.

However, most of the conventional monitoring devices only have the function of image recording. Therefore, when the situation on the production line is known, it is generally necessary to manually search the image for recording the image of the production line and determine the factors of error or delay. .

Some embodiments of the present invention provide a production line monitoring method for a monitoring system, including: acquiring a plurality of images of an operator; judging an action type of the operator in the plurality of images based on an image recognition model; determining the occurrence time and action of the action type period; and record the type of action, the time of occurrence, and the action period.

Some embodiments of the present invention provide a production line monitoring method for a monitoring system, comprising: acquiring video information, wherein the video information includes a plurality of video video clips; judging the action type of each video clip based on an image recognition model; receiving user settings to change an action type of the first video segment of the plurality of video segments; and adjusting the image recognition model according to the action type of the first video segment.

Some embodiments of the present invention provide a monitoring system for production line monitoring, including a processor and a storage unit. The storage unit stores the program and the image recognition model. When the program is executed, the processor is made to: acquire the multiple images of the operator; determine the action type of the operator in the multiple images based on the image recognition model; determine the occurrence time and action cycle of the action type; and record the action type, occurrence time and action cycle .

The foregoing has outlined rather broadly the features and technical advantages of the present invention in order that the following embodiments of the invention may be better understood. Additional features and advantages of the invention will be described hereinafter that form the subject of the patentable scope of the invention. It should be appreciated by those skilled in the art that the conception and specific embodiment disclosed may be readily utilized as a basis for modifying or designing other structures or processes for carrying out the same purposes of the present invention. Those skilled in the art should also realize that such equivalent constructions do not depart from the spirit and scope of the invention as set forth in the appended claims.

Specific language has now been used to describe the embodiments or examples of the invention illustrated in the accompanying drawings. It should be understood that no limitation of the scope of the invention is intended herein. Any alterations or modifications of the described embodiments and any further application of the principles described in this document are considered to be common occurrences to those of ordinary skill in the art to which the invention pertains. Reference numerals may be repeated throughout the embodiments, but this does not necessarily imply that one or more features of one embodiment are applicable to another embodiment, even if such embodiments share the same reference numerals.

It will be understood that, although the terms first, second, third, etc. may be used herein to describe various elements, components, regions, layers or sections, these elements, components, regions, layers or sections are not limited by these terms. Rather, these terms are only used to distinguish one element, component, region, layer or section from another element, component, region, layer or section. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, layer or section without departing from the teachings of the present inventive concept.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to limit the inventive concept. As used herein, the singular forms "a/an" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should be further understood that, when used in this specification, the terms "comprises and comprising" indicate the presence of stated features, integers, steps, operations, elements or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof.

Errors caused by manual operations on the production line or delays caused by other factors often lead to bottlenecks in the output of the production line. However, the conventional monitoring devices on the production line are only simple image recordings. Therefore, it is still necessary to manually search the images to find out errors or delays. The efficiency and flexibility of this method of debugging are very low, and it cannot effectively improve the output bottleneck of the production line. Accordingly, in order to find out the factors that cause errors or delays on the production line more quickly and accurately, and thus improve the output efficiency of the production line, innovative monitoring methods and monitoring systems are required.

Please refer to FIG. 1A , which is a block diagram of a monitoring system 1 according to some embodiments of the present invention. The monitoring system 1 includes a processor 11 and a storage unit 13 . The storage unit 13 stores the program 130 and the image recognition model 132 . The image recognition model 132 may include models related to machine learning technology. Further, the image recognition model 132 is mainly a machine learning model generated by using complex training data according to a machine learning algorithm.

Specifically, in some embodiments, some image data and the action types actually corresponding to the image data can be used as training data for training the image recognition model 132 (ie, generating the image recognition model 132 ) based on a machine learning algorithm. ). In this way, the image recognition model 132 can be used to receive the image data and output the action type of the operator in the image. For example, after receiving the image sequence of the operator, the image recognition model 132 determines that the operator is performing an action of "pick up" or "put down", and outputs the action type as "pick up" or "put down".

The processor 11 and the storage unit 13 are electrically connected via a communication bus 17 . Through the communication bus 17 , the processor 11 can execute the program 130 stored in the storage unit 13 . When the program 130 is executed, one or more interrupts, such as software interrupts, can be generated, so that the processor 11 can execute the program 130 with the production line monitoring function. The function of routine 130 is further described below.

Please refer to FIG. 1B , which is a schematic diagram of the use of the monitoring system 1 according to some embodiments of the present invention. Specifically, when the operation of the production line machine 92 needs to be monitored and analyzed, the image capture device 91 can be installed in the environment where the production line machine 92 is located to capture images related to the production line machine 92 . The monitoring system 1 can be connected to the image capturing device 91 through a network connection (wired network or wireless network).

In some embodiments, when the operator 93 operates on the production line machine 92, the image capture device 91 can capture the plurality of images 910 of the operator 93 according to the position of the production line machine 92, and record the plurality of images through the network. The image 910 is transmitted to the monitoring system 1 . In other words, the monitoring system 1 can acquire the plurality of images 910 of the operator 93 from the image capturing device 91 through the network.

Then, using the aforementioned image recognition model 132 generated and stored in the storage unit 13 , the processor 11 of the monitoring system 1 can determine the action type of the operator 93 of the plurality of images 910 . Wherein, since the plurality of images 910 carry relevant information of timestamps, the processor 11 can determine the capture time of the plurality of images 910, and further determine the occurrence time and the action period of the action type represented by the plurality of images 910 . The processor 11 can record the action type, occurrence time and action period in the storage unit 13 for subsequent use.

Please refer to FIG. 2A , which is a block diagram of a monitoring system 2 according to some embodiments of the present invention. The monitoring system 2 includes a processor 21 , a storage unit 23 and an input device 25 . The storage unit 23 stores the program 230 , the image recognition model 232 and the training data 234 . The image recognition model 232 may include a model related to machine learning technology for receiving video data (ie, image sequence data) and outputting the action type of the operator in the video.

The processor 21 , the storage unit 23 and the input device 25 are electrically connected via a communication bus 27 . Through the communication bus 27 , the processor 21 can execute the program 230 stored in the storage unit 23 . When the program 230 is executed, one or more interrupts, such as software interrupts, can be generated, so that the processor 21 can execute the program 230 with the function of monitoring the production line. The function of routine 230 is further described below.

In some embodiments, the image recognition model 232 is mainly a machine learning model generated using the complex training data 234 according to a machine learning algorithm. Specifically, some video data and the action types actually corresponding to the video data can be used as training data for training the image recognition model 232 (ie, generating the image recognition model 232 ) based on a machine learning algorithm.

More specifically, each training data 234 may include: (1) video data; and (2) an action type corresponding to the video data, and when the program 230 is executed, the processor 21 retrieves from the storage unit 23 The training data 234 is used to train the image recognition model 232 according to the complex training data 234 by using a machine learning algorithm.

In other words, the video data of the complex training data 234 may be used as training input data during the training phase, and the motion types of the complex training data 234 may be used as the training output data during the training phase. After the processor 21 generates the image recognition model 232, the image recognition model 232 can be stored in the storage unit 23 for subsequent use.

It should be noted that, in some embodiments, the machine learning algorithm mainly introduces a Convolutional Neural Network (CNN) algorithm to construct an image recognition model 232 for judging an action type based on the training data 234 . In some examples, the CNN algorithm may include image processing and image recognition algorithms such as YOLO (you only look once) algorithm, R3D (ResNet 3D) algorithm, etc., but it is not intended to limit the state of the machine learning algorithm in the present invention. Sample.

In some embodiments, in the code used to train the CNN algorithm of the image recognition model 232, there is a training function for training the image recognition model 232. During training of image recognition model 232 , the training function may include a portion for receiving training data 234 .

Further, video data can be used as training input data, and action types corresponding to the video data can be used as training output data. Next, the training function can be executed after executing the main function of the code of the CNN algorithm to train the image recognition model 232 . After generating the image recognition model 232 based on the CNN algorithm and using the training data, the image recognition model 232 can be used to determine the action type corresponding to the input video.

Please refer to FIG. 2B , which is a schematic diagram of the use of the monitoring system 2 according to some embodiments of the present invention. Specifically, when the operation of the production line machine 82 needs to be monitored and analyzed, the image capture device 81 can be installed in the environment where the production line machine 82 is located to capture video related to the production line machine 82 . The monitoring system 2 can be connected to the image capturing device 81 through a network connection (wired network or wireless network).

In some embodiments, when the operator 83 operates on the production line machine 82 , the image capture device 81 can capture the video 810 of the operator 83 in real-time according to the position of the production line machine 82 (eg, : video streaming), and transmit the video 810 to the monitoring system 2 through the network. In other words, the monitoring system 2 can acquire the video 810 of the operator 83 from the image capturing device 81 through the network.

In some embodiments, in order to increase the conversion accuracy of the image recognition model 232 , the video captured on site by the production machine 82 can be used as feedback data to adjust the image recognition model 232 . Specifically, the video 810 may include a plurality of video segments. Using the aforementioned image recognition model 232 generated and stored in the storage unit 23, the processor 21 of the monitoring system 2 can determine the action type of the operator 83 for each video segment.

After the processor 21 uses the image recognition model 232 to determine the action type of the operator 83 of each video clip of the video 810, the monitoring system 2 can provide the video clip and the corresponding action type to the user, so that the user can determine whether There is a conversion bias of the image recognition model 232. In some embodiments, the monitoring system 2 may provide video clips and corresponding action types to the user through a display (not shown) and a graphical user interface (GUI).

Then, if the user determines that the specific video segment and its corresponding action type are the conversion errors of the image recognition model 232, the user can input the user settings through the input device 25, so as to change the action type of the specific video segment to the correct one action.

Then, the processor 21 can update the training data 234 using the specific video segment and the corrected motion type, and reuse the updated complex training data 234 to generate the image recognition model 232 . More specifically, the processor 21 can generate the image recognition model 234 based on a machine learning algorithm by using the original training data 234, at least one specific video segment, and at least one action type corresponding to the at least one specific video segment.

In this way, since the training data re-used by the image recognition model 232 includes the relevant data for the production line machine 82 and the operator 83 (ie, at least one specific video segment and at least one corresponding to the at least one specific video segment) action type), the updated image recognition model 232 will have higher conversion accuracy when applied to the environment of the production line machine 82 .

The technique of adjusting the image recognition model 232 using the video captured by the production machine 82 on-site as feedback data can be better understood through the following examples. For example, the video 810 includes ten video segments "C1-C10". Using the aforementioned image recognition model 232 generated and stored in the storage unit 23, the processor 21 of the monitoring system 2 can determine each of the video segments "C1-C10". The action type of the operator 83 of a video clip (eg "pick up" action or "drop down" action).

After the processor 21 uses the image recognition model 232 to determine the action types of the video clips "C1-C10", the monitoring system 2 provides the video clips "C1-C10" and their corresponding action types for use through the display and GUI That is, it is helpful for the user to judge whether there is a conversion error of the image recognition model 232 .

In this example, the action types of the video clips "C1" and "C8" are determined by the monitoring system 2 as "pick-up action" and "drop action", respectively. However, the user judges that the action types of the video clips "C1" and "C8" should be "drop action" and "pick up action" respectively. Therefore, the user inputs the user settings through the input device 25, and sets the video clip "C1" respectively. And the action type of "C8" is corrected to "drop action" and "pick up action". Then, the processor 21 updates the training data 234 with the video clips " C1 " and " C8 " and the corrected action type, and reuses the updated complex training data 234 to generate the image recognition model 232 .

In some embodiments, after the image recognition model 232 is updated through the aforementioned steps, when the operator 83 continues to operate on the production line machine 82 , the image capture device 81 can capture the position of the production line machine 82 to capture the operator 83 . and transmit the video 812 to the monitoring system 2 through the network. In other words, the monitoring system 2 can acquire the video 812 of the operator 83 from the image capturing device 81 through the network. The video 812 includes a plurality of video segments.

Then, using the image recognition model 232 that has been updated and stored in the storage unit 23 , the processor 21 of the monitoring system 2 can determine the action type of each video segment of the video 812 . Wherein, since each video clip carries relevant information of a time stamp, the processor 21 can determine the capture time of each video clip, and further determine the occurrence time and action cycle of the action type represented by each video clip. The processor 21 can record the action type and action period in the storage unit 23 for subsequent use.

In some embodiments, the processor 21 may determine whether the action period of the corresponding action type exceeds the period threshold for each video segment stored in the storage unit 23 . If so, mark the action type and the corresponding video clip, and record the action type, occurrence time and occurrence cycle corresponding to the video clip in the record file. In this way, the user can use the log file to efficiently call up the marked video clips in the video 812, and further understand the reason why the action cycle of the action types in these video clips exceeds the cycle threshold, so as to quickly Exclude factors causing delays.

For example, it is assumed that the "pick-up" action should be completed within 3 seconds, and the processor 21 determines whether the action period exceeds the 3-second value for all video clips corresponding to the "pick-up" action. If so, mark the action type and the corresponding video clip, and record the action type, occurrence time and occurrence cycle corresponding to the video clip in the record file. Then the user can use the log file to efficiently call up the marked video clips in the video 812, and further understand the reason why the action period of the action types in these video clips exceeds 3 seconds, so as to quickly eliminate the delay caused by the video clip. factor.

In some embodiments, the processor 21 may determine whether the time difference between the occurrence times of the corresponding two action types exceeds a time threshold for all the two consecutive video segments stored in the storage unit 23 . If so, mark the two action types and the corresponding two video clips, and record the corresponding action type, occurrence time and occurrence cycle of the two video clips in the record file. In this way, the user can use the record file to efficiently call up the two marked video clips in the video 812, and further understand the reason why the time difference between the occurrence times of the corresponding two action types exceeds the time threshold, so as to quickly factors that cause delays are excluded.

For example, it is assumed that between the "pick-up" action and the "pick-up" action that occurs continuously, the related part configuration operation should be completed within 10 seconds, then the processor 21 is directed to the "pick-up" action and the "pock-down" action that occurs continuously. The second video clip of the action determines whether the time difference exceeds 10 seconds. If so, mark the two action types and the corresponding two video clips, and record the corresponding action type, occurrence time and occurrence cycle of the two video clips in the record file. In this way, the user can use the record file to efficiently call up the two marked video clips in the video 812, and further understand the reason why the time difference between the occurrence times of the corresponding two action types exceeds 10 seconds, so as to quickly Exclude factors causing delays.

Please refer to FIG. 2C , which is a schematic diagram of a frame captured by the image capturing device 81 according to some embodiments of the present invention. In some embodiments, since the image or video captured by the image capturing device 81 has a larger range, the processor 21 will spend more hardware resources and time when processing the image or video using the image recognition model 232 .

However, not all the images or videos captured by the image capturing device 81 need to be monitored. Therefore, a smaller area to be monitored can be defined for the captured images or videos, and the processor 21 only needs to monitor a smaller area. In the area of the range, the image recognition model 232 is used for image or video processing, so that the processing speed can be greatly accelerated.

Please refer to FIG. 2D , which is another schematic diagram of a frame captured by the image capturing device 81 according to some embodiments of the present invention. Specifically, the user can input user settings through the input device 25 to define the monitoring area 80A on the image range captured by the image capturing device 81 , and the processor 21 only needs to target the image or video of the monitoring area 80A , using the image recognition model 232 for processing. In this way, since the size of the image or video of the monitoring area 80A is small, the processing speed of the monitoring system 2 can be greatly accelerated.

In some embodiments, when the environment of the production line machine 82 is changed (for example, the angle of the image capture device 81 is adjusted, the operator configuration is changed, the operator position is changed, etc.), it may cause a deviation from the range originally intended to be monitored. The area 80A is monitored and the conversion error of the image recognition model 232 is increased. At this time, the user can directly adjust the position of the monitoring area 80A, so as to reduce the deviation caused by the environmental change of the production line machine 82 site.

Please refer to FIG. 2E , which is another schematic diagram of a frame captured by the image capturing device 81 according to some embodiments of the present invention. Specifically, since the environment of the production line machine 82 changes, the image or video in the monitoring area 80A is not required to be monitored, and therefore, the conversion error of the image recognition model 232 may increase.

Please refer to FIG. 2F , which is another schematic diagram of a frame captured by the image capturing device 81 according to some embodiments of the present invention. Specifically, the user can input another user setting through the input device 25 to move the monitoring area 80A on the image range captured by the image capturing device 81 , so that the area to be monitored returns to normal.

In some embodiments, the images captured by the image capturing device 81 may be sent to the monitoring system 2 first. Then, the monitoring system 2 can display these pictures through a common display (not shown), and receive user settings through the input device 25 such as a keyboard or a mouse, so that the monitoring system 2 can complete the related operations.

In some embodiments, the images captured by the image capturing device 81 may be sent to the monitoring system 2 first. Then, the monitoring system 2 can transmit the pictures to the remote displays (such as handheld smart devices, notebook computers, etc.) through the network, and receive user settings through the input device 25 such as a network interface, so as to facilitate monitoring System 2 completes the relevant operations.

Please refer to FIG. 3A , which is a block diagram of a monitoring system 3 according to some embodiments of the present invention. The monitoring system 3 includes a processor 31 , a storage unit 33 and an input device 35 . The storage unit 33 stores the program 330, the image recognition model 332A, the image recognition model 332B, and the training data 334A and 334B. The image recognition models 332A and 332B may include models related to machine learning technology, which are used to determine the operator's action type or the quantity change of objects in the video data (ie, the image sequence data).

The processor 31 , the storage unit 33 and the input device 35 are electrically connected via the communication bus 37 . Through the communication bus 37 , the processor 31 can execute the program 330 stored in the storage unit 33 . One or more interrupts, such as software interrupts, may be generated when the program 330 is executed, so that the processor 31 can execute the program 330 with the function of monitoring the production line. The function of routine 330 is further described below.

In some embodiments, the image recognition model 332A is primarily a machine learning model generated using the complex training data 334A according to a machine learning algorithm. Specifically, some video data and the action types actually corresponding to the video data can be used as training data for training the image recognition model 332A (ie, generating the image recognition model 332A) based on a machine learning algorithm.

More specifically, each training data 334A may include: (1) video data; and (2) an action type corresponding to the video data, and when the program 330 is executed, the processor 31 retrieves from the storage unit 33 The training data 334A is used to train the image recognition model 332A according to the complex training data 334A using a machine learning algorithm.

In other words, the video data of the complex training data 334A may be used as training input data during the training phase, and the motion types of the complex training data 334A may be used as the training output data during the training phase. After the processor 31 generates the image recognition model 332A, the image recognition model 332A can be stored in the storage unit 33 for subsequent use.

In some embodiments, among the plurality of training data 334A, the video data used as training input data includes video data of the operator's actions. The action types corresponding to the video data can be used as training output data. Next, the code of the CNN algorithm can be executed to train the image recognition model 332A. After generating the image recognition model 332A based on the CNN algorithm and using the training data, the image recognition model 332A can be used to determine the action type corresponding to the input video.

In some embodiments, the image recognition model 332B is mainly a machine learning model generated using the complex training data 334B according to a machine learning algorithm. In detail, some video data and changes in the number of objects actually corresponding to the video data can be used as training data for training the image recognition model 332B (ie, generating the image recognition model 332B) based on a machine learning algorithm.

More specifically, each training data 334B may include: (1) video data; and (2) a change (eg, increase or decrease) in the number of objects corresponding to the video data, and when the program 330 is executed, the processor 31. The training data 334B is retrieved from the storage unit 33, and a machine learning algorithm is used to train the image recognition model 332B according to the complex training data 334B.

In other words, the video data of the complex training data 334B can be used as training input data during the training phase, and the object quantity changes of the complex training data 334B can be used as training output data during the training phase. After the processor 31 generates the image recognition model 332B, the image recognition model 332B can be stored in the storage unit 33 for subsequent use.

In some embodiments, among the plurality of training data 334B, the video data used as training input data includes image data of changes in the number of objects. The change in the number of objects corresponding to the video data can be used as training output data. Next, the code of the CNN algorithm can be executed to train the image recognition model 332B. After generating the image recognition model 332B based on the CNN algorithm and using the training data, the image recognition model 332B can be used to determine the change in the number of objects corresponding to the input video.

More specifically, the video data records the quantity changes of specific objects (eg, product components), and the quantity changes of specific objects can represent different behaviors. For example, when the number of specific objects decreases in the video data, it means that the operator's action is more likely to "pick up" the specific object. When the number of specific objects increases in this video data, it means that the operator's action is more likely to "drop" the specific object. Accordingly, using the change in the number of specific objects in the image data can help improve the accuracy of the action type judgment result.

Please refer to FIG. 3B , which is a schematic diagram of the use of the monitoring system 3 according to some embodiments of the present invention. Specifically, when the operation of the production line machine 72 needs to be monitored and analyzed, the image capture device 71 can be installed in the environment where the production line machine 72 is located to capture video related to the production line machine 72 . The monitoring system 3 can be connected to the image capturing device 71 through a network connection (wired network or wireless network).

In some embodiments, when the operator 73 operates on the production line machine 72 , the image capture device 71 can capture the video 710 (eg, video stream) of the operator 73 in real time according to the position of the production line machine 72 . , and transmit the video 710 to the monitoring system 3 through the network. In other words, the monitoring system 3 can acquire the video 710 of the operator 73 from the image capturing device 71 through the network. The video 710 includes a plurality of video segments.

Next, the user can input user settings through the input device 35 to define the monitoring areas 70A and 70B on the image range captured by the image capturing device 71 , and the processor 31 only needs to target the images of the monitoring areas 70A and 70B Or video, which is processed using the image recognition models 332A and 332B.

Then, using the aforementioned image recognition model 332A stored in the storage unit 33 , the processor 31 of the monitoring system 3 can determine the action types of the monitoring areas 70A and 70B in each video segment of the video 710 . Among them, since each video clip carries relevant information of a time stamp, the processor 31 can determine the capture time of each video clip, and further determine the action types represented by the monitoring areas 70A and 70B in each video clip. Occurrence time and action cycle. The processor 31 can record the action type and action cycle in the storage unit 33 for subsequent use.

In some embodiments, for the monitoring areas 70A and 70B of each video segment, the processor 31 of the monitoring system 3 may further use the image recognition model 332B to determine the change in the number of objects, and update the action type of the operator 73 accordingly. Please refer to FIG. 3C to FIG. 3D , which are schematic diagrams of images captured by the image capturing device 71 according to some embodiments of the present invention. For example, for the monitoring area 70A of a specific video segment, the processor 31 of the monitoring system 3 can first determine that the action type is "pick up" by using the image recognition model 332A.

Next, for the monitoring area 70A of the specific video segment, the processor 31 of the monitoring system 3 may further use the image recognition model 332B to determine that the number of objects 74 in the video segment is reduced. Accordingly, since the action type of the specific video segment in the monitoring area 70A is "pick-up", and the decrease in the number of objects 74 is indeed caused by "pick-up", the specific action type can be accurately confirmed as "pick-up" .

It should be noted that, for the monitoring area 70A of a specific video clip, when the processor 31 of the monitoring system 3 uses the image recognition model 332A to determine that the action type is "put down", but uses the image recognition model 332B to determine that the number of objects 74 in the video clip decreases, it means The judgment of the image recognition model 332A may be wrong. Accordingly, the processor 31 of the monitoring system 3 can update the action type of the corresponding specific video segment from "drop" to "pick up" based on determining that the number of objects 74 in the video segment is reduced by using the image recognition model 332B.

Please refer to FIG. 3E to FIG. 3F , which are schematic diagrams of images captured by the image capturing device 71 according to some embodiments of the present invention. For example, for the monitoring area 70B of a specific video segment, the processor 31 of the monitoring system 3 can first determine that the action type is "put down" by using the image recognition model 332A.

Next, for the monitoring area 70B of the specific video segment, the processor 31 of the monitoring system 3 can further use the image recognition model 332B to determine that the number of objects 74 in the video segment is increased. Accordingly, since the action type of the specific video segment in the monitoring area 70B is "drop", and the increase in the number of objects 74 is indeed caused by "drop", the specific action type can be accurately confirmed as "drop".

Similarly, for the monitoring area 70B of a specific video clip, when the processor 31 of the monitoring system 3 uses the image recognition model 332A to determine that the action type is "pick up", but uses the image recognition model 332B to determine that the number of objects 74 in the video clip increases, It indicates that the judgment of the image recognition model 332A may be wrong. Accordingly, the processor 31 of the monitoring system 3 can update the action type of the corresponding specific video segment from "pick up" to "put down" based on judging by the image recognition model 332B that the number of objects 74 in the video segment increases.

It should be noted that, in the foregoing embodiment, when the processor uses the image recognition model to determine the action type of the image or video data, the image recognition model can be used to identify and track the operator's hand, and further use the image recognition model to identify and track the operator's hand. Action, to determine the action type of the operator of the image or video.

Some embodiments of the present invention include a production line monitoring method, the flowchart of which is shown in FIGS. 4A-4B. The production line monitoring methods of these embodiments are implemented by a monitoring system (such as the monitoring system of the aforementioned embodiments). The detailed operation of the method is as follows.

First, step S401 is executed by the monitoring system to acquire multiple images of the operator. Among them, the monitoring system can acquire multiple images of the operator through the image capturing device installed on the machine of the production line. Step S402 is executed by the monitoring system, and based on the image recognition model, the action type of the operator of the plurality of images is determined. The image recognition model may include a model related to machine learning technology for receiving image data and outputting the action type of the operator in the image.

Next, since the plurality of images may carry relevant information of time stamps, the monitoring system executes step S403 to determine the occurrence time of the action type and the action period according to the action type of the operator of the plurality of images. Step S404 is executed by the monitoring system, and the action type, occurrence time and action period are recorded for subsequent use.

In some embodiments, after step S402, in order to increase the judgment accuracy, the monitoring system may perform step S402', based on another image recognition model, to update the action type according to the change in the number of objects in the plurality of images.

Some embodiments of the present invention include a production line monitoring method, the flow chart of which is shown in FIGS. 5A to 5F . The production line monitoring methods of these embodiments are implemented by a monitoring system (such as the monitoring system of the aforementioned embodiments). The detailed operation of the method is as follows.

In some embodiments, the production line monitoring method needs to provide an image recognition model including machine learning technology for receiving the image data and outputting the operator's action type in the image. Therefore, the training data is used to train and generate the image recognition model.

Please refer to FIG. 5A , which is a flow chart of image recognition model generation of the production line monitoring method according to some embodiments of the present invention. Step S501 is executed by the monitoring system, and based on the machine learning algorithm, the image recognition model is generated by using the complex training data. Wherein, each training data includes training input and training output. The training input includes training video clips, and the training output includes training action types corresponding to the training video clips. Step S502 is executed by the monitoring system to store the image recognition model for subsequent use.

In some embodiments, in order to increase the accuracy of the image recognition model, the video captured on site of the production machine can be used as feedback to adjust the image recognition model. Please refer to FIG. 5B , which is a flowchart of an image recognition model update of a production line monitoring method according to some embodiments of the present invention. Step S503 is executed by the monitoring system to acquire video information. Wherein, the monitoring system can obtain the video of the operator by the image capture device arranged on the machine of the production line, and the video includes a plurality of video segments.

Step S504 is executed by the monitoring system to determine the action type of the operator of each video segment based on the generated image recognition model. Step S505 is executed by the monitoring system, and the video clip and the corresponding action type are provided to the user, so that the user can judge whether there is a conversion error of the image recognition model.

When the user determines that there is a conversion error of the image recognition model, causing a specific video segment and its corresponding action type to be inconsistent, the user can change the specific video segment and its corresponding action type. Step S506 is executed by the monitoring system to receive user settings to change the action type of the video clip.

After all the video clips are judged, the monitoring system executes step S507 to adjust the image recognition model according to the specific video clip and the changed action type. Specifically, the surveillance system generates an image recognition model according to the original training data, a specific video clip, an action type corresponding to the specific video clip, and a machine learning algorithm.

In this way, since the re-training of the image recognition model has relevant information for the production line machine and operator (ie a specific video segment and the action type corresponding to the specific video segment), the updated image recognition model is applied to the production line. The machine's judgment will have higher accuracy.

In some embodiments, the production line machine status can be monitored based on the updated image recognition model. Please refer to FIG. 5C , which is a flowchart of a production line monitoring method according to some embodiments of the present invention. Step S508 is executed by the monitoring system to obtain the video of the operator of the production line machine. Wherein, the video includes a plurality of video segments. Step S509 is executed by the monitoring system to determine the action type of the operator of each video segment based on the image recognition model.

Next, since the plurality of video clips may carry relevant information of time stamps, the monitoring system executes step S510 to determine the occurrence time and the action period of the action type according to the action type of the operator of each video clip. Step S511 is executed by the monitoring system, and the action type, occurrence time and action period are recorded for subsequent use.

In some embodiments, the monitoring system can determine whether the action of the video is delayed. Please refer to FIG. 5D , which is a flowchart of a production line monitoring method according to some embodiments of the present invention. Step S512 is executed by the monitoring system to determine whether the action period exceeds the period threshold value according to the action type of the operator of each video segment. If so, the monitoring system executes step S513, marking the action type and the corresponding video clip, and recording the action type, occurrence time and occurrence cycle corresponding to the video clip in the record file, so that the user can use the record file to efficiently record in the record file. The marked video clip is called up in the video. If not, the monitoring system repeatedly executes step S512 to determine whether the action period exceeds the period threshold for the action type of the operator of the next video segment.

In some embodiments, the monitoring system can determine whether a delay occurs between actions of the video. Please refer to FIG. 5E , which is a flowchart of a production line monitoring method according to some embodiments of the present invention. Step S514 is executed by the monitoring system to calculate the time difference between the occurrence times of the action types of the two video segments. Step S515 is executed by the monitoring system to determine whether the time difference exceeds the time threshold. If so, the monitoring system executes step S516, marking the action type and the corresponding two video clips, and recording the action type, occurrence time and occurrence cycle corresponding to the two video clips in the log file, so that the user can use the log file efficiently. The marked video clip is called up in the video. If not, the monitoring system executes step S514 repeatedly, calculates the time difference between the occurrence times of the corresponding action types for the next pair of video clips, and determines whether the action period exceeds the period threshold.

In some embodiments, the following steps can be selectively added to accelerate the speed and efficiency of image processing. Please refer to FIG. 5F , which is a flowchart of a production line monitoring method according to some embodiments of the present invention. Step S517 is executed by the monitoring system to receive user settings for defining a monitoring area on the video to be captured. In other words, the user setting is used to define the monitoring area on the range of the image captured by the image capturing device. Among them, since the image or video size of the monitoring area is small, the processing speed of the monitoring system can be greatly accelerated.

In some embodiments, the following steps can be selectively added to reduce deviations caused by environmental changes at the production line machine site. Please refer to FIG. 5G , which is a flowchart of a production line monitoring method according to some embodiments of the present invention. Step S518 is executed by the monitoring system to receive user settings for moving the monitoring area. In other words, the user setting is used to move the monitoring area on the range of the image captured by the image capturing device.

The aforementioned monitoring system and production line monitoring method of the present invention can more quickly and accurately find out the factors that cause errors or delays on the production line by means of automation and artificial intelligence, thereby improving the output efficiency of the production line and effectively improving the output of the production line. bottleneck.

It should be particularly understood that the processor mentioned in the above embodiments may be a central processing unit (CPU), other hardware circuit elements capable of executing relevant instructions, or a combination of computing circuits known to those skilled in the art based on the above disclosure. In addition, the storage unit mentioned in the above embodiments may include a memory (such as ROM, RAM, etc.) or a storage device (such as flash memory, HDD, SSD, etc.) for storing data.

Further, the communication bus mentioned in the above embodiments may include a communication interface for transferring data between elements such as a processor, a storage unit, an input device, etc., and may include an electrical bus interface, an optical bus interface, or Even a wireless bus interface. However, such descriptions are not intended to limit the examples of hard implementations of the invention.

Although the present invention and its advantages have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims. For example, many of the processes discussed above can be implemented in different ways and replaced by other processes or combinations thereof.

Furthermore, the scope of this application is not intended to be limited to the specific embodiments of the process, machine, manufacture, composition of matter, means, methods, and steps described in this specification. As will be readily understood by those of ordinary skill in the art from the disclosure of the present invention, the corresponding embodiments described herein may be utilized in accordance with the present invention to perform substantially the same functions as those of the corresponding embodiments described herein. A process, machine, manufacture, composition of matter, component, method or step currently existing or subsequently to be developed that results in substantially the same result. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.

1: Monitoring system 2: Monitoring system 3: Monitoring system 11: Processor 13: Storage unit 17: Communication bus 21: Processor 23: Storage unit 25: Input device 27: Communication bus 31: Processor 33: Storage unit 35: Input device 37: Communication bus 70A: Surveillance area 70B: Surveillance area 71: Image capture device 72: Production line machine 73: Operator 74: Object 80A: Surveillance area 81: Image capture device 82: Production line machine 83: Operator 91: Image capture device 92: Production line machine 93: Operator 130: Program 132: Image recognition model 230: Program 232: Image recognition model 234: Training Materials 330: Program 332A: Image Recognition Model 332B: Image Recognition Model 334: Training Materials 710: Video 810: Video 812: Video 910: Video S401~S404: Steps S501~S518: Steps

Aspects of the present invention are best understood in light of the following embodiments when read in conjunction with the accompanying drawings. It should be noted that in accordance with standard practice in the industry, the various features are not drawn to scale. In fact, the dimensions of the various features may be arbitrarily increased or decreased for clarity of discussion.

A more thorough understanding of the present invention may be obtained by reference to the embodiments and the scope of the claims, when considered in conjunction with the accompanying drawings, wherein like reference numerals refer to like elements throughout.

FIG. 1A is a block diagram of a monitoring system according to some embodiments of the present invention.

FIG. 1B is a schematic diagram of the use of the monitoring system according to some embodiments of the present invention.

2A is a block diagram of a monitoring system according to some embodiments of the present invention.

FIG. 2B is a schematic diagram of the use of the monitoring system according to some embodiments of the present invention.

2C to 2F are schematic diagrams of images captured by the image capturing device according to some embodiments of the present invention.

3A is a block diagram of a monitoring system according to some embodiments of the present invention.

FIG. 3B is a schematic diagram of the use of the monitoring system according to some embodiments of the present invention.

3C to 3F are schematic diagrams of frames captured by the image capturing device according to some embodiments of the present invention.

4A-4B are flowcharts of production line monitoring methods according to some embodiments of the present invention.

5A to 5G are flowcharts of production line monitoring methods according to some embodiments of the present invention.

S401~S404: Steps

Claims (20)

A production line monitoring method for a monitoring system, comprising: Obtain multiple images of an operator; Based on an image recognition model, determine an action type of the operator in the images; determine an occurrence time and an action period of the action type; and Record the action type, the occurrence time and the action period. Such as the production line monitoring method of request item 1, it also includes: Determine whether the action cycle exceeds a cycle threshold; When the action cycle exceeds the cycle threshold, the action type is marked. Such as the production line monitoring method of request item 1, it also includes: Calculate a time difference between the occurrence time of the action type and the occurrence time of another action type; Determine whether the time difference exceeds a time threshold; When the time difference exceeds the time threshold, the action type and the other action type are marked. Such as the production line monitoring method of request item 1, it also includes: receiving a user setting for defining a surveillance area on the images; Wherein, the step of judging the action type of the operator of the images based on the image recognition model further includes: Based on the image recognition model, the action type in the surveillance area of the images is determined. For example, the production line monitoring method of claim 4, further includes: Another user setting is received for moving the surveillance area defined on the images. The production line monitoring method of claim 4, wherein the images have an image size, the monitoring area has an area size, and the monitoring area size is smaller than the image size. Such as the production line monitoring method of request item 1, it also includes: Based on the image recognition model, at least one hand of the operator in the images is identified, and the action type of the at least one hand of the operator is determined. Such as the production line monitoring method of request item 1, it also includes: Based on a Machine Learning algorithm, the image recognition model is generated using plural training data, wherein each training data includes a training input and a training output, the training input includes a plurality of training images, and the training output includes and the The training image corresponds to one of the training action types. Such as the production line monitoring method of request item 1, it also includes: Based on another image recognition model, the action type of the operator of the images is updated according to a change in the number of objects in the images. A production line monitoring method for a monitoring system, comprising: obtaining a video, wherein the video includes a plurality of video segments; determining an action type of each of the video clips based on an image recognition model; receiving a user setting to change the action type for a first video clip of one of the video clips; and The image recognition model is adjusted according to the action type of the first video segment. For example, the production line monitoring method of claim 10 further includes: The image recognition model is generated by using plural training data and a machine learning algorithm, wherein each training data includes a training input and a training output, the training input includes a training video segment, and the training output includes and The training video clip corresponds to a training action type. The production line monitoring method of claim 11, wherein the step of adjusting the image recognition model according to the action type of the first video clip further comprises: The image recognition model is generated using the training data, the first video segment, the action type corresponding to the first video segment, and the machine learning algorithm. A monitoring system for production line monitoring, comprising: a processor; and a storage unit, storing a program and an image recognition model, and when the program is executed, the processor: Obtain multiple images of an operator; based on the image recognition model, determine an action type of the operator in the images; determine an occurrence time and an action period of the action type; and Record the action type, the occurrence time and the action period. The monitoring system of claim 13, wherein the program, when executed, further causes the processor to: Determine whether the action cycle exceeds a cycle threshold; When the action cycle exceeds the cycle threshold, the action type is marked. The monitoring system of claim 13, wherein the program, when executed, further causes the processor to: Calculate a time difference between the occurrence time of the action type and the occurrence time of another action type; Determine whether the time difference exceeds a time threshold; When the time difference exceeds the time threshold, the action type and the other action type are marked. As in the monitoring system of claim 13, it further includes: an input device for receiving a user setting; where the program, when executed, further causes the processor to: define a surveillance area on the images according to the user settings; Based on the image recognition model, the action type in the surveillance area of the images is determined. The monitoring system of claim 16, wherein the input device is further configured to receive settings from another user, the program, when executed, further causes the processor to: Move the surveillance area defined on the images according to the other user settings. The monitoring system of claim 13, wherein the program, when executed, further causes the processor to: Based on a Machine Learning algorithm, the image recognition model is generated using plural training data, wherein each training data includes a training input and a training output, the training input includes a plurality of training images, and the training output includes and the The training image corresponds to one of the training action types. As in the monitoring system of claim 18, it further includes: an input device for receiving a user setting; where the program, when executed, further causes the processor to: The action type of the operator who changes the images according to the user settings; The image recognition model is adjusted according to the action type of the operator of the images. The monitoring system of claim 18, wherein the program, when executed, further causes the processor to: Based on the machine learning algorithm, the image recognition model is generated using the training data, the images, and the action type of the operator corresponding to the images.

Images