TWI839583B - Product line monitoring method and monitoring system thereof - Google Patents

Abstract

The present disclosure provides a product line monitoring method and a monitoring system thereof. The monitoring system is configured to: obtain a plurality of images of an operator; based on an image recognition model, determine a motion type of the operation in the images; determine a time of occurrence and a motion period of the operator motion; and record the time of occurrence and the motion period of the operator motion.

Description

Production line monitoring method and monitoring system

The present invention relates to a production line monitoring method and a monitoring system thereof, and more specifically, to a production line monitoring method and a monitoring system thereof based on machine learning technology.

In the manufacturing process of traditional industrial products, the assembly of parts of various devices still requires manual assistance. Specifically, a single device often needs to be equipped with many parts, and the assembly of different parts on the device is usually completed manually by operators at each station of the factory's production line.

However, errors in manual operation or delays caused by various factors often cause bottlenecks in the production line output. Therefore, monitoring devices are needed on the production line to record and determine the causes of the bottlenecks in the production line output, so as to improve subsequent efficiency.

However, most of the monitoring devices we know of only have the function of image recording. Therefore, when we know something happened on the production line, we generally still need to manually search the images recording the production line and determine the factors of error or delay.

Some embodiments of the present invention provide a production line monitoring method for a monitoring system, comprising: obtaining multiple images of an operator; judging the action type of the operator in the multiple images based on an image recognition model; determining the occurrence time and action cycle of the action type; and recording the action type, occurrence time and action cycle.

Some embodiments of the present invention provide a production line monitoring method for a monitoring system, comprising: obtaining a video, wherein the video comprises a plurality of video segments; determining the action type of each video segment based on an image recognition model; receiving a user setting to change the action type of a first video segment of the plurality of video segments; and adjusting the image recognition model according to the action type of the first video segment.

Some embodiments of the present invention provide a monitoring system for production line monitoring, including a processor and a storage unit. The storage unit stores programs and image recognition models. When the program is executed, the processor: obtains multiple images of the operator; based on the image recognition model, determines the action type of the operator in the multiple images; determines the occurrence time and action cycle of the action type; and records the action type, occurrence time and action cycle.

The above has been a fairly broad overview of the features and technical advantages of the present invention so that the following embodiments of the present invention may be better understood. Additional features and advantages of the present invention are described below and form the subject of the patent claims of the present invention. Those skilled in the art should understand that the disclosed concepts and specific embodiments can be easily used as a basis for modifying or designing other structures or processes for achieving the same purpose of the present invention. Those skilled in the art should also recognize that such equivalent structures do not depart from the spirit and scope of the present invention as set forth in the attached patent claims.

1: Monitoring system

11: Processor

13: Storage unit

130: Program

132: Image recognition model

17: Communication bus

2: Monitoring system

21: Processor

23: Storage unit

230: Program

232: Image recognition model

234: Training information

25: Input device

27: Communication bus

3: Monitoring system

31: Processor

33: Storage unit

330: Program

332A: Image recognition model

332B: Image recognition model

334: Training information

35: Input device

37: Communication bus

70A: Surveillance area

70B: Surveillance area

71: Image capture device

710: Video

72: Production line machine

73: Operator

74: Objects

80A: Surveillance area

81: Image capture device

810: Video

812: Video

82: Production line machine

83: Operator

91: Image capture device

910: Image

92: Production line machine

93: Operator

S401~S404: Steps

S501~S518: Steps

Various aspects of the present invention may be best understood from the following embodiments when read in conjunction with the accompanying drawings. It should be noted that, in accordance with standard practice in the industry, the various features are not drawn to scale. In fact, the sizes of the various features may be arbitrarily increased or decreased for clarity of discussion.

A more thorough understanding of the present invention may be obtained by reference to the embodiments and claims when considered in conjunction with the accompanying drawings, wherein like reference numerals refer to similar elements throughout the accompanying drawings.

FIG. 1A is a block diagram of a monitoring system of some embodiments of the present invention.

FIG1B is a schematic diagram of the use of the monitoring system of some embodiments of the present invention.

FIG. 2A is a block diagram of a monitoring system of some embodiments of the present invention.

FIG2B is a schematic diagram of the use of the monitoring system of some embodiments of the present invention.

Figures 2C to 2F are schematic diagrams of images captured by the image capture device of some embodiments of the present invention.

FIG. 3A is a block diagram of a monitoring system of some embodiments of the present invention.

FIG3B is a schematic diagram of the use of the monitoring system of some embodiments of the present invention.

Figures 3C to 3F are schematic diagrams of images captured by the image capture device of some embodiments of the present invention.

Figures 4A to 4B are flow charts of production line monitoring methods of some embodiments of the present invention.

Figures 5A to 5G are flow charts of production line monitoring methods of some embodiments of the present invention.

Embodiments or examples of the present invention shown in the accompanying drawings are now described using specific language. It should be understood that no limitation of the scope of the present invention is intended herein. Any changes or modifications of the described embodiments and any further application of the principles described in this document are considered to be common occurrences for those of ordinary skill in the art to which the present invention relates. Reference numerals may be repeated throughout the embodiments, but this does not necessarily mean that one or more features of one embodiment are applicable to another embodiment, even if such embodiments share the same reference numerals.

It should be understood that although the terms first, second, third, etc. may be used herein to describe various elements, components, regions, layers, or parts, such elements, components, regions, layers, or parts are not limited by such terms. Instead, such terms are only used to distinguish one element, component, region, layer, or part from another element, component, region, layer, or part. Therefore, without departing from the teachings of the present invention, the first element, component, region, layer, or part discussed below may be referred to as the second element, component, region, layer, or part.

The terms used herein are only for describing specific example embodiments and are not intended to limit the inventive concept. As used herein, the singular forms "a/an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that when used in this specification, the terms "comprises and comprising" indicate the presence of the stated features, integers, steps, operations, elements or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components or groups thereof.

Errors in manual operation or delays caused by other factors on the production line often lead to bottlenecks in the production line output. However, it is known that the monitoring device on the production line is only a simple image record. Therefore, it is still necessary to manually search the image to find the factors of errors or delays. The efficiency and flexibility of this debugging method are very low, and it cannot effectively improve the bottleneck of production line output. Therefore, in order to more quickly and accurately find the factors that cause errors or delays on the production line, and then improve the efficiency of production line output, innovative monitoring methods and monitoring systems are needed.

Please refer to FIG. 1A, which is a block diagram of a monitoring system 1 of some embodiments of the present invention. The monitoring system 1 includes a processor 11 and a storage unit 13. The storage unit 13 stores a program 130 and an image recognition model 132. The image recognition model 132 may include a model related to machine learning technology. In particular, the image recognition model 132 is mainly a machine learning model generated by using a plurality of training data according to a machine learning algorithm.

Specifically, in some embodiments, some image data and the action types actually corresponding to these image data can be used as training data to train the image recognition model 132 based on the machine learning algorithm (i.e., to generate the image recognition model 132). In this way, the image recognition model 132 can be used to receive image data and output the action type of the operator in the image. For example, after receiving the image sequence of the operator, the image recognition model 132 determines that the operator is performing the action of "picking up" or "putting down", and outputs the action type as "picking up" or "putting down".

The processor 11 and the storage unit 13 are electrically connected via a communication bus 17. Through the communication bus 17, the processor 11 can execute a program 130 stored in the storage unit 13. When the program 130 is executed, one or more interrupts may be generated, such as a software interrupt, so that the processor 11 executes the program 130 having the production line monitoring function. The function of the program 130 will be further described below.

Please refer to FIG. 1B , which is a schematic diagram of the use of the monitoring system 1 of some embodiments of the present invention. In detail, when the operation of the production line machine 92 needs to be monitored and analyzed, the image capture device 91 can be installed in the environment where the production line machine 92 is located to capture images related to the production line machine 92. Among them, the monitoring system 1 can be connected to the image capture device 91 via a network connection (wired network or wireless network).

In some embodiments, when the operator 93 operates on the production line machine 92, the image capture device 91 can capture multiple images 910 of the operator 93 with respect to the position of the production line machine 92, and transmit the multiple images 910 to the monitoring system 1 via the network. In other words, the monitoring system 1 can obtain the multiple images 910 of the operator 93 from the image capture device 91 via the network.

Next, using the image recognition model 132 generated and stored in the storage unit 13, the processor 11 of the monitoring system 1 can determine the action type of the operator 93 of the multiple images 910. Since the multiple images 910 carry relevant information of the timestamp, the processor 11 can determine the capture time of the multiple images 910, and further determine the occurrence time and action cycle of the action type represented by the multiple images 910. The processor 11 can record the action type, occurrence time and action cycle in the storage unit 13 for subsequent use.

Please refer to FIG. 2A, which is a block diagram of a monitoring system 2 of some embodiments of the present invention. The monitoring system 2 includes a processor 21, a storage unit 23, and an input device 25. The storage unit 23 stores a program 230, an image recognition model 232, and training data 234. The image recognition model 232 may include a model related to machine learning technology, which is used to receive video data (i.e., image sequence data) and output the action type of the operator in the video.

The processor 21, the storage unit 23, and the input device 25 are electrically connected via the communication bus 27. Through the communication bus 27, the processor 21 can execute the program 230 stored in the storage unit 23. When the program 230 is executed, one or more interrupts can be generated, such as software interrupts, so that the processor 21 executes the program 230 with the production line monitoring function. The function of the program 230 will be further described below.

In some embodiments, the image recognition model 232 is mainly a machine learning model generated by using a plurality of training data 234 according to a machine learning algorithm. Specifically, some video data and the action types actually corresponding to these video data can be used as training data to train the image recognition model 232 based on the machine learning algorithm (i.e., to generate the image recognition model 232).

In more detail, each training data 234 may include: (1) video data; and (2) an action type corresponding to the video data. When the program 230 is executed, the processor 21 extracts the training data 234 from the storage unit 23 and uses a machine learning algorithm to train the image recognition model 232 based on the multiple training data 234.

In other words, the video data of the plurality of training data 234 can be used as training input data during the training phase, and the action types of the plurality of training data 234 can be used as training output data during the training phase. After the processor 21 generates the image recognition model 232, the image recognition model 232 can be stored in the storage unit 23 for subsequent use.

It should be noted that in some embodiments, the machine learning algorithm mainly introduces a convolutional neural network (CNN) algorithm to construct an image recognition model 232 for determining the type of action based on training data 234. In some examples, the CNN algorithm may include image processing and image recognition algorithms such as the YOLO (you only look once) algorithm and the R3D (ResNet 3D) algorithm, but it is not intended to limit the machine learning algorithm in the present invention.

In some embodiments, in the code of the CNN algorithm for training the image recognition model 232, there is a training function for training the image recognition model 232. During the training of the image recognition model 232, the training function may include a portion for receiving the training data 234.

Furthermore, the video data can be used as training input data, and the action type corresponding to the video data can be used as training output data. Then, the training function can be executed after the main function of the CNN algorithm code is executed to train the image recognition model 232. After the image recognition model 232 is generated based on the CNN algorithm and using the training data, the image recognition model 232 can be used to determine the action type corresponding to the input video.

Please refer to FIG. 2B, which is a schematic diagram of the use of the monitoring system 2 of some embodiments of the present invention. In detail, when the operation of the production line machine 82 needs to be monitored and analyzed, the image capture device 81 can be installed in the environment where the production line machine 82 is located to capture the video related to the production line machine 82. Among them, the monitoring system 2 can be connected to the image capture device 81 through a network connection (wired network or wireless network).

In some embodiments, when the operator 83 operates on the production line machine 82, the image capture device 81 can capture the video 810 of the operator 83 in real time (e.g., video streaming) with respect to the position of the production line machine 82, and transmit the video 810 to the monitoring system 2 via the network. In other words, the monitoring system 2 can obtain the video 810 of the operator 83 from the image capture device 81 via the network.

In some embodiments, in order to increase the accuracy of the image recognition model 232 conversion, the video captured on-site by the production machine 82 can be used as feedback data to adjust the image recognition model 232. In detail, the video 810 can include multiple video clips. Using the image recognition model 232 generated and stored in the storage unit 23, the processor 21 of the monitoring system 2 can determine the action type of the operator 83 in each video clip.

After the processor 21 uses the image recognition model 232 to determine the action type of the operator 83 in each video segment of the video 810, the monitoring system 2 can provide the video segment and the corresponding action type to the user, so that the user can determine whether there is a conversion error in the image recognition model 232. In some embodiments, the monitoring system 2 can provide the video segment and the corresponding action type to the user through a display (not shown) and a graphical user interface (GUI).

Then, if the user determines that the specific video clip and its corresponding action type are conversion errors of the image recognition model 232, the user can input user settings through the input device 25 to change the action type of the specific video clip to a correct action.

Subsequently, the processor 21 can use the specific video clip and the corrected action type to update the training data 234, and reuse the updated multiple training data 234 to generate the image recognition model 232. More specifically, the processor 21 can use the original training data 234, at least one specific video clip, and at least one action type corresponding to the at least one specific video clip to generate the image recognition model 232 based on the machine learning algorithm.

In this way, since the training data used for the image recognition model 232 includes relevant data for the production line machine 82 and the operator 83 (i.e., at least one specific video clip and at least one action type corresponding to the at least one specific video clip), the updated image recognition model 232 will have a higher conversion accuracy when applied to the environment of the production line machine 82.

The technology of using the video captured on-site by the production machine 82 as feedback data to adjust the image recognition model 232 can be more clearly understood through the following example. For example, the video 810 includes ten video segments "C1~C10". Using the image recognition model 232 generated and stored in the storage unit 23, the processor 21 of the monitoring system 2 can determine the action type of the operator 83 in each video segment "C1~C10" (for example: "pick up" action or "put down" action).

After the processor 21 uses the image recognition model 232 to determine the action type of the video segment "C1~C10", the monitoring system 2 provides the video segment "C1~C10" and its corresponding action type to the user through the display and GUI, so that the user can determine whether there is a conversion error in the image recognition model 232.

In this example, the action types of the video clips "C1" and "C8" are judged by the monitoring system 2 as "pick-up action" and "put down action", respectively. However, the user judges that the action types of the video clips "C1" and "C8" should be "put down action" and "pick-up action", respectively. Therefore, the user inputs the user setting through the input device 25 to correct the action types of the video clips "C1" and "C8" to "put down action" and "pick-up action", respectively. Subsequently, the processor 21 updates the training data 234 with the video clips "C1" and "C8" and the corrected action types, and reuses the updated multiple training data 234 to generate the image recognition model 232.

In some embodiments, after the image recognition model 232 is updated through the aforementioned steps, when the operator 83 continues to operate on the production line machine 82, the image capture device 81 can capture the video 812 of the operator 83 according to the position of the production line machine 82, and transmit the video 812 to the monitoring system 2 through the network. In other words, the monitoring system 2 can obtain the video 812 of the operator 83 from the image capture device 81 through the network. The video 812 includes multiple video clips.

Then, using the image recognition model 232 that has been updated and stored in the storage unit 23, the processor 21 of the monitoring system 2 can determine the action type of each video segment of the video 812. Since each video segment carries relevant information of the timestamp, the processor 21 can determine the capture time of each video segment, and further determine the occurrence time and action cycle of the action type represented by each video segment. The processor 21 can record the action type and action cycle in the storage unit 23 for subsequent use.

In some embodiments, the processor 21 can determine whether the action cycle of the corresponding action type exceeds the cycle threshold for each video clip stored in the storage unit 23. If so, the action type and the corresponding video clip are marked, and the action type, occurrence time and occurrence cycle corresponding to the video clip are recorded in the log file. In this way, the user can use the log file to efficiently call out the marked video clips in the video 812, and further understand the reasons why the action cycle of the action type in these video clips exceeds the cycle threshold, so as to quickly eliminate the factors causing the delay.

For example, the default "pick up" action should be completed within 3 seconds, then the processor 21 determines whether the action cycle of all video clips corresponding to the "pick up" action exceeds the 3-second value. If so, the action type and the corresponding video clip are marked, and the action type, occurrence time and occurrence cycle corresponding to the video clip are recorded in the log file. The user can then use the log file to efficiently call out the marked video clips in the video 812, and further understand the reasons why the action cycle of the action type in these video clips exceeds 3 seconds, so as to quickly eliminate the factors causing the delay.

In some embodiments, the processor 21 can determine whether the time difference between the occurrence times of the two corresponding action types exceeds the time threshold for all two consecutive video clips stored in the storage unit 23. If so, the two action types and the corresponding two video clips are marked, and the action types, occurrence times, and occurrence cycles of the two video clips are recorded in the log file. In this way, the user can use the log file to efficiently call out the two marked video clips in the video 812, and further understand the reason why the time difference between the occurrence times of the two corresponding action types exceeds the time threshold, so as to quickly eliminate the factors causing the delay.

For example, it is assumed that the related parts configuration operations between the consecutive "pick up" action and the "put down" action should be completed within 10 seconds. The processor 21 determines whether the time difference between the two video clips of the consecutive "pick up" action and the "put down" action exceeds 10 seconds. If so, the two action types and the corresponding two video clips are marked, and the action types, occurrence times, and occurrence cycles of the two video clips are recorded in the log file. In this way, the user can use the log file to efficiently call out the two marked video clips in the video 812, and further understand the reason why the time difference between the occurrence times of the corresponding two action types exceeds 10 seconds, so as to quickly eliminate the factors causing the delay.

Please refer to FIG. 2C, which is a schematic diagram of a screen captured by the image capture device 81 of some embodiments of the present invention. In some embodiments, since the image or video captured by the image capture device 81 has a larger range, the processor 21 will consume more hardware resources and time when processing the image or video using the image recognition model 232.

However, not all images or videos captured by the image capture device 81 need to be monitored. Therefore, a smaller area that needs to be monitored can be defined for the captured images or videos, and the processor 21 only needs to process the images or videos in the smaller area using the image recognition model 232. In this way, the processing speed can be greatly accelerated.

Please refer to FIG. 2D, which is another schematic diagram of the screen captured by the image capture device 81 of some embodiments of the present invention. Specifically, the user can input user settings through the input device 25 to define the monitoring area 80A on the image range captured by the image capture device 81, and the processor 21 only needs to process the image or video of the monitoring area 80A using the image recognition model 232. In this way, since the image or video size of the monitoring area 80A is smaller, the processing speed of the monitoring system 2 can be greatly accelerated.

In some embodiments, when the environment of the production line machine 82 changes (for example, the angle of the image capture device 81 is adjusted, the operator configuration changes, the operator position changes, etc.), the originally intended monitoring range may deviate from the monitoring area 80A, and cause the image recognition model 232 conversion error to increase. At this time, the user can directly adjust the position of the monitoring area 80A to reduce the deviation caused by the environmental changes of the production line machine 82.

Please refer to FIG. 2E, which is another schematic diagram of the image captured by the image capture device 81 of some embodiments of the present invention. Specifically, due to changes in the environment of the production line machine 82, the image or video in the monitoring area 80A is not the content that needs to be monitored, which may cause the conversion error of the image recognition model 232 to increase.

Please refer to FIG. 2F, which is another schematic diagram of the screen captured by the image capture device 81 of some embodiments of the present invention. Specifically, the user can input another user setting through the input device 25 to move the monitoring area 80A on the image range captured by the image capture device 81 so that the area to be monitored returns to normal.

In some embodiments, the images captured by the image capture device 81 may be first transmitted to the monitoring system 2. Subsequently, the monitoring system 2 may display these images through a general display (not shown) and receive user settings through an input device 25 such as a keyboard or a mouse, so that the monitoring system 2 can complete related operations.

In some embodiments, the images captured by the image capture device 81 may be first transmitted to the monitoring system 2. Subsequently, the monitoring system 2 may transmit the images to a remote display (e.g., a handheld smart device, a laptop, etc.) via a network, and receive user settings via an input device 25 such as a network interface, so that the monitoring system 2 can complete related operations.

Please refer to FIG. 3A, which is a block diagram of a monitoring system 3 of some embodiments of the present invention. The monitoring system 3 includes a processor 31, a storage unit 33, and an input device 35. The storage unit 33 stores a program 330, an image recognition model 332A, an image recognition model 332B, and training data 334A, 334B. Among them, the image recognition models 332A, 332B can include models related to machine learning technology, which are used to judge the operator's action type or the quantity change of objects in the video data (i.e., image sequence data).

The processor 31, the storage unit 33, and the input device 35 are electrically connected via the communication bus 37. Through the communication bus 37, the processor 31 can execute the program 330 stored in the storage unit 33. When the program 330 is executed, one or more interrupts can be generated, such as software interrupts, so that the processor 31 executes the program 330 with the production line monitoring function. The function of the program 330 will be further described below.

In some embodiments, the image recognition model 332A is mainly a machine learning model generated by using a plurality of training data 334A according to a machine learning algorithm. Specifically, some video data and the action types actually corresponding to these video data can be used as training data to train the image recognition model 332A based on the machine learning algorithm (i.e., to generate the image recognition model 332A).

In more detail, each training data 334A may include: (1) video data; and (2) an action type corresponding to the video data. When the program 330 is executed, the processor 31 extracts the training data 334A from the storage unit 33 and uses a machine learning algorithm to train the image recognition model 332A based on the multiple training data 334A.

In other words, the video data of the plurality of training data 334A can be used as training input data during the training phase, and the action types of the plurality of training data 334A can be used as training output data during the training phase. After the processor 31 generates the image recognition model 332A, the image recognition model 332A can be stored in the storage unit 33 for subsequent use.

In some embodiments, the video data used as training input data in the plurality of training data 334A includes image data of the operator's actions. The action type corresponding to the video data can be used as training output data. Then, the code of the CNN algorithm can be executed to train the image recognition model 332A. After the image recognition model 332A is generated based on the CNN algorithm and using the training data, the image recognition model 332A can be used to determine the action type corresponding to the input video.

In some embodiments, the image recognition model 332B is mainly a machine learning model generated by a plurality of training data 334B according to a machine learning algorithm. Specifically, some video data and the actual change in the number of objects corresponding to the video data can be used as training data to train the image recognition model 332B based on the machine learning algorithm (i.e., to generate the image recognition model 332B).

In more detail, each training data 334B may include: (1) video data; and (2) a change in the number of objects corresponding to the video data (e.g., increase or decrease). When the program 330 is executed, the processor 31 extracts the training data 334B from the storage unit 33 and uses a machine learning algorithm to train the image recognition model 332B based on the multiple training data 334B.

In other words, the video data of the plurality of training data 334B can be used as training input data during the training phase, and the change in the number of objects in the plurality of training data 334B can be used as training output data during the training phase. After the processor 31 generates the image recognition model 332B, the image recognition model 332B can be stored in the storage unit 33 for subsequent use.

In some embodiments, the video data used as training input data in the plurality of training data 334B includes image data of object quantity changes. The object quantity changes corresponding to the video data can be used as training output data. Then, the CNN algorithm code can be executed to train the image recognition model 332B. After the image recognition model 332B is generated based on the CNN algorithm and using the training data, the image recognition model 332B can be used to determine the object quantity changes corresponding to the input video.

In more detail, the video data records the quantity changes of specific objects (e.g., product components), and the quantity changes of specific objects can represent different action behaviors. For example, when the quantity of a specific object decreases in the video data, it means that the operator's action is more likely to "pick up" the specific object. When the quantity of a specific object increases in the video data, it means that the operator's action is more likely to "put down" the specific object. Based on this, using the quantity changes of specific objects in the image data can help improve the accuracy of the action type judgment results.

Please refer to FIG. 3B, which is a schematic diagram of the use of the monitoring system 3 of some embodiments of the present invention. In detail, when the operation of the production line machine 72 needs to be monitored and analyzed, the image capture device 71 can be installed in the environment where the production line machine 72 is located to capture the video related to the production line machine 72. Among them, the monitoring system 3 can be connected to the image capture device 71 through a network connection (wired network or wireless network).

In some embodiments, when the operator 73 operates on the production line machine 72, the image capture device 71 can capture the video 710 of the operator 73 in real time (e.g., video streaming) based on the position of the production line machine 72, and transmit the video 710 to the monitoring system 3 via the network. In other words, the monitoring system 3 can obtain the video 710 of the operator 73 from the image capture device 71 via the network. The video 710 includes multiple video clips.

Then, the user can input user settings through the input device 35 to define the monitoring areas 70A and 70B on the image range captured by the image capture device 71, and the processor 31 only needs to process the images or videos of the monitoring areas 70A and 70B using the image recognition models 332A and 332B.

Subsequently, using the aforementioned image recognition model 332A stored in the storage unit 33, the processor 31 of the monitoring system 3 can determine the action type of the monitoring areas 70A and 70B in each video segment of the video 710. Since each video segment carries relevant information of the timestamp, the processor 31 can determine the capture time of each video segment, and further determine the occurrence time and action cycle of the action type represented by the monitoring areas 70A and 70B in each video segment. The processor 31 can record the action type and action cycle in the storage unit 33 for subsequent use.

In some embodiments, for each monitoring area 70A and 70B of a video clip, the processor 31 of the monitoring system 3 can further use the image recognition model 332B to determine the change in the number of objects and update the action type of the operator 73 accordingly. Please refer to Figures 3C to 3D, which are schematic diagrams of the images captured by the image capture device 71 of some embodiments of the present invention. For example, for the monitoring area 70A of a specific video clip, the processor 31 of the monitoring system 3 can first use the image recognition model 332A to determine that the action type is "pick up".

Next, for the monitoring area 70A of this specific video clip, the processor 31 of the monitoring system 3 can further use the image recognition model 332B to determine that the number of objects 74 in the video clip has decreased. Accordingly, since the action type of the specific video clip in the monitoring area 70A is "pick up", and the decrease in the number of objects 74 is indeed caused by "pick up", the specific action type can be accurately confirmed as "pick up".

It should be noted that for the monitoring area 70A of a specific video clip, when the processor 31 of the monitoring system 3 uses the image recognition model 332A to determine that the action type is "put down", but uses the image recognition model 332B to determine that the number of objects 74 in the video clip has decreased, it means that the judgment of the image recognition model 332A may be wrong. Accordingly, based on the use of the image recognition model 332B to determine that the number of objects 74 in the video clip has decreased, the processor 31 of the monitoring system 3 can update the action type of the corresponding specific video clip from "put down" to "pick up".

Please refer to Figures 3E to 3F, which are schematic diagrams of the images captured by the image capture device 71 of some embodiments of the present invention. For example, for the monitoring area 70B of a specific video clip, the processor 31 of the monitoring system 3 can first use the image recognition model 332A to determine that the action type is "put down".

Next, for the monitoring area 70B of this specific video clip, the processor 31 of the monitoring system 3 can further use the image recognition model 332B to determine that the number of objects 74 in the video clip has increased. Accordingly, since the action type of the specific video clip in the monitoring area 70B is "dropping", and the increase in the number of objects 74 is indeed caused by "dropping", the specific action type can be accurately confirmed as "dropping".

Similarly, for the monitoring area 70B of a specific video clip, when the processor 31 of the monitoring system 3 uses the image recognition model 332A to determine that the action type is "pick up", but uses the image recognition model 332B to determine that the number of objects 74 in the video clip has increased, it means that the judgment of the image recognition model 332A may be wrong. Accordingly, based on the judgment that the number of objects 74 in the video clip has increased using the image recognition model 332B, the processor 31 of the monitoring system 3 can update the action type of the corresponding specific video clip from "pick up" to "put down".

It should be noted that in the aforementioned embodiment, when the processor uses the image recognition model to determine the action type of the image or video data, the image recognition model can be used to first identify and track the operator's hand, and then further determine the action type of the operator in the image or video through the operator's hand action.

Some embodiments of the present invention include a production line monitoring method, the flow chart of which is shown in Figures 4A to 4B. The production line monitoring method of these embodiments is implemented by a monitoring system (such as the monitoring system of the aforementioned embodiment). The detailed operation of the method is as follows.

First, the monitoring system executes step S401 to obtain multiple images of the operator. The monitoring system can obtain multiple images of the operator through an image capture device installed on the production line machine. The monitoring system executes step S402 to determine the action type of the operator in the multiple images based on the image recognition model. The image recognition model can include a model related to machine learning technology, which is used to receive image data and output the action type of the operator in the image.

Next, since multiple images can carry relevant information with time stamps, the monitoring system executes step S403 to determine the occurrence time and action cycle of the action type according to the action type of the operator of the multiple images. The monitoring system executes step S404 to record the action type, occurrence time and action cycle for subsequent use.

In some embodiments, after step S402, in order to increase the accuracy of judgment, the monitoring system may execute step S402' to update the action type according to the change in the number of objects in the multiple images based on another image recognition model.

Some embodiments of the present invention include a production line monitoring method, the flow chart of which is shown in Figures 5A to 5F. The production line monitoring method of these embodiments is implemented by a monitoring system (such as the monitoring system of the aforementioned embodiment). The detailed operation of the method is as follows.

In some embodiments, the production line monitoring method needs to provide an image recognition model including machine learning technology to receive image data and output the operator's action type in the image. Therefore, it is necessary to first use training data to train and generate the image recognition model.

Please refer to FIG. 5A, which is a flowchart of the image recognition model generation of the production line monitoring method of some embodiments of the present invention. The monitoring system executes step S501 to generate an image recognition model based on a machine learning algorithm using multiple training data. Each training data includes a training input and a training output. The training input includes a training video clip, and the training output includes a training action type corresponding to the training video clip. The monitoring system executes step S502 to store the image recognition model for subsequent use.

In some embodiments, in order to increase the accuracy of the image recognition model, the video captured on-site at the production machine can be used as feedback to adjust the image recognition model. Please refer to Figure 5B, which is a flow chart of the image recognition model update of the production line monitoring method of some embodiments of the present invention. The monitoring system executes step S503 to obtain the video. Among them, the monitoring system can obtain the operator's video by the image capture device installed on the production line machine, and the video includes multiple video clips.

The monitoring system executes step S504 to determine the action type of the operator in each video clip based on the image recognition model generated above. The monitoring system executes step S505 to provide the video clip and the corresponding action type to the user so that the user can determine whether there is a conversion error in the image recognition model.

When the user determines that there is a conversion error in the image recognition model, resulting in a mismatch between the specific video clip and its corresponding action type, the user can change the specific video clip and its corresponding action type. The monitoring system executes step S506 to receive the user's setting to change the action type of this video clip.

After all video clips have been judged, the monitoring system executes step S507 to adjust the image recognition model according to the specific video clip and the changed action type. In detail, the monitoring system generates the image recognition model based on the original training data, the specific video clip, the action type corresponding to the specific video clip, and the machine learning algorithm.

In this way, since the retraining of the image recognition model contains relevant information about the production line machine and the operator (i.e. the specific video clip and the action type corresponding to this specific video clip), the updated image recognition model will have a higher accuracy when applied to the judgment of the production line machine.

In some embodiments, the status of the production line machine can be monitored based on the updated image recognition model. Please refer to Figure 5C, which is a flow chart of the production line monitoring method of some embodiments of the present invention. The monitoring system executes step S508 to obtain the video of the operator of the production line machine. The video includes multiple video clips. The monitoring system executes step S509 to determine the action type of the operator in each video clip based on the image recognition model.

Next, since multiple video clips may carry relevant information of the timestamp, the monitoring system executes step S510 to determine the occurrence time and action cycle of the action type according to the action type of the operator in each video clip. The monitoring system executes step S511 to record the action type, occurrence time and action cycle for subsequent use.

In some embodiments, the monitoring system can determine whether the action of the video is delayed. Please refer to Figure 5D, which is a flow chart of the production line monitoring method of some embodiments of the present invention. The monitoring system executes step S512 to determine whether the action cycle of the operator in each video clip exceeds the cycle threshold. If so, the monitoring system executes step S513 to mark the action type and the corresponding video clip, and record the action type, occurrence time and occurrence cycle corresponding to the video clip in the log file, so that the user can use the log file to efficiently call up the marked video clip in the video. If not, the monitoring system repeats step S512 to determine whether the action cycle of the operator in the next video clip exceeds the cycle threshold value.

In some embodiments, the monitoring system can determine whether a delay occurs between actions in the video. Please refer to Figure 5E, which is a flow chart of the production line monitoring method of some embodiments of the present invention. The monitoring system executes step S514 to calculate the time difference between the occurrence times of the action types of the two video clips. The monitoring system executes step S515 to determine whether the time difference exceeds the time threshold. If so, the monitoring system executes step S516 to label this action type and the corresponding two video clips, and record the corresponding action type, occurrence time and occurrence cycle of the two video clips in the log file, so that the user can use the log file to efficiently call out the labeled video clip in the video. If not, the monitoring system repeats step S514 to calculate the time difference between the occurrence times of the corresponding action types for the next pair of video clips to determine whether the action cycle exceeds the cycle threshold.

In some embodiments, the following steps may be optionally added to speed up the speed and efficiency of image processing. Please refer to FIG. 5F, which is a flow chart of a production line monitoring method of some embodiments of the present invention. The monitoring system executes step S517 to receive user settings to define a monitoring area on the video to be captured. In other words, the user settings are used to define the monitoring area on the image range captured by the image capture device. Among them, since the image or video size of the monitoring area is smaller, the processing speed of the monitoring system can be greatly accelerated.

In some embodiments, the following steps may be optionally added to reduce the deviation caused by environmental changes in the production line machine site. Please refer to Figure 5G, which is a flow chart of the production line monitoring method of some embodiments of the present invention. The monitoring system executes step S518 to receive user settings to move the monitoring area. In other words, the user setting is used to move the monitoring area on the image range captured by the image capture device.

The monitoring system and production line monitoring method of the present invention can more quickly and accurately identify the factors causing errors or delays on the production line through automation and artificial intelligence, thereby improving the efficiency of the production line output and effectively improving the bottleneck of the production line output.

It should be particularly understood that the processor mentioned in the above embodiment may be a central processing unit (CPU), other hardware circuit elements capable of executing relevant instructions, or a combination of computing circuits known to those skilled in the art based on the above disclosure. In addition, the storage unit mentioned in the above embodiment may include a memory (such as ROM, RAM, etc.) or a storage device (such as flash memory, HDD, SSD, etc.) for storing data.

Furthermore, the communication bus mentioned in the above embodiments may include a communication interface for transmitting data between components such as a processor, a storage unit, an input device, etc., and may include an electrical bus interface, an optical bus interface, or even a wireless bus interface. However, such description is not intended to limit the hardware implementation of the present invention.

Although the present invention and its advantages have been described in detail, it should be understood that various changes, substitutions and alterations may be made herein without departing from the spirit and scope of the present invention as defined by the appended claims. For example, many of the processes discussed above may be implemented in different ways and replaced by other processes or combinations thereof.

In addition, the scope of this application is not intended to be limited to the specific embodiments of the processes, machines, manufactures, material compositions, components, methods, and steps described in this specification. As will be readily understood by those skilled in the art from the disclosure of this invention, currently existing or subsequently developed processes, machines, manufactures, material compositions, components, methods, or steps that perform substantially the same functions as those in the corresponding embodiments described herein or achieve substantially the same results as those in the corresponding embodiments described herein can be utilized according to the present invention. Therefore, the scope of the attached application is intended to include such processes, machines, manufactures, material compositions, components, methods, or steps within its scope.

S401~S404: Steps

Claims (16)

A production line monitoring method for a monitoring system, comprising: based on a machine learning Learning) algorithm, using multiple training data to generate an image recognition model, wherein each training data includes a training input and a training output, the training input includes multiple training images, and the training output includes a training action type corresponding to the training images; obtaining multiple images of an operator; based on the image recognition model, determining an action type of the operator in the images; based on another image recognition model, updating the action type of the operator in the images according to a change in the quantity of an object in the images; determining an occurrence time and an action cycle of the action type updated according to the change in the quantity of the object in the images; and recording the action type, the occurrence time and the action cycle. The production line monitoring method of claim 1 further includes: determining whether the action cycle exceeds a cycle threshold value; when the action cycle exceeds the cycle threshold value, marking the action type. The production line monitoring method of claim 1 further includes: calculating a time difference between the occurrence time of the action type and the occurrence time of another action type; determining whether the time difference exceeds a time threshold; when the time difference exceeds the time threshold, marking the action type and the other action type. The production line monitoring method of claim 1 further includes: receiving a user setting to define a monitoring area on the images; wherein the step of determining the action type of the operator of the images based on the image recognition model further includes: determining the action type within the monitoring area of the images based on the image recognition model. The production line monitoring method of claim 4 further includes: receiving another user setting to move the monitoring area defined on the images. As in claim 4, the production line monitoring method, wherein the images have an image size, the monitoring area has an area size, and the monitoring area size is smaller than the image size. The production line monitoring method of claim 1 further includes: based on the image recognition model, identifying at least one hand of the operator in the images, and determining the action type of the at least one hand of the operator. A production line monitoring method for a monitoring system, comprising: using a plurality of training data and a machine learning (Machine Learning) algorithm to generate an image recognition model, wherein each training data includes a training input and a training output, wherein the training input includes a training video segment, and the training output includes a training action type corresponding to the training video segment; obtain a video, wherein the video includes a plurality of video segments; Based on the image recognition model, determine an action type of each of the video segments; Based on another image recognition model, update the action type of each of the video segments according to a change in the number of objects in each of the video segments; receive a user setting to change the action type of a first video segment of the video segments updated according to the change in the number of objects in each of the video segments; and adjust the image recognition model according to the action type of the first video segment. As in claim 8, the step of adjusting the image recognition model according to the action type of the first video clip further comprises: generating the image recognition model using the training data, the first video clip, the action type corresponding to the first video clip, and the machine learning algorithm. A monitoring system for production line monitoring includes: a processor; and a storage unit storing a program, wherein when the program is executed, the processor: based on a machine learning algorithm, generates an image recognition model using a plurality of training data, and stores the image recognition model in the storage unit, wherein each training data includes a training input and a training output, wherein the training input includes a plurality of training images, and the training output includes a training action type corresponding to the training images; obtains a plurality of images of an operator; based on the images, generates an image recognition model using a plurality of training data, and stores the image recognition model in the storage unit. Recognition model, determine an action type of the operator of the images; based on another image recognition model, update the action type of the operator of the images according to a change in the quantity of an object in the images; determine an occurrence time and an action cycle of the action type updated according to the change in the quantity of the objects in the images; and record the action type, the occurrence time and the action cycle. A monitoring system as claimed in claim 10, wherein when the program is executed, the processor is further caused to: determine whether the action cycle exceeds a cycle threshold; when the action cycle exceeds the cycle threshold, mark the action type. A monitoring system as claimed in claim 10, wherein when the program is executed, the processor is further caused to: calculate a time difference between the occurrence time of the action type and the occurrence time of another action type; determine whether the time difference exceeds a time threshold; when the time difference exceeds the time threshold, mark the action type and the other action type. The monitoring system of claim 10 further comprises: an input device for receiving a user setting; wherein, when the program is executed, the processor further enables: defining a monitoring area on the images according to the user setting; and determining the action type within the monitoring area of the images based on the image recognition model. As in claim 13, the monitoring system, wherein the input device is further used to receive another user setting, and when the program is executed, the processor is further caused to: move the monitoring area defined on the images according to the other user setting. The monitoring system of claim 10 further comprises: an input device for receiving a user setting; wherein when the program is executed, the processor is further caused to: change the action type of the operator of the images according to the user setting; and adjust the image recognition model according to the action type of the operator of the images. The monitoring system of claim 10, wherein when the program is executed, the processor is further caused to: generate the image recognition model based on the machine learning algorithm using the training data, the images, and the action type of the operator corresponding to the images.

Images