TWI826129B - Cycle time detection and correction system and method - Google Patents

Abstract

A cycle time detection and correction method includes multiple steps performed by a computing device, where these steps include: obtaining a video from a camera device, obtaining a bounding box from an input device, inputting the bounding box and the video to a cycle time detection model to generate a preliminary report, wherein the bounding box is configured to mark a region of interest in the video, the preliminary report includes a plurality of candidate events, and each candidate event includes a start time and a candidate cycle time; receiving a revision label associated with at least one candidate event from the input device; and tuning hyper-parameters of the cycle time detection model according to the revision label.

Description

Cycle time detection and correction systems and methods

The present invention relates to image processing and artificial intelligence, and in particular to a video-based cycle time detection and correction system and method.

Manufacturing cycle time is defined as the time cost of a specific step in the production process. In practical applications, managers often calculate average cycle time by counting the number of times a specific step is performed over a period of time. Cycle time is a key performance indicator in manufacturing and, more precisely, an indicator of production efficiency. For example, cycle time can be used in productivity improvement programs. There are many steps required to produce a product. Managers are interested in the time cost of each step. Based on the cycle time, managers can infer the efficiency of each step and decide whether to make that step more efficient. Therefore, cycle time can be used for productivity improvement initiatives.

Two common methods of measuring cycle time include manual measurement and sensor measurement. Manual measurement means that the cycle time is measured by a human being. That is, one worker starts a timer, observes another worker's operations, and calculates the cycle time. However, manual measurement can lead to inefficiencies and biases. First, measuring cycle time requires workers to stand around counting time, which increases opportunity costs and leads to inefficiencies. Secondly, a person in different scenarios Productivity may change under conditions, also known as observer bias. Therefore, manual measurement can be a biased method.

The sensor measurement represents the cycle time detected by the sensor. That is, the sensor detects whether something passes by and records the time of the event. However, sensor measurements have their drawbacks. First, sensors are expensive. Second, sensor measurements only detect simple and specific events. In addition, people need different sensors to detect different events, so the cost of sensors may become higher and higher over time.

In view of this, the present invention proposes a cycle time detection and correction system and method to solve the above problems.

A cycle time detection and correction method according to an embodiment of the present invention includes executing with a computing device: obtaining video from a camera device, obtaining a bounding box from an input device, and inputting the bounding box and video to a cycle time detection model to Generate a preliminary report, wherein the bounding box is used to mark the region of interest in the video, the preliminary report includes a plurality of candidate events, each candidate event includes a start time and a candidate cycle time; receiving from the input device a message associated with at least one candidate event Revise the label; and adjust the hyperparameters of the cycle time detection model based on the revision label.

A cycle time detection and correction system according to an embodiment of the present invention includes a camera device, an input device and a computing device. The camera device captures the environment to produce video. The input device receives bounding boxes and revision labels that mark regions of interest in the video. The computing device communicates with the camera device and the input device. The computing device is used to input the bounding box and video into the cycle time detection model to generate a preliminary report. Bounding boxes are used to mark areas of interest in videos. The preliminary report includes multiple candidate events, each candidate event includes the start time and candidate period time. The computing device is further configured to adjust the hyperparameters of the cycle time detection model according to the revision label, where the revision label is associated with at least one candidate event.

To sum up, the present invention provides a video-based automatic cycle time detection and correction system and method, which has the advantages of automation, flexibility and sustainable correction.

The above description of the present disclosure and the following description of the embodiments are used to demonstrate and explain the spirit and principles of the present invention, and to provide further explanation of the patent application scope of the present invention.

100: Cycle time detection and correction system

10:Camera device

30:Input device

50:Computing device

S1~S7,S30~S38: steps

Figure 1 is a block diagram of a cycle time detection and correction system according to an embodiment of the present invention; Figure 2 is a flow chart of a cycle time detection and correction method according to an embodiment of the present invention; Figure 3 is a diagram of the steps in Figure 2 Detailed flow chart; and Figure 4 is an example of a cycle time distribution chart in the preliminary report.

The detailed features and advantages of the present invention are described in detail below in the implementation mode. The content is sufficient to enable anyone skilled in the relevant art to understand the technical content of the present invention and implement it according to the content disclosed in this specification, the patent scope and the drawings. , anyone familiar with the relevant art can easily understand the relevant objectives and advantages of the present invention. The following examples further illustrate the aspects of the present invention in detail, but do not limit the scope of the present invention in any way.

FIG. 1 is a block diagram of a cycle time detection and correction system according to an embodiment of the present invention. As shown in Figure 1, the cycle time detection and correction system 100 includes a camera device 10, The input device 30 and the computing device 50 are communicatively connected to the camera device 10 and the input device 30 .

The camera device 10 captures the environment to generate video. There are periodic events in the environment, such as operators assembling parts on a production line.

The input device 30 receives the user's operation and marks a bounding box and a revision label of the area of interest in the video. In one embodiment, the input device 30 is further configured to receive an event threshold, an upper boundary, and a lower boundary. In another embodiment, the input device is further configured to receive period hints. For revision labels, event thresholds, upper boundaries, lower boundaries and period prompts, please see the instructions below.

In one embodiment, the input device 30 can be implemented by one or more of the following examples: a keyboard, a mouse, a touch screen, a button, or any device with similar functions. The invention does not limit the input device 30 . Hardware type.

The computing device 50 is used to input the bounding box and video into the cycle time detection model to generate a preliminary report. The preliminary report includes multiple candidate events, and each candidate event includes a start time and a candidate cycle time. The computing device 50 is further configured to adjust the hyperparameters of the cycle time detection model according to the revision label, where the revision label is associated with at least one candidate event.

In one embodiment, the computing device 50 may be implemented using one or more of the following examples: a microcontroller (MCU), an application processor (AP), or a field programmable gate array (field programmable). gate array (FPGA), Application Specific Integrated Circuit (ASIC) system-on-a-chip (SOC), deep learning accelerator (deep learning accelerator), or any electronic device with similar functions device, the present invention does not limit the hardware type of the computing device 50 .

Overall, the present invention is based on the repeatability of video. When shooting videos with periodic events, many frames are repeated, and the system cycle time detection and correction system 100 designed by the present invention can Achieve the goal of detecting periodic events. The operation of this system 100 includes four stages: data exchange, artificial intelligence (AI) inference, report generation, and hyperparameter adjustment (tuning). The following describes the operation of the system cycle time detection and correction system 100. Please refer to FIG. 2, which is a flow chart of a system cycle time detection and correction method according to an embodiment of the present invention, including steps S1, S3, S5 and S7.

In step S1 , the computing device 50 obtains video from the camera device 10 . Step S1 belongs to the aforementioned data exchange stage.

In one embodiment, the cycle time detection and correction system 100 proposed by the present invention further includes a user interface. The user interface is, for example, in the form of a web page, allowing the user to set a site name, site address, and model parameters. The site represents the location where the camera device 10 is installed, and the site address is, for example, the Internet of the camera device 10 . Internet protocol address (IP address), which ensures that the system 100 can stream the captured video through Real Time Streaming Protocol (RTSP) or Hypertext Transfer Protocol (HTTP) . The computing device 50 further stores a site list, which records the information filled in by the user through the user interface. Users can also view the site list and add or delete sites from it. After adding a site, the computing device 50 assigns a number to the new site for identification.

The user interface also allows users to set parameters required in the second stage (AI inference), such as bounding boxes and period prompts for specifying regions of interest (ROI) in the video. Users can crop ROI from videos to ignore irrelevant information information, and period prompts can be set as parameters to guide the AI model to focus on certain specific periods in the video.

In step S3, the computing device 50 obtains the bounding box from the input device 30, and inputs the bounding box and the video into the cycle time detection model to generate a preliminary report. Step S3 includes the aforementioned AI inference stage and report generation stage. At this stage, the system 100 starts streaming frames and detects cycles based on the AI model. Figure 3 is a detailed flow chart of step S3, including steps S30, S32, S34, S36 and S38. Specifically, in step S32, the computing device 50 crops the video according to the bounding box. In step S32, the computing device 50 inputs multiple frames of the video into the neural network model to generate multiple feature vectors. In step S34, the computing device 50 performs principal component analysis (PCA) on these feature vectors to reduce the dimensions of these feature vectors and generate multiple low-dimensional signals. In step S36, the computing device 50 then performs short-time Fourier Transform (STFT) based on these low-dimensional signals to generate a plurality of smoothed signals. In step S38, the computing device 50 identifies periodic events based on multiple peaks in the smoothed signals to obtain the multiple candidate events.

In one embodiment, the computing device 50 starts a thread to stream the frame and store the frame in a buffer, and starts another thread to obtain the frame from the buffer, calculate The AI infers results and stores the results.

In one embodiment, the dimensionality of the feature vector is reduced by projecting the vector into a one-dimensional space.

In one embodiment, before performing STFT based on multiple low-dimensional signals to generate multiple smoothed signals, the computing device 50 sets the sliding window size in the STFT according to the period prompt, for example, the period prompt is set to 65 seconds.

In one embodiment, the computing device 50 filters the peaks in the smoothed signal according to an event threshold to remove noise, that is, retaining events whose peak prominence is higher than the event threshold as candidate events. .

During the report generation stage, the system 100 presents tables and charts through the user interface for users to view. An example of a table is shown below, and an example of a chart is shown in Figure 4.

Each column in Table 1 represents a candidate event. Each candidate event includes a start time (event time), candidate cycle time and AI label. The above information is generated by the cycle time detection model. For example, the event time is inferred by AI, and then the cycle time is estimated by calculating the time difference between two consecutive events. Finally, the cycle time is compared with the normal interval to determine the AI label. Candidate events located within the normal range are normal events, and candidate events located outside the normal range are abnormal events. The normal interval includes the aforementioned upper boundary and lower boundary, which can be set by the user through the input device 30 during the report generation stage or the data exchange stage. For example, the upper boundary can be set to 100 seconds and the lower boundary can be set to 30 seconds. The corresponding diagram is shown in Figure 4. Revision labels and descriptions are filled in by the user. For example, in the example in Table 1, the user corrects an abnormal event with an event time of 0:06:28 and a cycle time of 134 seconds into a normal event.

In one embodiment, if an abnormal periodic event is detected, the computing device 50 further sends an alarm signal to notify the user. In addition, the computing device 50 fine-tunes the cycle time detection model by collecting revision tags from user feedback, so that the AI can simulate human decision-making and continuously update and evolve to fit the current situation.

In step S7, the computing device 50 adjusts the hyperparameters of the cycle time detection model according to the revision tag. Step S7 belongs to the aforementioned hyperparameter adjustment stage. Hyperparameters are a common requirement for AI algorithms. Different hyperparameters adapt to different scenarios or data, so choosing appropriate hyperparameters can improve the performance of the algorithm. The cycle time detection and correction system 100 proposed by the present invention will automatically fine-tune the hyperparameters based on the user's input in the third stage "report generation" and continue to make corrections. In one embodiment, the hyperparameters that can be modified include event thresholds and normal intervals (upper and lower boundaries). Furthermore, event thresholds can be used to filter peak prominence. Events that are too low can achieve the effect of removing noise. However, the peak prominence does not easily correspond to the concept of real life, so it is not easy for users to adjust manually. In this case, it is better to have the system automatically fine-tune it.

Event thresholds are used to find events, and the AI model finds candidate events in the signal by finding the peaks of the smoothed signal. Therefore, the computing device 50 fine-tunes the event threshold by updating the event threshold to the lowest peak prominence in the historical data.

The normal interval is used to classify events, and the computing device 50 marks events located between the lower boundary and the upper boundary as normal events, and marks other events as abnormal events. Therefore, the computing device 50 fine-tunes the normal interval by moving the boundaries to include the modified events.

To sum up, the present invention provides a video-based automatic cycle time detection and correction system and method, which has the advantages of automation, flexibility and continuous correction. First, the streaming video data is used for periodic image detection, and then the detected period is marked as normal or abnormal, and the user is notified through real-time alarms or periodic reports. In addition, the invention also designs a feedback mechanism to avoid false alarms. Users can freely correct the result inference results, revise labels to provide feedback and explain the reasons to facilitate future improvement plans, or fine-tune the AI inference model to simulate human decision-making and get closer to the current situation. The system and method proposed by the present invention have the following three advantages:

First, automation. The system proposed by the present invention can automatically identify the cycle based on the video without human intervention. Therefore, it relieves workers of the opportunity cost of manual counting. In addition, the system proposed by the present invention runs in the background and workers work as usual. Therefore, observer bias in manual measurements does not occur.

Second, flexibility. When measuring with conventional sensors, the sensor can only detect specific events. In contrast, since video contains a large amount of periodic information, the system proposed in the present invention is suitable for a variety of sites. In detail, the system proposed by the present invention is applicable to all sites with camera devices.

Third, continue to correct. The system proposed by the present invention provides users with a user interface for modifying abnormality detection results. This interface can be used not only to generate reports, but also to fine-tune the AI models within the system. For example, fine-tuning a neural network or adjusting the hyperparameters of an algorithm. Revision labels can be viewed as labeled data in machine learning and can be used for model training. Using labeled data, the neural network can be fine-tuned to fit the current data. In addition, labeled data can also be used to adjust the hyperparameters of the algorithm, thereby reducing the error rate or false positive rate. Through continuous modification, the system proposed in the present invention can simulate human decision-making and be close to the current situation.

Although the present invention is disclosed in the foregoing embodiments, they are not intended to limit the present invention. All changes and modifications made without departing from the spirit and scope of the present invention shall fall within the scope of patent protection of the present invention. Regarding the protection scope defined by the present invention, please refer to the attached patent application scope.

S1~S7: steps

Claims (8)

A cycle time detection and correction method includes executing with a computing device: obtaining a video from a camera device; obtaining a certain bounding box from an input device, and inputting the bounding box and the video to a cycle time detection model to generate a preliminary report, wherein the bounding box is used to mark a region of interest in the video, the preliminary report including a plurality of candidate events, each of the candidate events including a start time and a candidate cycle time; from the input The device receives a revision tag associated with at least one of the candidate events; and adjusts hyperparameters of the cycle time detection model according to the revision tag, wherein the bounding box is obtained from the input device and the bounding box is input and applying the video to the cycle time detection model to generate the preliminary report includes: cropping the video according to the bounding box; inputting multiple frames of the video into a neural network model to generate multiple feature vectors; performing a principal component analysis on the feature vectors to reduce the dimensions of the feature vectors to generate multiple low-dimensional signals; performing short-time Fourier transform on the low-dimensional signals to generate multiple smoothed signals; and based on the smoothed signals Multiple peaks in the periodic events are identified to obtain these candidate events. The cycle time detection and correction method of claim 1, further comprising: receiving an upper boundary and a lower boundary from the input device; and Before identifying the periodic events based on the peaks in the smoothed signal to obtain the candidate events, the peaks are screened based on the normal event ranges covered by the upper boundary and the lower boundary. The cycle time detection and correction method as described in claim 1 further includes: receiving a cycle prompt from the input device; and before performing short-time Fourier transform based on the low-dimensional signals to generate a plurality of smoothed signals, based on This period prompt sets the sliding window size in this short-time Fourier transform. The cycle time detection and correction method as described in claim 2, wherein each of the events further includes an event tag, the event tag is used to indicate that each of the events is a normal event or an abnormal event; the The revision label is used to revise the candidate event belonging to the normal event into an abnormal event, or to revise the candidate event belonging to the abnormal event into a normal event; and adjusting the hyperparameters of the cycle time detection model according to the revision label include: Adjust the numerical settings of the upper boundary and the lower boundary according to the revision label. The cycle time detection and correction method described in claim 3, wherein adjusting the hyperparameters of the cycle time detection model based on the revision label includes: adjusting the numerical setting of the cycle prompt based on the revision label. A cycle time detection and correction system includes: a camera device that shoots an environment to generate a video; an input device that receives a bounding box and a revision tag that mark an area of interest in the video; and A computing device, communicatively connected to the camera device and the input device, the computing device is used to input the bounding box and the video into a cycle time detection model to generate a preliminary report, wherein the bounding box is used in the video Marking a region of interest, the preliminary report includes a plurality of candidate events, each of the candidate events includes a start time and a candidate cycle time, and the computing device is further configured to adjust the cycle time detection model according to the revision tag. Hyperparameters, wherein the revision label is associated with at least one of the candidate events, wherein the computing device is further configured to: crop the video according to the bounding box; input multiple frames of the video into a neural network model to Generate multiple feature vectors; perform a principal component analysis on these feature vectors to reduce the dimensions of the feature vectors to generate multiple low-dimensional signals; perform short-time Fourier transforms based on the low-dimensional signals to generate multiple smoothings signals; and identifying periodic events based on multiple peaks in the smoothed signals to obtain the candidate events. The cycle time detection and correction system of claim 6, wherein the input device is further used to receive an upper boundary and a lower boundary, and the computing device further screens the normal event ranges covered by the upper boundary and the lower boundary. some peaks. The cycle time detection and correction system of claim 7, wherein the input device is further configured to receive a cycle prompt, and the computing device further sets the sliding window size in the short-time Fourier transform according to the cycle prompt.

Images