TW202201275A - Device and method for scoring hand work motion and storage medium - Google Patents

Abstract

A method for scoring hand work motion includes: converting acquired hand work image frames to HSV images and obtaining binary images where the skin is located and multiple rectangular frames where the hand is located from the HSV images; segmenting first hand images from the binary images based on the multiple rectangular frames and inputting the first hand images to a first predetermined model; building tracking data accumulated with tracking time based on analysis results of the first predetermined model; invoking a predetermined hand series detection model to detect the hand work image frames if coordinates of key points of the hand obtained by the first predetermined model are unreliable; using a second predetermined model to assign hand labels to each hand in the tracking data for data classification; pre-processing classified tracking data to get precise data of each hand; scoring hand work motion of each hand based on the precise data of each hand and standard precise data. A device for scoring hand work motion and a storage medium are also provided.

Description

Device, method and computer-readable storage medium for scoring hand movement

The present invention relates to the technical field of data processing, and in particular, to a hand operation action scoring device, method and computer-readable storage medium.

In recent years, with the development of deep learning technology, conventional hand recognition technology generally uses hand detection models based on various architectures of convolutional neural networks to perform hand recognition. These models generally require that the input image can only be stored in one hand, which has certain limitations.

In modern factories, the accuracy of workers' hand movements can have a certain impact on the yield and production efficiency of the production line. The conventional method of evaluating the hand movements of workers is generally that the assessor directly observes and scores the worker's hand movements in real time. The accuracy of the assessment is affected by the human factors of the assessor, and the accuracy cannot be guaranteed.

In view of this, it is necessary to provide a hand work action scoring device, method and computer-readable storage medium, which can intelligently analyze the difference between the worker's hand work action and the standard work action, and give corresponding scores.

An embodiment of the present invention provides a hand work action scoring method, the method includes: acquiring a hand work image, and decoding the hand work image to obtain a hand work image frame; The image frame is converted into an HSV image, and a binarized image representing the area where the skin is located and a plurality of rectangular frames representing the area where the hand is located are obtained from the HSV image; The first hand image is segmented from the valued image, and the first hand image is analyzed by using the first preset model; based on the analysis result of the first preset model, a tracking data; monitoring the tracking data using a second preset model and assigning a hand tag to each hand portion of the tracking data to classify the tracking data based on the hand tag, wherein each of the The hand corresponds to a unique hand label; the classified tracking data is preprocessed to obtain the precise data of each of the hands; Refinement data were scored for each of the described hand movements.

An embodiment of the present invention provides a hand operation action scoring device, the device includes a processor and a memory, the memory stores a plurality of computer programs, and the processor is used to execute the computer programs stored in the memory. The steps are as follows: acquiring a hand work image, and decoding the hand work image to obtain a hand work image frame; converting the hand work image frame into an HSV image, and extracting the image from the HSV image Obtain a binarized image representing the area where the skin is located and a plurality of rectangular frames representing the area where the hand is located; segment the first hand image from the binarized image according to the multiple rectangular frames, and use the second A preset model analyzes the first hand image; based on the analysis result of the first preset model, a tracking data accumulated with tracking time is constructed; the second preset model is used to monitor the tracking data and provide Each hand in the tracking data is assigned a hand label to classify the tracking data based on the hand label, wherein each hand corresponds to a unique hand label; the tracking after the classification The data are preprocessed to obtain the fine data of each of the hands; and the work movements of each of the hands are scored according to the fine data of each of the hands and the fine data of the reference hand work.

An embodiment of the present invention provides a computer-readable storage medium. The computer-readable storage medium stores a plurality of instructions, and a plurality of the instructions can be executed by one or more processors, so as to realize the following steps: obtaining a hand The hand work image is decoded to obtain the hand work image frame; the hand work image frame is converted into an HSV image, and the area representing the skin is obtained from the HSV image The binarized image and a plurality of rectangular frames representing the area where the hand is located; the first hand image is segmented from the binarized image according to the plurality of rectangular frames, and the The first hand image is analyzed; based on the analysis result of the first preset model, a tracking data accumulated with the tracking time is constructed; the second preset model is used to monitor the tracking data and record the tracking data in the tracking data; Each hand is assigned a hand label to classify the tracking data based on the hand label, wherein each hand corresponds to a unique hand label; preprocessing the classified tracking data, to obtain the fine data of each of the hands; and to score the work movements of each of the hands according to the fine data of each of the hands and the fine data of the reference hand work.

Compared with the prior art, the above-mentioned device, method and computer-readable storage medium for evaluating hand movement can process the real-time image of the person's hand movement, and accurately locate the hand key in the process of hand movement. Point features, and compare with the standard hand movements, intelligently analyze the difference between the subject's hand movements and the standard job movements, and give corresponding scores, which is conducive to the evaluation and inspection of workers and improve the production line. Yield and efficiency.

Please refer to FIG. 1 , which is a schematic diagram of a preferred embodiment of the device for evaluating hand movements according to the present invention.

The hand operation action scoring device 100 can realize the analysis of the worker's hand action action, and by comparing with the reference hand action action, realize the scoring of the worker's hand action action. The hand work movement scoring apparatus 100 may include a memory 10 , a processor 20 , and a hand work action scoring program 30 stored in the memory 10 and running on the processor 20 . When the processor 20 executes the hand work action scoring program 30, it implements the steps in the embodiment of the hand work action scoring method, such as steps S500-S512 shown in FIG. 5 . Alternatively, the processor 20 implements the functions of the modules shown in FIG. 2 , such as modules 101 to 108 , when the processor 20 executes the hand operation action scoring program 30 .

The hand activity scoring program 30 can be divided into one or more modules, and the one or more modules are stored in the memory 10 and executed by the processor 20 to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of accomplishing specific functions, and the instruction segments are used to describe the execution process of the hand movement evaluation program 30 in the hand movement evaluation device 100 . For example, the hand movement scoring program 30 can be divided into the acquisition module 101, the first detection module 102, the analysis module 103, the tracking module 104, the second detection module 105, and the sorting module 106 in FIG. 2 . , a calibration module 107 and a scoring module 108 . For the specific functions of each module, please refer to the function of each module in Figure 2 below.

Those skilled in the art can understand that the schematic diagram is only an example of the hand work action scoring apparatus 100, and does not constitute a limitation on the hand work action scoring apparatus 100, and may include more or less components than those shown in the figure, or a combination thereof Certain components, or different components, for example, the hand work action scoring apparatus 100 may further include an input display device, a communication module, a bus bar, and the like.

The processor 20 may be a central processing unit (CPU), other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or an off-the-shelf program. Design Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor 20 can also be any conventional processor, etc. The processor 20 can use various interfaces and bus bars to connect various parts of the hand movement evaluation device 100 .

The memory 10 can be used to store the hand movement scoring program 30 and/or the module. The processor 20 executes or executes the computer program and/or the module stored in the memory 10 and calls the program and/or module stored in the memory 10 Various functions of the hand operation action scoring device 100 are realized. The memory 10 may include high-speed random access memory, and may also include non-volatile memory such as hard disk drives, internal memory, plug-in hard disk drives, Smart Media Cards (SMC), secure digital (Secure Digital, SD) card, flash memory card (Flash Card), at least one disk memory device, flash memory device, or other volatile solid state memory device.

FIG. 2 is a functional module diagram of a preferred embodiment of the hand operation action scoring program of the present invention.

Referring to FIG. 2 and FIG. 3 , the hand operation action scoring program 30 may include an acquisition module 101 , a first detection module 102 , an analysis module 103 , a tracking module 104 , a second detection module 105 , and an arrangement module 106. The calibration module 107 and the scoring module 108. In one embodiment, the above-mentioned modules may be programmable software instructions stored in the memory 10 and invoked by the processor 20 for execution. It can be understood that, in other embodiments, the above-mentioned modules can also be program instructions or firmware solidified in the processor 20 .

The acquisition module 101 is used for acquiring a hand operation image, and decoding the hand operation image to obtain a hand operation image frame.

In one embodiment, an image recording device (eg, a camera) can be used to record hand work images of one or more designated workers (evaluators) on a designated production line when performing hand operations. The hand operation action scoring apparatus 100 can communicate with the image recording device, and then the acquisition module 101 can acquire the hand operation image. When the acquisition module 101 acquires the hand operation image, the hand operation image can be decoded to obtain a plurality of hand operation image frames arranged in sequence.

In one embodiment, the hand operation image frame may include one or more hands, and the acquisition module 101 may sequentially transmit each of the hand operation image frames to the first detection module 102 for analysis. .

The first detection module 102 is used to convert the hand operation image frame into an HSV (Hue, Saturation, Value) image, and obtain a binarized image representing the area where the skin is located and from the HSV image. Multiple rectangles representing the area where the hand is located.

In one embodiment, the first detection module 102 can convert the hand operation image frame transmitted by the acquisition module 101 into an HSV image, and then acquire a binarized image representing the area where the skin is located from the HSV image. Like and represent a series of rectangular boxes in the area where the hand is located, these rectangular boxes can be transmitted to the analysis module 103 for analyzing the key points of the hand.

In one embodiment, the first detection module 102 obtains from the HSV image according to the upper and lower limit values of the dynamic H channel, the upper and lower limit values of the dynamic S channel, and the upper and lower limit values of the dynamic V channel, indicating where the skin is located. The binarized image of the area and multiple rectangular boxes representing the area where the hand is located, thereby avoiding that the upper and lower limits of the HSV three channels (ie, the H channel, the S channel and the V channel) are fixed values, which cannot be adapted to many different situations. The problem of hand detection can be avoided to avoid the situation of missed detection or false detection of the hand. The first detection module 102 can dynamically update the upper and lower limits of its own HSV three channels. Specifically, the first detection module 102 can update the upper and lower limits of the HSV three channels according to the feedback results of the analysis module 103. When the feedback result given by the module 103 is a positive result, the first detection module 102 can update the upper and lower limit values of the H channel, the upper and lower limit values of the S channel, and the upper and lower limit values of the V channel according to the preset update rule; When the feedback result given by the group 103 is a negative result, the first detection module 102 will not update the upper and lower limit values of the three HSV channels. The preset update rule can be preset according to actual usage requirements, and the preset update rule defines an adjustment rule for the upper and lower limit values of each HSV channel in advance.

It can be understood that the first detection module 102 updates the upper and lower limit values of the HSV three channels according to the feedback result of the current hand operation image frame output by the analysis module 103, and the updated upper and lower limit values of the HSV three channels are used for A hand operation image frame is detected to obtain a binarized image representing the area where the skin is located and a plurality of rectangular frames representing the area where the hand is located in the next hand operation image frame.

In one embodiment, when the hand operation image frame is the initial frame image, the first detection module 102 can use the preset upper and lower limit values of the three HSV channels to obtain from the HSV image to indicate where the skin is located. A binarized image of the region and a number of rectangular boxes representing the region where the hand is located. That is, when the hand operation image frame is the initial frame image, the upper and lower limit values of the H channel are the first default upper and lower limit values, and the upper and lower limit values of the S channel are the second default upper and lower limit values, The upper and lower limit values of the V channel are the third default upper and lower limit values.

In one embodiment, when the image of the subject's hand operation is obtained, the surrounding environment of the subject may be photographed together. The first detection module 102 detects the hand by analyzing the difference between the skin color primitive value and the environment primitive value in the image. When other objects with similar skin color primitive values exist in the environment, or when the skin color of the subject is not within the upper and lower limits of the default HSV three channels, or when there are other lighting conditions in the environment, Misidentification of the hand may occur. Therefore, by introducing the analysis module 103, the preset upper and lower limit values of the three HSV channels are updated.

The analysis module 103 is configured to segment a first hand image from the binarized image according to the plurality of the rectangular frames, and analyze the first hand image by using a first preset model.

In one embodiment, the analysis module 103 can segment the image region to be analyzed from the binarized image according to a plurality of the rectangular frames, and the segmented image region is the first image region. hand image, and then input the first hand image into the first preset model for analysis. The first preset model can be a pre-trained hand key point analysis model. For each of the rectangular boxes, the hand key point analysis model can analyze and obtain 21 hand key point coordinates, confidence and hand Feature vector. The confidence level characterizes the robustness of the hand keypoint analysis model. The hand key point coordinates and confidence levels obtained by the analysis of the hand key point analysis model can be transmitted to the tracking module 104, and the hand feature vector can be transmitted to the second preset model (the sorting module 106).

In one embodiment, the hand key point analysis model can be obtained by training based on a preset hand key point training data set, and the hand key point analysis model can be obtained by two cascaded Hourglass networks, several residuals. module and at least one convolutional layer. During model training, the input image of the model can be [256, 256, 3] color RGB three-channel hand image, and the output can be [21, 64, 64] heat image with 21 channels, Then, according to the heat image, the model can obtain the positions of 21 key points in the input image, that is, the coordinates of the 21 hand key points in the input hand image.

In one embodiment, the hand key point analysis model can also calculate a confidence average value according to the confidence degree corresponding to each of the rectangular boxes, and determine whether the confidence degree average value is greater than a preset value, so as to return a positive value. The result or negative result is given to the first detection module 102 . Specifically, if the average confidence value is greater than the preset value, the hand key point analysis model outputs a positive result and feeds it back to the first detection module 102, and then the first detection module 102 will determine the positive result according to the positive result. The result updates the upper and lower limit values of the HSV three channels. If the average confidence value is not greater than the preset value, the hand key point analysis model outputs a negative result and feeds it back to the first detection module 102, and the first detection module 102 will not update the HSV this time The upper and lower limits of the three channels. The preset value can be set and adjusted according to actual needs, which is not limited herein.

The tracking module 104 is used for constructing a tracking data accumulated with tracking time based on the analysis result of the first preset model.

In one embodiment, the first preset model may be a hand key point analysis model, and the analysis results of the first preset model include hand key point coordinates and confidence levels, and the tracking module 104 may The hand key point coordinates and confidence levels obtained by the key point analysis model build and maintain the tracking data accumulated over the tracking time. The tracking data is a structure including hand key point coordinates and confidence levels.

In one embodiment, when the tracking module 104 constructs and maintains the tracking data, the tracking module 104 based on the confidence based on the data of each hand operation image frame received and processed by the hand key point analysis model It determines whether the tracking data generated based on the image frame of the hand operation is reliable. If it is reliable, the tracking module 104 will add the tracking data generated based on the image frame of the hand operation to the tracking data it maintains. , if it is not reliable, the tracking module 104 calls the second detection module 105 to re-detect the image frame of this hand operation. The second detection module 105 may include at least one preset hand series detection model, such as Including pre-trained YOLO model and SSD model. Each preset hand series detection model detects the image frames of this hand operation respectively, and each can obtain a series of rectangular boxes, which are input into the hand key point analysis model for analysis, and the respective hand key points are obtained. Coordinates and confidence. The tracking module 104 comprehensively compares the coordinates and confidence levels of the hand key points corresponding to each group of processing methods, and selects a set of hand key point coordinates and confidence levels that best matches the current tracking data, so as to maintain the information provided by the tracking module 104 . Build tracking data.

In one embodiment, the way for the tracking module 104 to determine, based on the confidence, whether the tracking data generated based on the image frame of the hand operation is reliable is as follows: calculating the average of the confidence and determining whether the average of the confidence exceeds a preset threshold. , if it exceeds, it is determined that the tracking data generated based on the current hand operation image frame is reliable, otherwise it is determined to be unreliable. The preset threshold can be set and adjusted according to actual needs. The rules for the tracking module 104 to select a set of hand key point coordinates and confidence levels that best match the current tracking data may include any one of the following: a. The group with the highest average confidence level; b. The change of the hand key point coordinates of the image frame should not change much. The hand key point coordinates best match the key point coordinates of the front and rear hand operation image frames (for example, the Euclidean distance between the hand key point coordinates is the smallest). A group; c. The combination of rule a and rule b is selected according to the comprehensive result of rule a and rule b by setting different weight coefficients for rule a and rule b.

For example, the second detection module 105 includes the YOLO model and the SSD model, and the tracking module 104 comprehensively compares the coordinates and confidence levels of the hand key points corresponding to the two sets of processing methods. The first group is a series of rectangular frames obtained by detecting the hand job image frame by the YOLO model, and the second group is a series of rectangular frames obtained by detecting the hand job image frame by the SSD model.

In one embodiment, when the tracking module 104 determines based on the confidence that the tracking data generated based on the current hand operation image frame is unreliable, the tracking module 104 can also directly call the tracking data in the second detection module 105. The YOLO model or the SSD model is used to re-detect the image frame of this hand operation, and the corresponding hand key point coordinates and confidence levels are obtained, and maintained in the tracking data (that is, the corresponding YOLO model and the SSD model are not comprehensively compared. hand key point coordinates and confidence).

In one embodiment, when the tracking module 104 constructs the tracking data accumulated with the tracking time based on the hand key point coordinates and confidence levels obtained by the hand key point analysis model, it can also be based on each of the hand key points. The confidence level corresponding to the rectangular frame determines whether the hand key point coordinates obtained by analyzing the first hand image of the hand operation image frame by the hand key point analysis model are reliable. If it is determined to be unreliable, then The tracking module 104 will call a plurality of preset hand series detection models in the second detection module 105 to detect the hand operation image frame respectively to obtain a plurality of rectangular frames representing the region where the hand is located, so as to obtain by segmentation a plurality of second hand images corresponding to the plurality of the preset hand series detection models. The hand key point analysis model then analyzes each of the second hand images to obtain hand key point coordinates, confidence levels and hand feature vectors corresponding to each of the preset hand series detection models . The tracking module 104 comprehensively compares each set of hand key point coordinates and confidence levels obtained by the analysis of the hand key point analysis model to select a set of hand key point coordinates and confidence levels that best matches the tracking data. to update its previously generated tracking data.

It can be understood that the first hand image corresponds to a set of hand key point coordinates and confidence levels, and each second hand image corresponds to a set of hand key point coordinates and confidence levels respectively.

Understandably, the second detection module 105 can be omitted. When the subject's two hands are crossed or only part of the hand area is present, the confidence of the key points of the hand obtained by the analysis module 103 may be low. In this case, the second detection module 105 needs to be used. Among them, the YOLO model and/or the SSD model are used as auxiliary detection.

The sorting module 106 is used for monitoring the tracking data using the second preset model and assigning a hand label to each hand part in the tracking data, so as to classify the tracking data based on the hand label.

In one embodiment, each of the hands corresponds to a unique hand label. The second preset model may be a pre-trained hand ReID model. Organizing module 106 may monitor the tracking data using the hand ReID model and assign the hand label to each hand in the tracking data according to hand feature vectors derived from the hand keypoint analysis model, such that The same hand can have the same hand ID (hand tag). The hand ReID model organizes the tracking data according to the hand ID, so that the data of the same hand ID can be sorted into the same data set, so that the tracking data can be classified based on the hand ID, and the tracking data can be tracked after classification and sorting. The data may be referred to as "ReID data".

In one embodiment, a plurality of images including hands of different people can be pre-recorded as the training data of the hand ReID model, and the output of the second Hourglass network in the hand key point analysis model can be used as the hand ReID. The input of the model, that is, the image frames of the hands of different people are input to two cascaded Hourglass networks, and the output of the second Hourglass network is used as the input data of the hand ReID model. The ReID model is also trained using the Triplet Loss loss function.

In one embodiment, when the hand is removed from the screen during the shooting of the subject's hand operation image, and then returns to the screen, the sorting module 106 uses the second preset model to monitor the situation. The tracking data also assigns hand tags to the hand parts in the tracking data to classify the tracking data based on the hand tags.

The calibration module 107 is used for preprocessing the classified tracking data to obtain the precise data of each hand.

In one embodiment, the classified tracking data is ReID data, and the preprocessing can be a preset data processing method. For example, the preprocessing includes: using a preset outlier removal algorithm to remove the ReID data. the abnormal data, and use the preset interpolation method to perform regression processing on the node where the removed abnormal data is located. The data processed by the calibration module 107 may be called "fine data".

In one embodiment, the abnormal data may be data that significantly deviates from a predetermined normal data value interval. Since the tracking data is based on the data accumulated over the tracking time, in order to avoid data vacancies after the abnormal data is eliminated, the calibration module 107 can use a preset interpolation method to perform regression processing on the node where the eliminated abnormal data is located, so as to supplement the approximate data to The node where the abnormal data is located.

The scoring module 108 is used for scoring the work action of each hand according to the fine data of each hand and the fine data of the reference hand work.

In one embodiment, the scoring module 108 can implement the scoring of the test subject's hand work based on the hand precision data of the standard operating procedure and the hand precision data of the test subject's work flow. The scoring module 108 can use the DTW algorithm to align the precise data of the time series, and score according to the Euclidean distance of the hand key point coordinates of all the hand operation image frames after the alignment. The DTW algorithm can compare the similarity between the two by calculating the Euclidean distance between the hand movement of the standard operating procedure and the hand movement of the test subject. The lower the Euclidean distance is, the more similar the two are. The higher the degree, the higher the score, the higher the Euclidean distance, the lower the similarity, the lower the score.

Specifically, the scoring module 108 uses the DTW algorithm to align the fine data of each hand with the fine data of the reference hand work, and then calculates the hand in the fine data of each hand respectively. The Euclidean distance between the coordinates of the key points and the coordinates of the key points of the hand in the precise data of the reference hand work, and finally, according to the calculation result of the Euclidean distance of each hand, the work action of each hand is scored. This hand movement scoring method is conducive to the evaluation and inspection of workers' hand movements, ensuring the yield and efficiency of the production line, and can be applied to the training of new workers' hand movements. Requirements apply to other scenarios.

In one embodiment, the scoring module 108 uses the DTW algorithm to align the time-series data, and takes into account the time difference between the duration of the work flow of the assessee and the duration of the benchmark work flow into the scoring, which can be based on The Euclidean distance of all key points of all image frames after alignment is scored. The lower the distance, the higher the score, and the higher the distance, the lower the score. However, if the change of the Euclidean distance difference shows a logarithmic trend, and the greater the difference between the two operation processes, the greater the disturbance of the distance difference calculated by the Euclidean distance of the key points, so it can also be evaluated according to dozens of groups. The test data and the hand data of the benchmark operation process are calculated by the Euclidean distance difference to obtain the upper and lower bounds of the Euclidean distance difference, and through logarithmic transformation, the distance difference is mapped to a score of 0 to 100 points, so that the closer it is The closer the score to the upper bound of the distance difference is to zero, the closer to the lower bound of the distance difference, the closer to the full score (100 points).

As shown in FIG. 4 , in one embodiment, the scoring module 108 can be subdivided into a preprocessing unit, a three-dimensional spatial coordinate alignment unit, a dynamic time warping unit, and a logarithmic transformation unit. The preprocessing unit can perform length-centralized processing on the precise data of the subject's hand and the precise data of the benchmark hand work, specifically: the left and right hand key point data appearing in each frame of the benchmark hand work, each segment The length of the key points needs to be consistent in each frame. The lengths between the key points appearing in all frames in the entire benchmark workflow (for example, the length of a total of 20 segments) can be averaged, and then the key points of each frame can be averaged. The length is adjusted to the average length, so that the length of each key point in each frame of the benchmark hand work is the same. The length of each key point in each frame of the hand work is the same to solve the problem of different viewing angles. Then, the length of the left and right hand key points appearing in each frame of the subject's hand work is adjusted to be the same as the reference hand work. The length of the key points is the same. After adjustment, the coordinates of the 21 key points in the left and right hands will change, and this process must be done for each frame of the subject's hand work to overcome the key points caused by the length of the fingers of different people. The problem of coordinate deviation. The three-dimensional space coordinate alignment unit can realize the alignment of the reference hand operation and the zeroth key point data of the subject's hand operation (the starting key point data, such as defining the coordinates of the key point of the palm as the zeroth key point data) Go to the origin of the world coordinate system (x, y, z) = (0, 0, 0), so that the influence of the displacement of the hands can be filtered out, and the score is purely based on hand gestures. The dynamic time warping unit can use the DTW algorithm to align the time series data (the precise data of the subject's hand and the precise data of the reference hand operation), and calculate the Euclidean of all key points of all image frames after alignment The distance is poor. The logarithmic transformation unit is used to map the Euclidean distance difference to a score between 0 and 100 using a logarithmic transformation, which is implemented for scoring the subject's handwork.

FIG. 5 is a flow chart of a method for scoring hand work movements according to an embodiment of the present invention. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

In step S500, a hand operation image is acquired, and the hand operation image is decoded to obtain a hand operation image frame.

In one embodiment, an image recording device (eg, a camera) can be used to record hand work images of one or more designated workers (evaluators) on a designated production line when performing hand operations. The hand operation image can be obtained by communicating with the image recording device. When the hand operation image is acquired, the hand operation image may be decoded to obtain a plurality of hand operation image frames arranged in sequence.

In one embodiment, the hand work image frame may include one or more hands.

Step S502, converting the hand operation image frame into an HSV image, and obtaining a binarized image representing the area where the skin is located and a plurality of rectangular frames representing the area where the hand is located from the HSV image.

In one embodiment, the hand operation image frame may be converted into an HSV image first, and then a binarized image representing the area of the skin and one of the area representing the hand may be obtained from the HSV image. Series of rectangular boxes.

In one embodiment, the binarization representing the area of the skin can be obtained from the HSV image according to the upper and lower limits of the dynamic H channel, the upper and lower limits of the dynamic S channel, and the upper and lower limits of the dynamic V channel. The image and a plurality of rectangular boxes representing the area where the hand is located can avoid the detection of the hand that cannot be adapted to various situations due to the fixed upper and lower limit values of the HSV three channels (ie, the H channel, the S channel and the V channel). problem, to avoid the situation of missed detection or false detection of the hand. The upper and lower limits of the HSV three channels can be dynamically updated. Specifically, the upper and lower limits of the HSV three channels can be updated according to the feedback results of the first preset model described below. For example, when the feedback results given by the first preset model When the result is a positive result, update the upper and lower limits of the H channel, the upper and lower limits of the S channel, and the upper and lower limits of the V channel according to the preset update rules; when the feedback result given by the first preset model is a negative result, no HSV The upper and lower limit values of the three channels are updated. The preset update rule can be preset according to actual usage requirements, and the preset update rule defines an adjustment rule for the upper and lower limit values of each HSV channel in advance.

It can be understood that the upper and lower limit values of the HSV three channels are selected and updated according to the feedback result of the current hand operation image frame output by the first preset model, and the updated upper and lower limit values of the HSV three channels are used for the next hand operation. The working image frame is detected to obtain a binarized image representing the area where the skin is located and a plurality of rectangular frames representing the area where the hand is located in the next working image frame of the hand.

In one embodiment, when the hand operation image frame is the initial frame image, the preset upper and lower limit values of the three HSV channels can be used to obtain a binarized image representing the area of the skin from the HSV image. The image and multiple rectangles representing the area where the hand is located. That is, when the hand operation image frame is the initial frame image, the upper and lower limit values of the H channel are the first default upper and lower limit values, and the upper and lower limit values of the S channel are the second default upper and lower limit values, The upper and lower limit values of the V channel are the third default upper and lower limit values.

In one embodiment, when the image of the subject's hand operation is obtained, the surrounding environment of the subject may be photographed together. The hand is detected by analyzing the difference between the skin color primitive value and the environment primitive value in the image. When other objects with similar skin color primitive values exist in the environment, or when the skin color of the subject is not within the upper and lower limits of the default HSV three channels, or when there are other lighting conditions in the environment, Misidentification of the hand may occur. Therefore, the default upper and lower limit values of the three HSV channels are updated using the feedback result of the current hand operation image frame output by the first preset model (hand key point analysis model).

Step S504 , segment a first hand image from the binarized image according to a plurality of the rectangular frames, and analyze the first hand image by using a first preset model.

In one embodiment, an image area to be analyzed can be segmented from the binarized image according to a plurality of the rectangular frames, and the segmented image area is the first hand image. , and then input the first hand image to the first preset model for analysis. The first preset model can be a pre-trained hand key point analysis model. For each of the rectangular boxes, the hand key point analysis model can analyze and obtain 21 hand key point coordinates, confidence and hand Feature vector. The confidence level characterizes the robustness of the hand keypoint analysis model.

In one embodiment, the hand key point analysis model can be obtained by training based on a preset hand key point training data set, and the hand key point analysis model can be obtained by two cascaded Hourglass networks, several residuals. module and at least one convolutional layer. During model training, the input image of the model can be [256, 256, 3] color RGB three-channel hand image, and the output can be [21, 64, 64] heat image with 21 channels, Then, according to the heat image, the model can obtain the positions of 21 key points in the input image, that is, the coordinates of the 21 hand key points in the input hand image.

In one embodiment, the hand key point analysis model can also calculate a confidence average value according to the confidence degree corresponding to each of the rectangular boxes, and determine whether the confidence degree average value is greater than a preset value, so as to return a positive value. result or negative result. Specifically, if the average confidence value is greater than the preset value, the hand key point analysis model outputs a positive result, and then the upper and lower limit values of the three HSV channels can be updated according to the positive result. If the average confidence value is not greater than the preset value, the hand key point analysis model outputs a negative result, and thus the upper and lower limits of the HSV three channels will not be updated this time. The preset value can be set and adjusted according to actual needs, which is not limited herein.

Step S506 , constructing tracking data accumulated with tracking time based on the analysis result of the first preset model.

In one embodiment, the first preset model can be a hand key point analysis model, and the analysis result of the first preset model includes the hand key point coordinates and confidence, and can be analyzed according to the hand key point. The hand key point coordinates and confidence levels obtained by the model build and maintain the tracking data accumulated over the tracking time. The tracking data is a structure including hand key point coordinates and confidence levels.

In one embodiment, in the process of constructing and maintaining the tracking data, the data of each hand operation image frame received and processed by the hand key point analysis model is also determined based on the confidence level. Whether the tracking data generated by the operation image frame is reliable, if it is reliable, the tracking data generated based on the current hand operation image frame will be added to the tracking data maintained by it. A hand series detection model is set to re-detect the image frame of this hand operation. The plurality of preset hand series detection models may include the pre-trained YOLO model and the SSD model. Each preset hand series detection model detects the image frames of this hand operation respectively, and each can obtain a series of rectangular boxes, which are input into the hand key point analysis model for analysis, and the respective hand key points are obtained. Coordinates and confidence. By comprehensively comparing the hand key point coordinates and confidence levels corresponding to each group of processing methods, a set of hand key point coordinates and confidence levels that best match the current tracking data can be selected for maintenance in the constructed tracking data.

In one embodiment, the method of judging whether the tracking data generated based on the current hand operation image frame is reliable based on the confidence level is: calculating the confidence level mean value, and judging whether the confidence level mean value exceeds a preset threshold, and if it exceeds, Then it is determined that the tracking data generated based on the image frame of this hand operation is reliable, otherwise it is determined to be unreliable. The preset threshold can be set and adjusted according to actual needs. The rules for selecting a set of hand keypoint coordinates and confidences that best match the current tracking data can include any of the following: a. The set with the highest confidence average; b. The change of the coordinates of the hand key points should not change much, and the coordinates of the hand key points match the key point coordinates of the front and rear hand operation image frames most closely (for example, the Euclidean distance between the coordinates of the hand key points is the smallest); c . The combination of rule a and rule b is selected according to the comprehensive result of rule a and rule b by setting different weight coefficients for rule a and rule b.

For example, taking a plurality of preset hand series detection models including the YOLO model and the SSD model as an example, the coordinates and confidence levels of the hand key points corresponding to the two sets of processing methods are comprehensively compared. The first group is a series of rectangular frames obtained by detecting the hand job image frame by the YOLO model, and the second group is a series of rectangular frames obtained by detecting the hand job image frame by the SSD model.

In one embodiment, when it is judged based on the confidence that the tracking data generated based on the image frame of this hand operation is unreliable, the YOLO model or the SSD model can also be directly invoked to conduct the image frame of this hand operation. Detect again, get the corresponding hand key point coordinates and confidence, and maintain it in the tracking data (that is, do not comprehensively compare the hand key point coordinates and confidence corresponding to the YOLO model and the SSD model).

In one embodiment, when the tracking data accumulated with the tracking time is constructed based on the hand key point coordinates and confidence levels obtained by the hand key point analysis model, the corresponding The confidence level judges whether the hand key point coordinates obtained by analyzing the first hand image of the hand operation image frame by the hand key point analysis model are reliable. It is assumed that the hand series detection model respectively detects the hand operation image frames to obtain a plurality of rectangular frames representing the region where the hand is located, so as to obtain a plurality of first hand series corresponding to the plurality of preset hand series detection models. Second-hand image. The hand key point analysis model then analyzes each of the second hand images to obtain hand key point coordinates, confidence levels and hand feature vectors corresponding to each of the preset hand series detection models . By comprehensively comparing each set of hand key point coordinates and confidence levels obtained by the analysis of the hand key point analysis model, a set of hand key point coordinates and confidence levels that best match the tracking data are selected to Update its previously generated tracking data.

It can be understood that the first hand image corresponds to a set of hand key point coordinates and confidence levels, and each second hand image corresponds to a set of hand key point coordinates and confidence levels respectively.

It is understandable that when the subject's two hands are crossed or only part of the hand area appears, the confidence of the hand key points obtained by the hand key point analysis model may be low. In this case, it is necessary to use YOLO. Model and SSD model as auxiliary detection.

Step S508 , monitoring the tracking data using a second preset model and assigning a hand label to each hand part in the tracking data, so as to classify the tracking data based on the hand label.

In one embodiment, each of the hands corresponds to a unique hand label. The second preset model may be a pre-trained hand ReID model. The tracking data can be monitored using the hand ReID model and the hand labels can be assigned to each hand in the tracking data according to hand feature vectors derived from the hand keypoint analysis model, so that the same hand Ability to have the same hand ID (hand tag). The hand ReID model organizes the tracking data according to the hand ID, so that the data of the same hand ID can be sorted into the same data set, so that the tracking data can be classified based on the hand ID, and the tracking data can be tracked after classification and sorting. The data may be referred to as "ReID data".

In one embodiment, a plurality of images including hands of different people can be pre-recorded as the training data of the hand ReID model, and the output of the second Hourglass network in the hand key point analysis model can be used as the hand ReID. The input of the model, that is, the image frames of the hands of different people are input to two cascaded Hourglass networks, and the output of the second Hourglass network is used as the input data of the hand ReID model. The ReID model is also trained using the Triplet Loss loss function.

In one embodiment, when the hand is removed from the screen during the filming of the subject's hand operation and then returns to the screen, the tracking data will be monitored using a second preset model. A hand tag is assigned to the hand part in the tracking data, so as to classify the tracking data based on the hand tag.

Step S510, preprocessing the classified tracking data to obtain the precise data of each hand.

In one embodiment, the classified tracking data is ReID data, and the preprocessing can be a preset data processing method. For example, the preprocessing includes: using a preset outlier removal algorithm to remove the ReID data. the abnormal data, and use the preset interpolation method to perform regression processing on the node where the removed abnormal data is located. The data after elimination and regression processing can be called "fine data".

In one embodiment, the abnormal data may be data that significantly deviates from a predetermined normal data value interval. Since the tracking data is based on the data accumulated over the tracking time, in order to avoid data vacancies caused by the elimination of abnormal data, a preset interpolation method can be used to perform regression processing on the node where the abnormal data is located, so as to supplement the approximate data to the location of the abnormal data. node.

Step S512 , according to the fine data of each hand and the fine data of the reference hand work, score the work action of each hand.

In one embodiment, the scoring of the work movements of each of the hands may refer to scoring the work movements of two hands of each evaluator. The handwork of the subjects can be scored based on the hand proficiency data of the standard operating procedure and the hand proficiency data of the test subject's work flow. The DTW algorithm can be used to align the precise data of the time series, and the scores are scored according to the Euclidean distance of the coordinates of the hand key points of all the hand operation image frames after alignment. The DTW algorithm can compare the similarity between the two by calculating the Euclidean distance between the hand movement of the standard operating procedure and the hand movement of the test subject. The lower the Euclidean distance is, the more similar the two are. The higher the degree, the higher the score, the higher the Euclidean distance, the lower the similarity, the lower the score.

Specifically, the DTW algorithm can be used to align the fine data of each hand with the fine data of the reference hand operation, and then calculate the coordinates of key points of the hand in the fine data of each hand respectively. The Euclidean distance from the coordinates of the key points of the hand in the precise data of the reference hand work, and finally, according to the calculation result of the Euclidean distance of each said hand, the work action of each said hand is scored.

In one embodiment, the DTW algorithm can be used to align the time-series data, and the time difference between the operation process of the subject and the time of the benchmark operation process can be taken into consideration in the scoring. Like the Euclidean distance of all key points of the frame, the lower the distance, the higher the score, and the higher the distance, the lower the score. However, if the change of the Euclidean distance difference shows a logarithmic trend, and the greater the difference between the two operation processes, the greater the disturbance of the distance difference calculated by the Euclidean distance of the key points, so it can also be evaluated according to dozens of groups. The test data and the hand data of the benchmark operation process are calculated by the Euclidean distance difference to obtain the upper and lower bounds of the Euclidean distance difference, and through logarithmic transformation, the distance difference is mapped to a score of 0 to 100 points, so that the closer it is The closer the score to the upper bound of the distance difference is to zero, the closer to the lower bound of the distance difference, the closer to the full score (100 points).

In one embodiment, the following steps can be used to grade the operation movements of each evaluator's two hands: a. The key point data of the left and right hands appearing in each frame of the benchmark hand operation, the key points of each segment The length needs to be consistent in each frame. You can average the lengths between the key points that appear in all frames in the entire benchmark workflow (for example, a total of 20 lengths), and then adjust the length of the key points in each frame. To the average length, so that the length of each key point in each frame of the benchmark hand work is the same. Similarly, the subject's hand work is also processed in the same way to ensure that the subject's hand work The length of each key point in each frame is the same to solve the problem of different perspectives, and then the length of the left and right hand key points appearing in each frame of the subject's hand work is adjusted to match the key point of the reference hand work. The lengths are the same. After adjustment, the coordinates of the 21 key points of the left and right hands will change, and this process must be done for each frame of the subject's hand work to overcome the coordinate deviation of the key points caused by the length of the fingers of different people. b. Align the benchmark hand work with the zeroth keypoint data of the subject's hand work (starting keypoint data, such as defining the keypoint coordinates of the palm as the zeroth keypoint data) to the world The origin of the coordinate system (x, y, z) = (0, 0, 0), so that the influence of the displacement of the hands can be filtered out, and the score is purely based on hand gestures; c. Use the DTW algorithm to evaluate the time series Align the data (the precise data of the subject's hand and the precise data of the reference hand operation), and calculate the Euclidean distance difference of all key points of all image frames after alignment; d. Use logarithmic transformation to convert the Euclidean distance difference Values are mapped to a scale of 0 to 100, which is implemented for scoring the subject's handwork.

The above-mentioned device, method and computer-readable storage medium for evaluating hand movements can process real-time images of the subject's hand movements, accurately locate the key point features of the hand during the hand operation, and compare them with standard hand movements. It compares the work movements of the workers, and intelligently analyzes the difference between the hand work movements of the testees and the standard work movements, and gives the corresponding scores, which is conducive to the evaluation and inspection of workers and improves the yield and efficiency of the production line.

To sum up, the present invention complies with the requirements of an invention patent, and a patent application can be filed in accordance with the law. However, the above descriptions are only the preferred embodiments of the present invention, and the scope of the present invention is not limited to the above-mentioned embodiments, and equivalent modifications or changes made by those who are familiar with the art of the present invention according to the spirit of the present invention are all applicable. Should be covered within the scope of the following patent applications.

10: Memory 20: Processor 30: Scoring program for hand work movements 101: Get Mods 102: The first detection module 103: Analysis Module 104: Tracking Module 105: Second detection module 106: Organize modules 107: Correction module 108: Scoring Module 100: Hand Work Action Scoring Device

FIG. 1 is a functional module diagram of a hand operation action scoring device according to an embodiment of the present invention.

FIG. 2 is a functional module diagram of a hand operation action scoring program according to an embodiment of the present invention.

FIG. 3 is an interactive schematic diagram of a functional module of a hand operation action scoring program according to an embodiment of the present invention.

FIG. 4 is a module diagram of a scoring module according to an embodiment of the present invention.

FIG. 5 is a flow chart of a method for scoring hand operation movements according to an embodiment of the present invention.

Claims (11)

A hand work action scoring method, the method comprising: acquiring a hand work image, and decoding the hand work image to obtain a hand work image frame; Converting the hand operation image frame into an HSV image, and obtaining a binarized image representing the area where the skin is located and a plurality of rectangular frames representing the area where the hand is located from the HSV image; Segmenting a first hand image from the binarized image according to a plurality of the rectangular frames, and analyzing the first hand image by using a first preset model; constructing tracking data accumulated over tracking time based on the analysis result of the first preset model; Monitor the tracking data using a second preset model and assign a hand label to each hand in the tracking data to classify the tracking data based on the hand label, wherein each hand corresponds to Unique hand label; preprocessing the classified tracking data to obtain precise data for each of the hands; and According to the fine data of each hand and the fine data of the reference hand work, the work action of each hand is scored. The method for scoring hand movements according to claim 1, wherein the step of obtaining a binarized image representing the area where the skin is located and a plurality of rectangular frames representing the area where the hand is located from the HSV image comprises: According to the upper and lower limit values of the H channel, the upper and lower limit values of the S channel, and the upper and lower limit values of the V channel, a binarized image representing the area where the skin is located and a plurality of rectangular boxes representing the area where the hand is located are obtained from the HSV image ; Wherein, if the hand operation image frame is the initial frame image, the upper and lower limit values of the H channel are the first default upper and lower limit values, and the upper and lower limit values of the S channel are the second default upper and lower limit values , and the upper and lower limit values of the V channel are the third default upper and lower limit values. The method for scoring hand movements according to claim 2, wherein the first preset model is a pre-trained hand key point analysis model, and the first preset model is used to analyze the first hand image The steps to perform the analysis include: The first hand image is analyzed by using the hand key point analysis model to obtain hand key point coordinates, confidence levels and hand feature vectors corresponding to each of the rectangular boxes. The method for scoring hand movements according to claim 3, further comprising: Calculate a confidence mean value according to the confidence level corresponding to each of the rectangular boxes; judging whether the confidence mean value is greater than a preset value; If the average confidence value is greater than the preset value, updating the upper and lower limit values of the H channel, the upper and lower limit values of the S channel, and the upper and lower limit values of the V channel based on a preset update rule; and If the average confidence value is not greater than the preset value, the upper and lower limit values of the H channel, the upper and lower limit values of the S channel, and the upper and lower limit values of the V channel are not updated. The method for scoring hand movements according to claim 3, wherein the step of constructing a tracking data accumulated with tracking time based on the analysis result of the first preset model comprises: Based on the hand key point coordinates and confidence levels obtained by the hand key point analysis model, the tracking data accumulated with the tracking time is constructed. The method for scoring hand movements according to claim 5, further comprising: Determine whether the hand key point coordinates obtained by analyzing the first hand image by the hand key point analysis model are reliable based on the confidence level corresponding to each of the rectangular frames; If the coordinates of the key points of the hand obtained by the analysis are determined to be unreliable, a plurality of preset hand series detection models are invoked to detect the hand operation image frames respectively to obtain a plurality of rectangular frames representing the region where the hand is located, Obtain a plurality of second hand images corresponding to a plurality of the preset hand series detection models by segmentation, wherein the plurality of the preset hand series detection models include at least the YOLO model and the SSD model; Using the hand key point analysis model to analyze each of the second hand images to obtain hand key point coordinates, confidence levels and hand feature vectors corresponding to each of the preset hand series detection models ;and Comparing each set of hand key point coordinates and confidence levels analyzed by the hand key point analysis model to select a set of hand key point coordinates and confidence levels that best match the tracking data to update the tracking data; Wherein, each of the second hand images corresponds to a set of hand key point coordinates and confidence levels respectively. The method for scoring hand movements according to claim 3, wherein the second preset model is a pre-trained hand ReID model, and the tracking data is monitored by using the second preset model and is the tracking data The steps for assigning a hand label to each hand part include: The tracking data is monitored using the hand ReID model and the hand labels are assigned to each hand in the tracking data according to the hand feature vector. The method for scoring hand movements according to claim 1, wherein the preprocessing includes: using a preset outlier elimination algorithm to eliminate abnormal data in the tracking data, and using a preset interpolation method to eliminate abnormal data. The node where it is located is subjected to regression processing. The method for scoring hand work movements according to claim 1, wherein the step of scoring the work movements of each of the hands according to the fine data of each of the hands and the fine data of the reference hand work comprises the following steps: : aligning the precision data of each of the hands with the precision data of the reference hand operation; respectively calculating the Euclidean distance between the hand key point coordinates in the precision data of each said hand and the hand key point coordinates in the precision data of the reference hand operation; and According to the calculation result of Euclidean distance of each hand, the work action of each hand is graded. A hand operation action scoring device, the device includes a processor and a memory, the memory stores a plurality of computer programs, and the processor is used to execute the computer programs stored in the memory. The steps of any one of the hand work action scoring methods. A computer-readable storage medium, the computer-readable storage medium stores a plurality of instructions, and a plurality of the instructions can be executed by one or more processors, so as to realize the requirements of any one of claim 1 to 9. Describe the steps of the scoring method for hand movements.

Images