CN108764070A - A kind of stroke dividing method and calligraphic copying guidance method based on writing video - Google Patents

Abstract

The invention discloses a stroke segmentation method based on writing video and a calligraphy copying guidance method, wherein, based on the stroke segmentation method of writing video, the convolutional neural network is redesigned, and the convolutional neural network and the cyclic neural network are combined to greatly improve The parameters of the neural network are reduced, and the training speed is accelerated; and the ability to express the spatial characteristics of the video frame is enhanced, and at the same time, the time-domain motion information of calligraphy writing is extracted to achieve high-accuracy fine-grained action recognition. The calligraphy copying instruction method of the present invention adopts the above-mentioned stroke segmentation method based on the writing video, and utilizes the stroke video obtained by stroke segmentation to realize accurate guidance on the calligraphy copying process.

Description

A stroke segmentation method and calligraphy copying guidance method based on writing video

technical field

The invention belongs to the technical field of computer recognition, and relates to a stroke segmentation method based on writing video and a calligraphy copying instruction method.

Background technique

Chinese calligraphy is a unique visual art, the crystallization of thousands of years of Chinese culture, and plays an important role in the inheritance of Chinese traditional culture. As China pays more and more attention to its traditional culture, more and more Chinese and foreigners begin to learn Chinese calligraphy. The basis of calligraphy learning lies in mastering the method of using the brush in the process of writing. As the saying goes, "The magic of calligraphy lies in the use of the brush" (from Kang Youwei in the Qing Dynasty). , which shows the importance of calligraphy strokes in the process of Chinese calligraphy writing.

The common learning method for beginners of calligraphy is to learn by copying. In this process, the first thing that needs to be mastered is the method of using the basic strokes. However, at present, there is a serious lack of teaching resources in Chinese calligraphy, which makes it difficult for many calligraphy learners to obtain high-quality calligraphy teaching, which will become one of the main factors that seriously hinder the learning and promotion of Chinese calligraphy. In order to guide calligraphy learners, the method of recognizing calligraphy actions is adopted.

There is a lack of related work on calligraphy writing action recognition in the prior art (whether it is for static images or dynamic videos). Similar works include recognition of human behavior or actions based on images or videos, mainly including traditional machine learning methods and deep learning methods. With the wide application of deep learning methods, a large number of methods based on convolutional neural networks (CNN) are used for action recognition in static video frames. Compared with traditional machine learning methods, the recognition effect has been significantly improved. Andrej et al. added time convolution to the CNN framework to obtain local motion information in the time domain, which improved by 2% compared to the static CNN recognition method. However, since the time convolution can only obtain temporal information of a few short frames, and in videos with small motion ranges, the static information of several frames of images is almost the same, and it is difficult to obtain temporal motion information. For videos with subtle motion changes, the recognition accuracy is limited.

Contents of the invention

Aiming at the problems existing in the prior art, the object of the present invention is to provide a stroke segmentation method based on writing video, which can accurately segment the writer's work of raising and lowering the pen, so as to achieve the purpose of stroke segmentation.

In order to achieve the above object, the present invention adopts the following technical solutions:

A stroke segmentation method based on writing video, the method is used to divide the writing video into multiple stroke videos according to strokes, comprising the following steps:

Step 1, obtain the writing video of writing single word, this writing video comprises multi-frame image; This multi-frame image is divided into a plurality of video groups, and each video group comprises n frames of continuous images;

Step 2, for each video group, input all images included in the video group into the convolutional neural network, and the convolutional neural network outputs the image space feature vector corresponding to the video group; the convolutional neural network includes two first volumes Product group, two second convolutional groups, three fully connected layers and a softmax layer, wherein the first convolutional group includes two convolutional layers and a pooling layer connected in sequence according to the direction of data transmission, the second convolutional layer The product group includes three convolutional layers and a pooling layer connected in sequence according to the direction of data transmission; three fully connected layers and a softmax layer are connected in sequence according to the direction of data transmission, and three fully connected layers and a softmax layer are convolution neural networks. The last four layers of the network;

Step 3, input the image space feature vector corresponding to the video group into the recurrent neural network, and the recurrent neural network outputs the state of the video group as writing state or non-writing state;

Step 4, select two adjacent video groups in all video groups whose state is a non-writing state, and all video groups between the selected adjacent two video groups are combined to form a stroke video; the state is non-writing All video groups before the first video group in the writing state are combined to form a stroke video; all video groups in the non-writing state after the last video group are combined to form a stroke video A stroke video.

Optionally, the two first convolution groups, the two second convolution groups, the three fully connected layers and the one softmax layer are sequentially connected according to the data transmission direction.

Optionally, the convolution windows of all convolution layers in the two first convolution groups and the two second convolution groups are 3×3, and the pool windows of all pool layers are 2×2;

The number of convolution kernels of the convolution layer in the first first convolution group is 64, the number of convolution kernels of the convolution layer of the second first convolution group is 128, and the first second The number of convolution kernels of the convolution layer in the convolution group is 256, and the number of convolution kernels of the convolution layer of the second convolution group is 512.

The present invention also provides a calligraphy copying instruction method, comprising the following steps:

Step 1, the writing video of the imitator and the standard writing video are divided into a plurality of copying stroke videos and a plurality of standard stroke videos respectively; wherein, the writing video of the imitator and the standard writing video are writing videos of the same word, more There is a one-to-one correspondence between a copy stroke video and multiple standard stroke videos according to the strokes;

Step 2, all adopt trajectory tracking method to copy stroke video and standard stroke video to process, respectively obtain the corresponding trajectory coordinate sequence of copying stroke video and the corresponding trajectory coordinate sequence of standard stroke video;

Step 3, utilize the dynamic time warping method to calculate the similarity between the track coordinate sequence of the copying stroke video and the track coordinate sequence of the standard stroke video corresponding to this copying stroke video;

Step 4, all similarities obtained in step 3 are combined to form a feature vector, and the feature vector is input into the linear regression model, and the linear regression model outputs a score value;

In the step 1, the copying person's writing video and the standard writing video are respectively divided into a plurality of copying stroke videos and a plurality of standard stroke videos, which are obtained according to the above-mentioned stroke segmentation method based on writing videos.

Compared with the prior art, the present invention has the following technical effects: the stroke segmentation method based on the writing video of the present invention redesigns the convolutional neural network, combines the convolutional neural network and the cyclic neural network, and greatly reduces the number of neural networks. The parameters speed up the training speed; and enhance the ability to express the spatial characteristics of the video frame, and at the same time extract the time-domain motion information of calligraphy writing to achieve high-accuracy fine-grained action recognition. The calligraphy copying instruction method of the present invention adopts the above-mentioned stroke segmentation method based on the writing video, and utilizes the stroke video obtained by stroke segmentation to realize accurate guidance on the calligraphy copying process.

The solution of the present invention will be further explained and described in detail in conjunction with the accompanying drawings and specific embodiments.

Description of drawings

Fig. 1 is a flowchart of the stroke segmentation method based on writing video of the present invention.

Detailed ways

The present invention discloses a method for segmenting strokes based on writing videos, as shown in Figure 1. The method is used to segment writing videos into multiple stroke videos according to strokes, and specifically includes the following steps

Step 1, obtain the writing video of writing a single word, this writing video comprises multi-frame image; This multi-frame image is divided into a plurality of video groups, each video group comprises n frames of continuous images, in this embodiment, n=5 .

Step 2, for each video group, input all images included in the video group into the convolutional neural network, and the convolutional neural network outputs the image space feature vector corresponding to the video group; the convolutional neural network includes two first volumes Product group, two second convolutional groups, three fully connected layers and a softmax layer, wherein the first convolutional group includes two convolutional layers and a pooling layer connected in sequence according to the direction of data transmission, the second convolutional layer The product group includes three convolutional layers and a pooling layer connected in sequence according to the direction of data transmission; three fully connected layers and a softmax layer are connected in sequence according to the direction of data transmission, and three fully connected layers and a softmax layer are convolution neural networks. The last four layers of the network.

In step 3, the image space feature vector corresponding to the video group is input into the cyclic neural network, and the cyclic neural network outputs the state of the video group as a writing state or a non-writing state.

Step 4, select two adjacent video groups in all video groups whose state is a non-writing state, and all video groups between the selected adjacent two video groups are combined to form a stroke video; the state is non-writing All video groups before the first video group in the writing state are combined to form a stroke video; all video groups in the non-writing state after the last video group are combined to form a stroke video A stroke video.

The present invention decomposes a large convolution kernel into several small convolution kernels for superposition. Using multiple convolutional layers with smaller convolutional kernels instead of one convolutional layer with larger convolutional kernels makes the network deeper, which not only reduces parameters, but also performs more nonlinear mappings, which can increase the network. Fitting expression ability can not only extract the overall movement information of the brush stroke, but also extract the detailed features of the stroke.

Preferably, in yet another embodiment, two first convolutional groups, two second convolutional groups, three fully connected layers and one softmax layer are sequentially connected according to the data transmission direction. The traditional large-window convolution is replaced by the combination of multiple small-window convolution layers, which makes the network layer deeper and has a stronger ability to express the spatial characteristics of the subtle gestures of calligraphy. On the other hand, it effectively reduces the parameters of the entire network. ,higher efficiency.

Optionally, in yet another embodiment, the convolution windows of all the convolutional layers in the two first convolutional groups and the two second convolutional groups are both 3×3, and the sizes of all pooling layers The pooling windows are all 2×2; the number of convolution kernels of the convolution layer in the first first convolution group is 64, and the number of convolution kernels of the convolution layer of the second first convolution group is The number is 128, the number of convolution kernels of the convolution layer in the first second convolution group is 256, and the number of convolution kernels of the convolution layer of the second second convolution group is 512.

The invention also discloses a calligraphy copying instruction method, comprising the following steps:

Step 1, the writing video of the imitator and the standard writing video are divided into a plurality of copying stroke videos and a plurality of standard stroke videos respectively; wherein, the writing video of the imitator and the standard writing video are writing videos of the same word, more There is a one-to-one correspondence between a copy stroke video and a plurality of standard stroke videos according to the strokes.

Step 2, the video of copied strokes and the video of standard strokes are processed by the trajectory tracking method, and the corresponding trajectory coordinate sequence of the copied stroke video and the corresponding trajectory coordinate sequence of the standard stroke video are obtained respectively; the trajectory tracking method here adopts the TLD trajectory tracking algorithm .

Step 3, using the dynamic time warping method (DTW) to calculate the similarity between the trajectory coordinate sequence of the copied stroke video and the trajectory coordinate sequence of the standard stroke video corresponding to the copied stroke video.

In step 4, all similarities obtained in step 3 are combined to form a feature vector, which is input into a linear regression model, and the linear regression model outputs a score value.

In the above step 1, the copying person's writing video and the standard writing video are respectively divided into a plurality of copying stroke videos and a plurality of standard stroke videos, which are obtained according to the above-mentioned stroke segmentation method based on the writing video.

Example

In this embodiment, a calligraphic writing video database is established, and the method of the present invention and existing methods are tested based on the database. The data source of the calligraphy video database is mainly the high-definition videos recorded by a calligraphy teacher and 6 students in the devil training camp of Mr. Yu Zhonghua of Zhejiang University when they were copying and writing calligraphy. A total of 7 calligraphers have recorded the Yan Ti regular script of Chinese characters such as "Shi, Gu, Da, Fu, Nei, Shang, No, Brother, Zuo, Zi". These ten characters are often practiced by students when they start practicing calligraphy. written Chinese characters, and each person writes each word 20 times, so there are 7*10*20=1400 complete single-character writing video segments in total, the video frame resolution is 1920*1080, and the frame frequency is 50 frames per second. Based on these video data, calligraphic strokes are segmented into stroke writing sub-videos, TLD is used to track brush trajectories, DTW is used to convert features of brush trajectories, and linear regression is used to evaluate calligraphy trajectories.

(1) Divide writing video into stroke video

Apply the method of the present invention (MCNN-LSTM) to more than 1400 videos of 10 Chinese characters, CNN and CNN-LSTM to identify the switching time points between each stroke of each complete character, during the writing process of these Chinese characters , the process between a pen down and a pen up is the process of writing a stroke; then according to the result of the stroke segmentation, the video of the complete character is divided into sub-videos of several strokes. The training data is to randomly select 300 videos from 1400 videos, and then mark the pen-down video frame sequence, the writing process frame sequence and the pen-lifting frame sequence, and the remaining other videos are used as test data. In the test, the input of CNN is a single frame data for video frame recognition, the input of CNN-LSTM and MCNN-LSTM is a frame sequence of 5 consecutive frames, for the recognition of continuous frame sequence, and the action frame sequence of pen lifting or pen down identify. The convolutional layer structure design of MCNN-LSTM is as mentioned above. The hidden layer of LSTM uses 256 neurons, and the stochastic gradient descent method is also used. The bath size is 50 and the learning rate is 0.001. The test results are shown in Table 1:

It can be seen from the data in Table 1 that the recognition rate of CNN has two rows of data. The reason is that this CNN mainly performs single-frame recognition, so the first row uses about 10,000 frames as training, and the remaining 30,000 frames as The frame recognition rate of the test. The second, third, and fourth lines are the recognition rate of the continuous frame sequence calculated after frame label checking, the second line is the video segment recognition rate of five consecutive frames in five consecutive frames, and the third line is five consecutive frames As long as there are 4 frames in the correct video segment recognition rate, the fourth row is the video segment recognition rate as long as 3 frames are correctly recognized in five consecutive frames. From the experimental data, it can be seen that the pure convolutional neural network is difficult to achieve good results in video segment recognition with timing information. Only 60% of the frames in the continuous frames are correctly recognized. The video segment recognition rate is close to the LSTM with timing characteristics. recognition results. In contrast, the features obtained by the continuous frame sequence through the convolutional neural network are used as the input of the LSTM, and then the recognition of the video segment is performed. On the basis of the original, a great performance improvement can be obtained. The signal processing method has excellent properties. In contrast, the MCNN designed in this paper has multiple layers of small convolution kernels superimposed. Compared with simple large convolution kernels or simple small convolution kernels, MCNN can better capture details and overall motion features, and then combine it with LSTMs combined. Therefore, MCNN-LSTM can achieve better recognition effect of video frame sequence. It can be seen from the experimental data that although the difference in the recognition rate between MCNN-LSTM and other methods in 10 words is uneven, it is generally better than the comparison method across the board.

Table 1

(2) Guidance on calligraphy copying

After using MCNN-LSTM to divide the video of complete characters into stroke sub-videos, use FasterRCNN-TLD to track the brush trajectory in these sub-videos, and obtain the trajectory information of each stroke of each person writing each character. Then the trajectory information is converted into the trajectory-dependent similarity feature information required in the linear regression model, and combined with the calligraphy teacher's score on each student's writing process to obtain each person's score when writing each stroke and the entire writing process. word score. Wherein Table 2 shows the comparison between the ratings of each stroke and the ratings of the whole word (the rating is the average value of the ratings written by each person 20 times) and the manual ratings of six students when writing "big", and the ratings of other characters are compared with similar to The manual scoring is their teacher, that is, the template writer's evaluation of each student's writing process. It can be seen from the data in the table that the evaluation system proposed in this paper basically conforms to manual evaluation. It lays the foundation for the automatic evaluation of the calligraphy writing process in the later stage.

Table 2

Claims (4)

1. A stroke segmentation method based on writing video, the method is used to divide writing video into a plurality of stroke videos according to strokes, it is characterized in that, comprising the following steps: Step 1, obtain the writing video of writing single word, this writing video comprises multi-frame image; This multi-frame image is divided into a plurality of video groups, and each video group comprises n frames of continuous images; Step 2, for each video group, input all images included in the video group into the convolutional neural network, and the convolutional neural network outputs the image space feature vector corresponding to the video group; the convolutional neural network includes two first volumes Product group, two second convolutional groups, three fully connected layers and a softmax layer, wherein the first convolutional group includes two convolutional layers and a pooling layer connected in sequence according to the direction of data transmission, the second convolutional layer The product group includes three convolutional layers and a pooling layer connected in sequence according to the direction of data transmission; three fully connected layers and a softmax layer are connected in sequence according to the direction of data transmission, and three fully connected layers and a softmax layer are convolution neural networks. The last four layers of the network; Step 3, input the image space feature vector corresponding to the video group into the recurrent neural network, and the recurrent neural network outputs the state of the video group as writing state or non-writing state; Step 4, select two adjacent video groups in all video groups whose state is a non-writing state, and all video groups between the selected adjacent two video groups are combined to form a stroke video; the state is non-writing All video groups before the first video group in the writing state are combined to form a stroke video; all video groups in the non-writing state after the last video group are combined to form a stroke video A stroke video. 2. the stroke segmentation method based on writing video as claimed in claim 1, is characterized in that, described two first convolution groups, two second convolution groups, three fully connected layers and a softmax layer according to data The transmission direction is connected sequentially. 3. the stroke segmentation method based on writing video as claimed in claim 2, is characterized in that, the size of the convolution window of all convolution layers in described two first convolution groups and two second convolution groups Both are 3×3, and the pooling windows of all pooling layers are 2×2; The number of convolution kernels of the convolution layer in the first first convolution group is 64, the number of convolution kernels of the convolution layer of the second first convolution group is 128, and the first second The number of convolution kernels of the convolution layer in the convolution group is 256, and the number of convolution kernels of the convolution layer of the second convolution group is 512. 4. A calligraphy copying instruction method, comprising the following steps: Step 1, the writing video of the imitator and the standard writing video are divided into a plurality of copying stroke videos and a plurality of standard stroke videos respectively; wherein, the writing video of the imitator and the standard writing video are writing videos of the same word, more There is a one-to-one correspondence between a copy stroke video and multiple standard stroke videos according to the strokes; Step 2, all adopt trajectory tracking method to copy stroke video and standard stroke video to process, respectively obtain the corresponding trajectory coordinate sequence of copying stroke video and the corresponding trajectory coordinate sequence of standard stroke video; Step 3, utilize the dynamic time warping method to calculate the similarity between the track coordinate sequence of the copying stroke video and the track coordinate sequence of the standard stroke video corresponding to this copying stroke video; Step 4, all similarities obtained in step 3 are combined to form a feature vector, and the feature vector is input into the linear regression model, and the linear regression model outputs a score value; In said step 1, the writing video and standard writing video of the imitator are respectively divided into a plurality of copying stroke videos and a plurality of standard stroke videos, according to the writing video based on any one of claims 1-3 The stroke segmentation method is obtained.