CN113011413A

CN113011413A - Method, device and system for processing handwritten image based on smart pen and storage medium

Info

Publication number: CN113011413A
Application number: CN202110409914.3A
Authority: CN
Inventors: 陈铿帆
Original assignee: Shenzhen Yingshuoyun Technology Co ltd
Current assignee: Shenzhen Yingshuoyun Technology Co ltd
Priority date: 2021-04-15
Filing date: 2021-04-15
Publication date: 2021-06-22

Abstract

The invention relates to the field of big data, and provides a method, a device and a system for processing a handwritten image based on an intelligent pen and a storage medium, which are used for improving the access efficiency of the handwritten image of the intelligent pen based on online education. The processing method based on the handwritten image of the intelligent pen comprises the following steps: sequentially performing character recognition, character segmentation, multi-layer stroke order characteristic extraction and character information generation on an intelligent pen handwritten image sequence based on online education through a preset stroke order recognition neural network model to obtain stroke order recognition information; filtering the stroke order recognition information and the intelligent pen handwritten image sequence based on online education to obtain preprocessed data; acquiring the file size of the preprocessed data, and comparing and analyzing the file size with a preset threshold value to obtain a comparison and analysis result; according to the comparison and analysis result, carrying out binary format conversion and/or fragmentation processing on the preprocessed data to obtain data to be stored; and storing the data to be stored to a preset database based on distributed file storage.

Description

Method, device and system for processing handwritten image based on smart pen and storage medium

Technical Field

The invention relates to the field of data processing of big data, in particular to a method, a device and a system for processing a handwritten image based on an intelligent pen and a storage medium.

Background

With the development of information technology and internet of things technology, online education platforms show more development trend. With the increase of users and online interaction of the online education platform, the more the intelligent pen handwritten image data are generated in real time, and the storage of the intelligent pen handwritten image data generated in real time plays an important role in the accuracy, harmony, efficiency and the like of data processing of the online education platform.

At present, the instant generated intelligent pen handwritten image data is generally stored through a relational database, but the storage mode has the defects of poor reading and writing performance, high concurrent reading and writing requirements, inconvenience in table structure expansion and the like of the instant generated intelligent pen handwritten image data, so that the access efficiency of the intelligent pen handwritten image based on online education is low.

Disclosure of Invention

The invention provides a processing method, a device and a system based on a smart pen handwritten image and a storage medium, which are used for improving the access efficiency of the smart pen handwritten image based on online education.

The invention provides a processing method based on a handwritten image by an intelligent pen in a first aspect, which comprises the following steps:

acquiring an intelligent pen handwritten image sequence based on online education, and sequentially performing character recognition, character segmentation, multi-layer stroke order feature extraction and character information generation on the intelligent pen handwritten image sequence based on online education through a preset stroke order recognition neural network model to obtain stroke order recognition information;

filtering the stroke order recognition information and the intelligent pen handwritten image sequence based on online education to obtain preprocessed data;

acquiring the file size of the preprocessed data, and comparing and analyzing the file size with a preset threshold value to obtain a comparison and analysis result;

according to the comparison and analysis result, carrying out binary format conversion and/or fragmentation processing on the preprocessed data to obtain data to be stored;

and storing the data to be stored to a preset database based on distributed file storage.

Optionally, in a first implementation manner of the first aspect of the present invention, the performing binary format conversion and/or fragmentation processing on the preprocessed data according to the comparative analysis result to obtain data to be stored includes:

classifying the preprocessed data according to the comparison and analysis result to obtain data to be converted and data to be fragmented, wherein the data to be converted is used for indicating that the size of the file is smaller than the preset threshold value, and the data to be fragmented is used for indicating that the size of the file is larger than or equal to the preset threshold value;

converting the data to be converted into an array with preset dimensionality to obtain a target array, and writing the target array into a preset binary file to obtain binary file data;

carrying out fragmentation processing on the data to be fragmented and writing the data to be fragmented into a preset fragmentation queue through a preset fragmentation mechanism to obtain target fragmentation data, wherein the fragmentation mechanism comprises the data size and the fragmentation mode of fragmentation;

and determining the binary file data and the target fragment data as data to be stored.

Optionally, in a second implementation manner of the first aspect of the present invention, the storing the data to be stored in a preset database based on distributed file storage includes:

the method comprises the steps of obtaining resource occupation ratios of storage nodes corresponding to a database stored on the basis of a distributed file, and sequencing the storage nodes according to the sequence from large to small of the resource occupation ratios of the storage nodes to obtain a storage node sequence;

caching the data to be stored to obtain cache data, and creating an index of the cache data;

acquiring a user identity identification number corresponding to the online education-based smart pen handwritten image sequence, and matching a corresponding user storage strategy according to the user identity identification number, wherein the user storage strategy comprises storage nodes and a storage proportion;

and storing the cache data for creating the index into a storage node sequence according to a preset user storage strategy.

Optionally, in a third implementation manner of the first aspect of the present invention, the caching the data to be stored to obtain cached data, and creating an index of the cached data includes:

analyzing the data to be stored based on time locality and space locality to obtain initial data;

performing group-associative high-level caching on the initial data to obtain cached data, and performing security measurement detection on the cached data to obtain target data;

and creating an index of the target data through a preset inverted index algorithm.

Optionally, in a fourth implementation manner of the first aspect of the present invention, the sequentially performing character recognition, character segmentation, multi-layer stroke order feature extraction, and character information generation on the online education-based smart pen handwritten image sequence through a preset stroke order recognition neural network model to obtain stroke order recognition information includes:

carrying out character recognition and character segmentation on the intelligent pen handwritten image sequence based on online education through an optical character recognition network layer in a preset stroke order recognition neural network model to obtain character images, wherein one intelligent pen handwritten image corresponds to one or more character images, and the stroke order recognition neural network model comprises the optical character recognition network layer, a target detection network layer and a classification network layer;

performing target frame detection, target frame extraction and multi-scale cross-channel-based convolution fusion on the character image through the target detection network layer to obtain a multi-scale feature map sequence set;

calculating, by the classification network layer, a probability of character matching in the online education-based smart pen handwritten image sequence based on a preset multi-language dictionary and the multi-scale feature map sequence set, wherein the multi-language dictionary comprises strokes and characters of multiple languages;

and determining character information according to the probability to obtain stroke order identification information, wherein the stroke order identification information comprises stroke order information and character information corresponding to the stroke order information.

Optionally, in a fifth implementation manner of the first aspect of the present invention, the filtering the stroke order recognition information and the smart pen handwritten image sequence based on online education to obtain preprocessed data includes:

performing null value identification on the stroke order identification information to obtain null value data, and deleting the null value data in the stroke order identification information to obtain target identification information;

matching a target image set corresponding to the target recognition information from the online education-based smart pen handwritten image sequence;

and creating a corresponding relation between the target image set and the target identification information, and determining the target image set and the target identification information with the corresponding relation as preprocessing data.

Optionally, in a sixth implementation manner of the first aspect of the present invention, after storing the data to be stored in a preset database based on distributed file storage, the method includes:

and acquiring the data condition of the database based on the distributed file storage, and configuring and connecting newly added nodes for the database based on the distributed file storage according to the data condition.

The invention provides a processing device based on a handwritten image by a smart pen, which comprises:

the recognition module is used for acquiring an intelligent pen handwritten image sequence based on online education, sequentially performing character recognition, character segmentation, multi-layer stroke order feature extraction and character information generation on the intelligent pen handwritten image sequence based on the online education through a preset stroke order recognition neural network model, and obtaining stroke order recognition information;

the filtering module is used for filtering the stroke order identification information and the intelligent pen handwritten image sequence based on the online education to obtain preprocessed data;

the analysis module is used for acquiring the file size of the preprocessed data, and comparing and analyzing the file size with a preset threshold value to obtain a comparison and analysis result;

the conversion fragmentation module is used for carrying out binary format conversion and/or fragmentation processing on the preprocessed data according to the comparison analysis result to obtain data to be stored;

and the storage module is used for storing the data to be stored to a preset database based on distributed file storage.

Optionally, in a first implementation manner of the second aspect of the present invention, the conversion fragmentation module is specifically configured to:

Optionally, in a second implementation manner of the second aspect of the present invention, the storage module includes:

the sorting unit is used for acquiring the resource occupation ratio of each storage node corresponding to the database stored based on the distributed file, and sorting the storage nodes according to the sequence from large to small of the resource occupation ratio value of each storage node to obtain a storage node sequence;

the cache creating unit is used for caching the data to be stored to obtain cache data and creating an index of the cache data;

the matching unit is used for acquiring a user identity identification number corresponding to the online education-based smart pen handwritten image sequence and matching a corresponding user storage strategy according to the user identity identification number, wherein the user storage strategy comprises storage nodes and a storage proportion;

and the storage unit is used for storing the cache data for creating the index into the storage node sequence according to a preset user storage strategy.

Optionally, in a third implementation manner of the second aspect of the present invention, the cache creating unit is specifically configured to:

Optionally, in a fourth implementation manner of the second aspect of the present invention, the identification module is specifically configured to:

calculating the probability of character matching in the intelligent pen handwriting image sequence based on a preset multi-language dictionary and the multi-scale feature map sequence set through the classification network layer, wherein the multi-language dictionary comprises strokes and characters of multiple languages;

Optionally, in a fifth implementation manner of the second aspect of the present invention, the filtering module is specifically configured to:

Optionally, in a sixth implementation manner of the second aspect of the present invention, the processing apparatus based on a smart pen handwritten image further includes:

and the configuration connection module is used for acquiring the data condition of the database based on the distributed file storage and performing configuration and connection of the newly added nodes on the database based on the distributed file storage according to the data condition.

The invention provides a processing system based on a smart pen handwritten image, which comprises a smart pen, a server and a database based on distributed file storage;

the server is connected with the intelligent pen and the database based on distributed file storage through a network;

the intelligent pen comprises a sound-sensitive switch module, a pressure sensor, an optical character recognition scanner, a processor, an image converter, a wireless network module, a temperature sensor, a tracking module and a wireless charging module; the sound-sensitive switch module is used for detecting the knocking sound on the intelligent pen and starting the intelligent pen according to the knocking sound;

the pressure sensor is used for detecting the dot matrix pressure corresponding to the intelligent pen, acquiring the writing parameters of the intelligent pen and starting the optical character recognition scanner according to the dot matrix pressure;

the optical character recognition scanner is used for shooting or scanning the content of the intelligent pen during manual writing to obtain video information or image information;

the image converter is used for generating a dot matrix image sequence based on online education for the video information or the image information;

the processor is used for marking the intelligent pen writing parameters in the dot matrix image sequence based on the online education to obtain an intelligent pen handwriting image sequence based on the online education;

the wireless network module is used for sending the intelligent pen handwriting image sequence based on online education to the server;

the temperature sensor is used for detecting the temperature value of the intelligent pen;

the tracking module is used for positioning the position of the intelligent pen and calculating the positioned path;

the wireless charging module is used for wirelessly charging the intelligent pen;

the server comprises a data transmission component and a processing module; the data transmission component is used for receiving a smart pen handwriting image sequence based on online education sent by the smart pen;

the processing module is used for sequentially identifying, filtering, analyzing and converting the stroke order identification information into fragments to obtain data to be stored;

the data transmission component is used for sending the data to be stored to the database based on the distributed file storage;

the database based on distributed file storage is used for storing data to be stored after the server processes the intelligent pen handwritten image sequence based on online education sent by the intelligent pen.

A fourth aspect of the present invention provides a computer-readable storage medium having stored therein instructions, which, when run on a computer, cause the computer to execute the above-mentioned processing method based on a smart pen handwritten image.

According to the technical scheme, an intelligent pen handwritten image sequence based on online education is obtained, character recognition, character segmentation, multi-layer stroke order feature extraction and character information generation are sequentially carried out on the intelligent pen handwritten image sequence based on the online education through a preset stroke order recognition neural network model, and stroke order recognition information is obtained; filtering the stroke order recognition information and the intelligent pen handwritten image sequence based on online education to obtain preprocessed data; acquiring the file size of the preprocessed data, and comparing and analyzing the file size with a preset threshold value to obtain a comparison and analysis result; according to the comparison and analysis result, carrying out binary format conversion and/or fragmentation processing on the preprocessed data to obtain data to be stored; and storing the data to be stored to a preset database based on distributed file storage. In the embodiment of the invention, character recognition, character segmentation, multi-layer stroke order characteristic extraction, character information generation and filtering processing are carried out on the intelligent pen hand-written image sequence based on online education, and binary format conversion and/or fragment processing are carried out on preprocessed data according to a comparison analysis result to obtain data to be stored; the method has the advantages that the data to be stored are stored in a preset database based on distributed file storage, unified step-by-step storage of the data to be stored is achieved, storage overhead of the data to be stored is reduced, storage load balance of the data to be stored is achieved, read-write performance and table structure expansion convenience of the data to be stored which are generated immediately are improved, high concurrent read-write requirements of the data to be stored which are generated immediately are reduced, storage accuracy of the data to be stored is improved, and accordingly access efficiency of handwritten images of the smart pen based on online education is improved.

Drawings

FIG. 1 is a diagram illustrating an embodiment of a smart pen-based handwritten image processing method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of another embodiment of a processing method based on a smart pen handwritten image in the embodiment of the invention;

FIG. 3 is a schematic diagram of an embodiment of a processing device based on a smart pen handwritten image according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of another embodiment of a processing device based on a smart pen handwritten image according to an embodiment of the present invention;

fig. 5 is a schematic diagram of an embodiment of a processing system based on a smart pen handwritten image in the embodiment of the present invention.

Detailed Description

The embodiment of the invention provides a method, a device and a system for processing a handwritten image based on an intelligent pen and a storage medium, which improve the access efficiency of the handwritten image of the intelligent pen based on online education.

The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

For convenience of understanding, a specific flow of an embodiment of the present invention is described below, and referring to fig. 1, an embodiment of a method for processing a handwritten image based on a smart pen in an embodiment of the present invention includes:

101. the method comprises the steps of obtaining an intelligent pen handwritten image sequence based on online education, sequentially carrying out character recognition, character segmentation, multi-layer stroke order feature extraction and character information generation on the intelligent pen handwritten image sequence based on the online education through a preset stroke order recognition neural network model, and obtaining stroke order recognition information.

It is understood that the executing subject of the present invention may be a processing device based on a smart pen handwriting image, and may also be a terminal or a server, which is not limited herein. The embodiment of the present invention is described by taking a server as an execution subject.

When a user carries out online question answering of online education based on an online education platform and manually writes on preset input equipment or an input screen or paper through a preset intelligent pen, the intelligent pen shoots or scans the content during the manual writing and generates an intelligent pen handwritten image sequence based on the online education, the sequence of the intelligent pen handwritten image based on the online education is a time sequence of the manual writing through the intelligent pen, and the intelligent pen handwritten image in the intelligent pen handwritten image sequence based on the online education is a dot matrix image for marking dot matrix coordinates, handwriting paths, pressure parameters, writing speeds and other intelligent pen writing parameters.

And the intelligent pen sends the generated intelligent pen handwritten image sequence based on online education to a server (corresponding to the cloud platform), and after receiving the intelligent pen handwritten image sequence based on online education, the server (corresponding to the cloud platform) performs data cleaning, denoising, image size conversion, image enhancement and filtering (including mean filtering, median filtering, image bilateral filtering and/or Gaussian filtering) on the intelligent pen handwritten image sequence based on online education to obtain the processed intelligent pen handwritten image sequence.

Calling a pre-trained stroke order recognition neural network model, recognizing characters in each smart pen handwritten image in the processed smart pen handwritten image sequence and segmenting the region where the characters are located according to the sequence of generation time to obtain character region images, and recognizing a multilayer feature extraction network in the neural network model through the stroke order, wherein the multilayer feature extraction network comprises but is not limited to a surf network and a feature extraction network of a scale invariant feature transformation algorithmTaking an algorithm (original fast and rotated brief, ORB) network, a Local Binary Pattern (LBP) network, a HAAR algorithm HAAR network and a Gated Repeat Unit (GRU) network, wherein the logical relationship among the multilayer feature extraction networks can be a sequential series connection relationship, namely the output of the previous network is the input of the next network, or a parallel relationship, namely the input of each network is the same, performing multi-level feature map extraction on character region images to obtain a plurality of feature maps so as to realize multi-level stroke order feature extraction, fusing the feature maps through a preset attention mechanism to obtain the fusion feature of each intelligent pen handwritten image, splicing and synthesizing the fusion feature of each intelligent pen handwritten image according to the sequence of the generation time to obtain the synthesized feature, and matching corresponding character information (the character information comprises the stroke order information and the stroke order information which corresponds to the synthesized feature) from a preset database according to the synthesized feature Text information), that is, character information generation, to obtain stroke order recognition information, where the stroke order recognition information includes stroke order information corresponding to each smart pen handwritten image and text information integrated based on a sequence of smart pen handwritten images of online education, for example: on-line education-based stroke order information vertical and horizontal corresponding to intelligent pen handwritten image sequence and

the comprehensive character information is 'human', wherein the stroke order recognition neural network model can perform optical character recognition, stroke order feature extraction, character information matching and generation and the like.

102. And filtering the stroke order recognition information and the intelligent pen handwritten image sequence based on online education to obtain preprocessed data.

The server calculates the similarity between the stroke order identification information and a preset stroke order template through a similarity calculation method, compares the similarity with a preset target value for analysis, classifies the stroke order identification information with the similarity smaller than or equal to the preset target value into information to be deleted, filters the information to be deleted in the stroke order identification information to obtain target information, and obtains a target image sequence corresponding to the target information from an intelligent pen handwriting image sequence based on online education to obtain preprocessing data, wherein the preprocessing data can comprise the target image sequence and the target information corresponding to the target image sequence, and the preprocessing data can only comprise the target image sequence.

Or after the server acquires a target image sequence corresponding to the target information from the smart pen handwritten image sequence based on online education, filtering (including mean filtering, median filtering, image bilateral filtering and/or Gaussian filtering) is carried out on each target image in the target image sequence to obtain preprocessed data.

103. And acquiring the file size of the preprocessed data, and comparing and analyzing the file size with a preset threshold value to obtain a comparison and analysis result.

The server calls a preset file size calculation script to calculate the file size of each data (a target image sequence and/or target information corresponding to the target image sequence) in the preprocessed data, judges whether the file size of each data is larger than a preset threshold value, if so, obtains a judgment result larger than the preset threshold value, determines the corresponding data as data to be fragmented, if not, obtains a judgment result smaller than or equal to the preset threshold value, and determines the corresponding data as data to be converted, thereby obtaining a comparative analysis result, wherein the comparative analysis result comprises the judgment result and the data corresponding to the judgment result (the data comprises the target image sequence and/or the target information corresponding to the target image sequence).

104. And according to the comparison and analysis result, carrying out binary format conversion and/or fragmentation processing on the preprocessed data to obtain the data to be stored.

And after the server obtains the comparative analysis result, acquiring the data to be fragmented and the data to be converted in the comparative analysis result, wherein the data to be fragmented is null value when the comparative analysis result does not have a judgment result larger than the judgment result, and the data to be converted is null value when the comparative analysis result does not have a judgment result smaller than or equal to the judgment result.

The server divides the data to be sliced into blocks with preset number according to the sequence order of the target image sequence through a preset file server gridfs based on a preset slicing mechanism and a copying mechanism to obtain target slicing data, wherein the size of each block is the same.

If the data to be converted has a target image sequence, the server calls a preset binary image converter to convert the target image sequence into a binary data stream, if the data to be converted has target information corresponding to the target image sequence, the target information is written into a preset object (such as bson. binary) to obtain written data, and binary conversion is performed on the written data through a preset binary algorithm to obtain binary file data.

And when the comparative analysis result has the data to be fragmented or the data to be converted and the data to be fragmented or the data to be converted is not null, determining the target fragmented data or the binary file data as the data to be stored.

105. And storing the data to be stored to a preset database based on distributed file storage.

The server stores the data to be stored into each storage node in a preset distributed file storage-based database MongonDB according to a preset storage node proportion, for example: the database MongonDB based on distributed file storage has a storage node A, a storage node B, a storage node C and a storage node D, wherein the storage node A, the storage node B, the storage node C and the storage node D are respectively 40%, 30%, 20% and 10%, 40% of data to be stored is stored in the storage node A of the database MongonDB, 30% of the data to be stored is stored in the storage node B of the database MongonDB, 20% of the data to be stored is stored in the storage node C of the database MongonDB, and 10% of the data to be stored is stored in the storage node D of the database MongonDB, wherein 40% of the data to be stored can be 40% of the data before the data to be stored is sorted, or can be 40% of the data to be stored randomly, and 30% of the data to be stored, 20% of the data to be stored and 10% of the data to be stored can be obtained in the same manner.

Optionally, a block chain is pre-created in the database based on the distributed file storage, where the block chain includes a plurality of block chain nodes, and each storage node in the database based on the distributed file storage corresponds to one or more block chain nodes. After obtaining data to be stored, a server acquires corresponding target storage nodes in a database based on distributed file storage, the number of the target storage nodes comprises one or more, the data to be stored is averagely stored to the target storage nodes, the target storage nodes generate storage requests according to the data to be stored and send the storage requests to corresponding block chain link points, after receiving the storage requests, the block chain link points sequentially analyze, verify and execute the storage requests to generate results, and the execution results are sent to the target storage nodes, so that the database based on distributed file storage and combined with the block chain stores the data to be stored, and the data to be stored has high availability, higher fault-tolerant capability and better safety.

Optionally, after the server stores the data to be stored in a preset database stored based on a distributed file, when a processing instruction is received, traversing a preset robot tree according to the processing instruction to obtain a corresponding robot, calling the robot, retrieving and reading the data to be stored in the database stored based on the distributed file to obtain the read data to be stored, and matching a pre-stored stroke order video and a stroke order dynamic picture in the database according to the read data to be stored to obtain corresponding target stroke order information, or performing stroke order error information identification and feedback on the read data to be stored by the server.

In the embodiment of the invention, character recognition, character segmentation, multi-layer stroke order characteristic extraction, character information generation and filtering processing are carried out on the intelligent pen hand-written image sequence based on online education, and binary format conversion and/or fragment processing are carried out on preprocessed data according to a comparison analysis result to obtain data to be stored; the method has the advantages that the data to be stored are stored in a preset database based on distributed file storage, unified step-by-step storage of the data to be stored is achieved, storage overhead of the data to be stored is reduced, storage load balance of the data to be stored is achieved, read-write performance and table structure expansion convenience of the data to be stored which are generated immediately are improved, high concurrent read-write requirements of the data to be stored which are generated immediately are reduced, storage accuracy of the data to be stored is improved, and accordingly access efficiency of handwritten images of the smart pen based on online education is improved.

Referring to fig. 2, another embodiment of a method for processing a handwritten image based on a smart pen according to an embodiment of the present invention includes:

201. the method comprises the steps of obtaining an intelligent pen handwritten image sequence based on online education, sequentially carrying out character recognition, character segmentation, multi-layer stroke order feature extraction and character information generation on the intelligent pen handwritten image sequence based on the online education through a preset stroke order recognition neural network model, and obtaining stroke order recognition information.

When a user carries out online question answering of online education based on an online education platform and manually writes on a preset input device or an input screen or paper through a preset intelligent pen, an optical character recognition scanner in the intelligent pen shoots or scans the content during manual writing to generate an intelligent pen handwritten video, the intelligent pen handwritten video is converted into an intelligent pen handwritten image sequence based on the online education through a preset conversion tool or a preset conversion script, the intelligent pen handwritten image sequence based on the online education is sent to a server, and the sequence of the intelligent pen handwritten image represents the time sequence of manual writing through the intelligent pen.

Specifically, the server identifies the optical character identification network layer in the neural network model through a preset stroke order, performs character identification and character segmentation on an intelligent pen handwritten image sequence based on online education to obtain character images, one intelligent pen handwritten image corresponds to one or more character images, and the neural network model for stroke order identification comprises the optical character identification network layer, a target detection network layer and a classification network layer; through a target detection network layer, carrying out target frame detection, target frame extraction and multi-scale cross-channel-based convolution fusion on the character image to obtain a multi-scale feature image sequence set; calculating the probability of character matching in the intelligent pen handwritten image sequence based on online education through a classification network layer based on a preset multi-language dictionary and a multi-scale feature map sequence set, wherein the multi-language dictionary comprises strokes and characters of multiple languages; and determining character information according to the probability to obtain stroke order identification information, wherein the stroke order identification information comprises stroke order information and character information corresponding to the stroke order information.

After the server obtains an intelligent pen handwriting image sequence based on online education, a preset stroke order Recognition neural network model is called, the stroke order Recognition neural network model comprises an Optical Character Recognition network layer, an object detection network layer and a classification network layer, wherein the Optical Character Recognition network layer can be formed by an Optical Character Recognition (OCR) algorithm, the object detection network layer can be formed by a combination of a fast convolutional neural network (fast-convolutional neural networks, Faster R-CNN), a single-stage object detection network RetinaNet and a single-stage object detection algorithm ET-YOLOv3, and the classification network layer can be formed by a plurality of classifiers.

The server carries out character recognition and character segmentation on each intelligent pen hand-written image in an intelligent pen hand-written image sequence based on online education through an optical character recognition network layer to obtain a character image of each intelligent pen hand-written image, wherein the number of the character images comprises one or more, an object detection network layer is used for sequentially carrying out anchor frame marking, cross-over ratio calculation, object frame determination and object frame segmentation on each character image based on a plurality of preset anchor frames to obtain an object frame image of each intelligent pen hand-written image, the number of the object frame images comprises one or more, and the object frame image is based on convolution activation layers (comprising 5 layers of convolution layers, 1 layer of normalization BN layer and 1 layer of activation layer), convolution activation blocks (comprising 1 layer of convolution layer, 1 layer of normalization BN layer and 1 layer of activation layer), convolution activation blocks (comprising 2 layers), an upper sampling layer and a fusion layer, and sequentially carrying out multi-scale cross-channel feature extraction on the target frame image of each intelligent pen handwritten image to obtain a multi-scale feature map, and arranging the multi-scale feature maps of all the intelligent pen handwritten images according to the sequence of the generation time to obtain a multi-scale feature map sequence set.

Through a classification network layer, based on a preset multi-language dictionary (the dictionary comprises strokes and characters) and a multi-scale feature map sequence set, calculating the probability of character matching in an intelligent pen handwriting image sequence based on online education, sequencing the probabilities from large to small, and determining the stroke order and/or words in the multi-language dictionary corresponding to the first sequenced probability as stroke order identification information. Through the execution process, accuracy and efficiency of stroke order identification information identification are improved.

202. And filtering the stroke order recognition information and the intelligent pen handwritten image sequence based on online education to obtain preprocessed data.

Specifically, the server performs null value identification on the stroke order identification information to obtain null value data, and deletes the null value data in the stroke order identification information to obtain target identification information; matching a target image set corresponding to target recognition information from a smart pen handwritten image sequence based on online education; and creating a corresponding relation between the target image set and the target identification information, and determining the target image set and the target identification information with the corresponding relation as preprocessing data.

The server judges whether preset null value fields and null value field values exist in the stroke order identification information, if not, execution is stopped, if yes, the corresponding stroke order identification information is determined to be null value data, the null value data are removed from the stroke order identification information to obtain target identification information, and the intelligent pen handwritten image sequence based on online education is classified according to the target identification information to obtain a corresponding target image set; and creating a corresponding relation between the target image set and the target identification information, and determining the target image set and the target identification information with the corresponding relation as preprocessing data.

Or, the server matches the order of writing recognition information with the sequence of smart pen handwritten images based on online education through a preset filtering rule tree to obtain the preprocessed data after filtering, wherein the filtering rule tree includes but is not limited to completeness, safety and reprocessing recognition rules of the order of writing recognition information and the sequence of smart pen handwritten images based on online education, and the order of writing recognition information and the sequence of smart pen handwritten images based on online education are filtered, so that the smart pen handwritten images and order recognition information which do not need to be stored are reduced, the amount of stored data is reduced, the accuracy of the stored smart pen handwritten images and order recognition information is improved, the follow-up operation and calculation of the smart pen handwritten images and order recognition information which do not need to be stored are reduced, and the access efficiency is improved conveniently.

203. And acquiring the file size of the preprocessed data, and comparing and analyzing the file size with a preset threshold value to obtain a comparison and analysis result.

The server calculates the file size of each data (the target image sequence and/or the target information corresponding to the target image sequence) in the preprocessed data by calling a preset file size calculation script, judges whether the file size of each data is larger than a preset threshold value, if so, obtains a judgment result larger than the preset threshold value, and if not, obtains a judgment result smaller than or equal to the preset threshold value, so as to obtain a comparative analysis result, wherein the comparative analysis result comprises the judgment result and the data corresponding to the judgment result (the data comprises the target image sequence and/or the target information corresponding to the target image sequence).

204. And according to the comparison and analysis result, carrying out binary format conversion and/or fragmentation processing on the preprocessed data to obtain the data to be stored.

Specifically, the server classifies the preprocessed data according to the comparison and analysis result to obtain data to be converted and data to be fragmented, wherein the data to be converted is used for indicating that the size of the file is smaller than a preset threshold value, and the data to be fragmented is used for indicating that the size of the file is larger than or equal to the preset threshold value; converting data to be converted into an array with preset dimensionality to obtain a target array, and writing the target array into a preset binary file to obtain binary file data; the method comprises the steps that data to be fragmented are fragmented and written into a preset fragmentation queue through a preset fragmentation mechanism to obtain target fragmentation data, wherein the fragmentation mechanism comprises the data size and the fragmentation mode of fragmentation; and determining the binary file data and the target fragment data as data to be stored.

The data to be converted or the data to be fragmented can be null. The server converts the data to be converted into an array with preset dimensionality (being one-dimensional) through a preset matrix transformation function reshape function to obtain a one-dimensional array, converts the one-dimensional array into a two-dimensional array with preset rows and columns to obtain a target array, and writes the target data into a preset binary file to obtain binary file data.

The method comprises the steps that a server identifies a fragment field interval (a fragment field interval comprises a time period field) in data to be fragmented, time fragmentation processing is carried out on the data to be fragmented through a time main key fragmentation mode in a preset fragmentation mechanism based on the fragment field interval and the preset data size of fragments to obtain initial fragment data, the hash value of a preset field (the preset field is a field corresponding to the preset fragmentation point) in the initial fragment data is calculated, modulo is carried out on the hash value and the preset total number of fragments to obtain a fragment dimension, and fragmentation is carried out on the initial fragment data according to the fragment dimension, the preset data size of the fragments and a preset consistent hash algorithm to obtain target fragment data.

205. And storing the data to be stored to a preset database based on distributed file storage.

Specifically, the server acquires the resource occupation ratio of each storage node corresponding to a database stored based on the distributed file, and sorts the storage nodes according to the sequence from large to small of the value of the resource occupation ratio of each storage node to obtain a storage node sequence; caching data to be stored to obtain cache data, and creating an index of the cache data; acquiring a user identity identification number corresponding to a handwritten image sequence of the intelligent pen based on online education, and matching a corresponding user storage strategy according to the user identity identification number, wherein the user storage strategy comprises storage nodes and a storage proportion; and storing the cache data for creating the index into a storage node sequence according to a preset user storage strategy.

The resource occupation ratio includes a Central Processing Unit (CPU) resource occupation ratio, a storage space data occupation ratio, and an idle resource occupation ratio. The server monitors, calculates and reads the resource conditions of the storage nodes corresponding to the database stored based on the distributed file in real time through a real-time monitoring mechanism, so as to obtain the resource ratio, and sorts the storage nodes according to the sequence of the resource ratio from small to large, so as to obtain a storage node sequence. Caching data to be stored through a preset high-speed cache to obtain cache data, and establishing an index of the cache data through a preset Hash algorithm. The server traverses a preset strategy tree according to the user identity identification number to obtain a corresponding user storage strategy, wherein the user storage strategy comprises storage nodes and storage proportions, a corresponding relation is established between the storage nodes and the storage proportions, the corresponding target storage nodes are matched from a storage node sequence according to the storage nodes in the user storage strategy, and the cache data of the created index is stored into the corresponding target storage nodes according to the storage proportions. Through the execution process, the data to be stored can be stored and read quickly, orderly and effectively, and therefore the access efficiency of the handwritten image of the intelligent pen based on online education is improved.

Specifically, the server analyzes data to be stored based on time locality and space locality to obtain initial data; performing group-associative high-level caching on the initial data to obtain cached data, and performing security measurement detection on the cached data to obtain target data; and creating an index of the target data through a preset inverted index algorithm.

The method comprises the steps that a server obtains access time of data to be stored, whether the access time is preset access time is judged, if yes, the data to be stored are determined to be initial data, if not, execution is stopped, access data related to the data to be stored are predicted through a preset prediction model, related access data are obtained, the data to be stored and the related access data are determined to be the initial data, and therefore analysis based on time locality and space locality of the data to be stored is achieved.

The server performs group selection, line matching and word selection on the initial data to obtain cache data so as to realize group-associative high-level caching of the initial data, and performs security measurement detection on parameters such as path traversal correctness, buffer overflow, accuracy of formatted character strings, review condition of an encryption algorithm, release condition of resources and the like on the cache data, so as to obtain target data.

206. And acquiring the data condition of the database based on the distributed file storage, and configuring and connecting newly added nodes for the database based on the distributed file storage according to the data condition.

The server starts a timer, performs timing through the timer, and counts the access quantity of the data to be stored based on a preset time period in the database based on distributed file storage and the data storage quantity of the storage node corresponding to the data to be stored when the timer times to a preset time; and judging whether the access quantity and the data storage quantity of the data to be stored meet preset addition conditions, if so, configuring and connecting new nodes for the database based on the distributed file storage, and if not, not configuring and connecting the new nodes for the database based on the distributed file storage, wherein the new nodes can be database servers or virtual storage nodes. The storage pressure of the database based on the distributed file storage is shared by the newly added nodes, so that the database based on the distributed file storage can quickly, effectively and accurately access the smart pen handwritten image based on the online education, and the access efficiency of the smart pen handwritten image based on the online education is improved.

In the embodiment of the invention, the data to be stored is stored in a unified and step-by-step manner, the storage cost of the data to be stored is reduced, the storage load balance of the data to be stored is realized, the read-write performance and the table structure expansion convenience of the data to be stored which is generated immediately are improved, the high concurrent read-write requirement of the data to be stored which is generated immediately is reduced, and the storage accuracy of the data to be stored is improved, so that the access efficiency of the handwritten image of the intelligent pen based on online education is improved, the storage pressure of the database based on distributed file storage is shared by the newly added nodes, the access of the handwritten image of the intelligent pen based on online education by the database based on distributed file storage is facilitated, and the access efficiency of the handwritten image of the intelligent pen based on online education is improved.

In the above description of the processing method based on the smart pen handwritten image in the embodiment of the present invention, referring to fig. 3, a processing device based on the smart pen handwritten image in the embodiment of the present invention is described below, and an embodiment of the processing device based on the smart pen handwritten image in the embodiment of the present invention includes:

the recognition module 301 is configured to obtain an intelligent pen handwritten image sequence based on online education, and sequentially perform character recognition, character segmentation, multi-layer stroke order feature extraction, and character information generation on the intelligent pen handwritten image sequence based on online education through a preset stroke order recognition neural network model to obtain stroke order recognition information;

a filtering module 302, configured to filter the stroke order recognition information and the smart pen handwritten image sequence based on online education to obtain preprocessed data;

the analysis module 303 is configured to obtain a file size of the preprocessed data, and compare and analyze the file size with a preset threshold to obtain a comparison and analysis result;

the conversion fragmentation module 304 is configured to perform binary format conversion and/or fragmentation processing on the preprocessed data according to the comparison analysis result to obtain data to be stored;

the storage module 305 is configured to store data to be stored in a preset database based on distributed file storage.

The function implementation of each module in the processing device based on the smart pen handwritten image corresponds to each step in the processing method embodiment based on the smart pen handwritten image, and the function and implementation process are not described in detail herein.

Referring to fig. 4, another embodiment of a processing device based on a smart pen handwriting image according to an embodiment of the present invention includes:

a storage module 305, configured to store data to be stored in a preset database based on distributed file storage;

and the configuration connection module 306 is configured to acquire a data status of the database based on the distributed file storage, and perform configuration and connection of the newly added node on the database based on the distributed file storage according to the data status.

Optionally, the conversion fragmentation module 304 may be further specifically configured to:

classifying the preprocessed data according to the comparison and analysis result to obtain data to be converted and data to be fragmented, wherein the data to be converted is used for indicating that the size of the file is smaller than a preset threshold value, and the data to be fragmented is used for indicating that the size of the file is larger than or equal to the preset threshold value;

converting data to be converted into an array with preset dimensionality to obtain a target array, and writing the target array into a preset binary file to obtain binary file data;

the method comprises the steps that data to be fragmented are fragmented and written into a preset fragmentation queue through a preset fragmentation mechanism to obtain target fragmentation data, wherein the fragmentation mechanism comprises the data size and the fragmentation mode of fragmentation;

Optionally, the storage module 305 includes:

the sorting unit 3051 is configured to obtain resource occupation ratios of storage nodes corresponding to a database based on distributed file storage, and sort the storage nodes in a descending order of the resource occupation ratios of the storage nodes to obtain a storage node sequence;

the cache creating unit 3052 is configured to cache data to be stored to obtain cache data, and create an index of the cache data;

the matching unit 3053 is configured to obtain a user identity identifier corresponding to the smart pen handwritten image sequence based on online education, and match a corresponding user storage policy according to the user identity identifier, where the user storage policy includes a storage node and a storage proportion;

the storage unit 3054 is configured to store the cache data for creating the index to the storage node sequence according to a preset user storage policy.

Optionally, the cache creating unit 3052 may be further specifically configured to:

analyzing data to be stored based on time locality and space locality to obtain initial data;

Optionally, the identification module 301 may be further specifically configured to:

carrying out character recognition and character segmentation on an intelligent pen handwritten image sequence based on online education through an optical character recognition network layer in a preset stroke order recognition neural network model to obtain character images, wherein one intelligent pen handwritten image corresponds to one or more character images, and the stroke order recognition neural network model comprises the optical character recognition network layer, a target detection network layer and a classification network layer;

through a target detection network layer, carrying out target frame detection, target frame extraction and multi-scale cross-channel-based convolution fusion on the character image to obtain a multi-scale feature image sequence set;

calculating the probability of character matching in the intelligent pen handwritten image sequence based on online education through a classification network layer based on a preset multi-language dictionary and a multi-scale feature map sequence set, wherein the multi-language dictionary comprises strokes and characters of multiple languages;

Optionally, the filtering module 302 may be further specifically configured to:

matching a target image set corresponding to target recognition information from a smart pen handwritten image sequence based on online education;

The function implementation of each module and each unit in the processing device based on the smart pen handwritten image corresponds to each step in the embodiment of the processing method based on the smart pen handwritten image, and the function and implementation process are not described in detail herein.

Fig. 3 and 4 above describe the processing apparatus based on the smart pen handwritten image in the embodiment of the present invention in detail from the perspective of the modular functional entity, and the processing system based on the smart pen handwritten image in the embodiment of the present invention is described in detail from the perspective of hardware processing.

Fig. 5 is a schematic structural diagram of a processing system based on a smart pen handwritten image, which includes a smart pen 5100, a server 5200 and a database 5300 based on distributed file storage according to an embodiment of the present invention; the server 5200 is connected with the intelligent pen and a database 5300 based on distributed file storage through a network; the smart pen comprises a sound-sensitive switch module 5101, a pressure sensor 5102, an optical character recognition scanner 5103, a processor 5104, an image converter 5105, a wireless network module 5106, a temperature sensor 5107, a tracking module 5108 and a wireless charging module 5109; the acoustic-sensitive switch module 5101 is used for detecting a knocking sound on the smart pen and starting the smart pen according to the knocking sound; the pressure sensor 5102 is used for detecting dot matrix pressure corresponding to the intelligent pen, acquiring writing parameters of the intelligent pen, and starting the optical character recognition scanner 5103 according to the dot matrix pressure; the optical character recognition scanner 5103 is configured to capture or scan content of the smart pen during manual writing, and obtain video information or image information; an image converter 5105 for generating a dot matrix image sequence based on online education for video information or image information; the processor 5104 is configured to mark the smart pen writing parameters in a dot matrix image sequence based on online education to obtain a smart pen writing image sequence based on online education; the wireless network module 5106 is configured to send the smart pen handwritten image sequence based on online education to the server 5200; the temperature sensor 5107 is used for detecting the temperature value of the smart pen; the tracking module 5108 is configured to locate a position where the smart pen is located, and calculate a located path; the wireless charging module 5109 is used for wirelessly charging the smart pen; the server 5200 includes a data transfer component 5210 and a processing module 5220; wherein, the data transmission component 5210 is configured to receive a sequence of smart pen handwritten images based on online education sent by the smart pen; a processing module 5220, configured to sequentially identify, filter, analyze, and convert the stroke order identification information into fragments to obtain data to be stored; a data transmission component 5210 for transmitting data to be stored to the database 5300 based on distributed file storage; the database 5300 based on distributed file storage is used for storing data to be stored, which is sent by the server 5200 and processed by the smart pen handwriting image sequence based on online education.

The smart pen 5100 further includes a power module 5110 and a signal transceiver 5111. The acoustic-sensitive switch module 5101 includes an acoustic-sensitive sensor and a switching circuit. The user taps any place on the body of the smart pen 5100 to make a tapping sound, the sound-sensitive sensor in the sound-sensitive switch module 5101 detects the sound, the smart pen 5100 is started through the switch circuit and the power supply module 5110, the nib of the smart pen 5100 is provided with the pressure sensor 5102, the user performs online questioning and answering of online education based on an online education platform, when manually writing on preset input equipment or an input screen or paper through the preset smart pen 5100, lattice pressure is generated, the lattice pressure is detected by the pressure sensor 5102, the lattice coordinates, handwriting paths, pressure parameters, writing speeds and other smart pen writing parameters corresponding to the nib of the smart pen 5100 are collected, the smart pen writing parameters are sent to the processor 4 through the signal transceiver 5111, when the user manually writes on the preset input equipment or the input screen or paper through the smart pen 5100, the method comprises the steps of starting an optical character recognition scanner 5103, shooting or scanning the content of the intelligent pen during manual writing by the optical character recognition scanner 5103 to obtain video information or image information, sending the video information or the image information to an image converter 5105 through a signal transceiver 5111, enabling the image converter 5105 to generate a dot matrix image sequence based on online education from the video information or the image information, sending the dot matrix image sequence based on the online education to a processor 5104, correspondingly marking intelligent pen writing parameters such as dot matrix coordinates, handwriting paths, pressure parameters and writing speed on the dot matrix image sequence based on the online education by the processor 5104 to obtain the intelligent pen handwriting image sequence based on the online education, and sending the intelligent pen handwriting image sequence based on the online education to a server 5200 through the signal transceiver 5111 and a wireless network module 5106.

The smart pen 5100 also includes a start-hold-stop circuit. The wireless network module 5106 includes a wireless Wifi transmission mode, a mobile communication signal 4G transmission mode, a Bluetooth Low Energy (BLE) transmission mode, and a mobile communication signal 5G transmission mode. When the smart pen 5100 is used, the temperature sensor 5107 can be used for detecting a temperature value of the smart pen 5100 and sending the temperature value to the processor 5104, the processor 5104 judges whether the temperature value is larger than a preset threshold value, if yes, the protection and stop circuit is started to control the smart pen 5100 to stop working, and if not, the detection is continued, and the temperature protection effect is achieved through the temperature sensor 5107. The tracking module 5108 includes a Global Positioning System (GPS) locator and a tracking calculation unit, and the smart pen 5100 can locate the position of the smart pen 5100 through the GPS locator in the tracking module 5108, and calculate the located path through the tracking calculation unit, so as to prevent the smart pen 5100 from being lost and facilitate the location recall of the smart pen 5100. The intelligent pen 5100 can be wirelessly charged through the wireless charging module 5109, and charging convenience of the intelligent pen is improved.

The smart pen 5100 also includes semiconductor optoelectronic devices 5112, such as a light pipe, a photocell, a photodiode, or a phototransistor, etc., semiconductor optoelectronic devices 5112. When the smart pen 5100 is started, and the semiconductor photoelectric device 5112 monitors that the illuminance of the environment where the smart pen 5100 is located is smaller than the preset brightness, illumination is performed, so that the smart pen 5100 can have enough brightness to operate the smart pen 5100 in the environment with low brightness.

The data transmission component 5210 may include a transceiver. The transceiver of the data transmission component 5210 may be a wired network transceiver, a wireless network transceiver, or a protocol transceiver.

The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium, and which may also be a volatile computer-readable storage medium, having stored therein instructions, which, when run on a computer, cause the computer to perform the steps of the processing method based on the smart pen handwritten image.

Further, the computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.

The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A processing method based on a handwritten image of a smart pen is characterized by comprising the following steps:

2. The smart pen-based handwritten image processing method according to claim 1, wherein said performing binary format conversion and/or slicing processing on the preprocessed data according to the result of the comparative analysis to obtain data to be stored comprises:

3. The smart pen-based handwritten image processing method according to claim 1, wherein said storing the data to be stored to a preset database based on distributed file storage includes:

4. The smart pen-based handwritten image processing method according to claim 3, wherein the caching the data to be stored to obtain cached data and creating an index of the cached data includes:

5. The smart pen handwritten image processing method according to claim 1, wherein said performing character recognition, character segmentation, multi-layer stroke order feature extraction and character information generation on the smart pen handwritten image sequence based on online education in sequence through a preset stroke order recognition neural network model to obtain stroke order recognition information includes:

6. The smart pen handwritten image-based processing method according to claim 1, wherein said filtering said order recognition information and said smart pen handwritten image sequence based on online education to obtain preprocessed data comprises:

7. The smart pen-based handwritten image processing method according to any of claims 1-6, characterized in that after storing the data to be stored in a preset database based on distributed file storage, the method includes:

8. A processing device based on a smart pen handwritten image, the processing device based on the smart pen handwritten image comprising:

9. A processing system based on a smart pen handwritten image comprises a smart pen, a server and a database based on distributed file storage;

10. A computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed by a processor, implement a method for processing a smart pen-based handwritten image according to any of claims 1-7.