CN115167750A - Handwritten note processing method, computer equipment and readable storage medium - Google Patents
Handwritten note processing method, computer equipment and readable storage medium Download PDFInfo
- Publication number
- CN115167750A CN115167750A CN202210883786.0A CN202210883786A CN115167750A CN 115167750 A CN115167750 A CN 115167750A CN 202210883786 A CN202210883786 A CN 202210883786A CN 115167750 A CN115167750 A CN 115167750A
- Authority
- CN
- China
- Prior art keywords
- track
- note
- graph
- preset type
- displaying
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 238000013136 deep learning model Methods 0.000 claims abstract description 35
- 238000000034 method Methods 0.000 claims abstract description 33
- 238000003909 pattern recognition Methods 0.000 claims abstract description 14
- 238000012549 training Methods 0.000 claims description 46
- 230000015654 memory Effects 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 10
- 230000004044 response Effects 0.000 claims description 7
- 238000012545 processing Methods 0.000 description 12
- 230000000306 recurrent effect Effects 0.000 description 10
- 238000013528 artificial neural network Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 238000004422 calculation algorithm Methods 0.000 description 7
- 230000002457 bidirectional effect Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000000605 extraction Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 238000003058 natural language processing Methods 0.000 description 4
- 230000007787 long-term memory Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000006403 short-term memory Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000002618 waking effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Medical Informatics (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Character Discrimination (AREA)
Abstract
The embodiment of the invention discloses a handwritten note processing method, computer equipment and a readable storage medium. In one embodiment, the method comprises: responding to a first operation of a user on the information presentation interface, and displaying a note input interface; acquiring relevant parameters of track points input by a user through handwriting on a note input interface, wherein the relevant parameters comprise coordinate parameters and time parameters; based on the relevant parameters, inputting the track formed by each continuous track point into a trained deep learning model for preset type pattern recognition: if the track belongs to the preset type of graph, the standard graph replaces the track to be displayed on the note input interface and stored in the note file; and if not, displaying the track on the note input interface and storing the track.
Description
Technical Field
The invention relates to the technical field of computers. And more particularly, to a handwritten note processing method, a computer device, and a readable storage medium.
Background
In scenes such as document browsing and electronic book reading performed by a user using an electronic device such as a tablet computer (Pad), there is a need to record notes in many cases. Taking electronic book reading as an example, currently, a common note recording mode is to record notes in an electronic book reading interface in an annotated mode, on one hand, there is a problem that summarized notes are not easy to record, and on the other hand, note contents usually only support characters and are relatively single. The user experience is poor.
Disclosure of Invention
An object of the present invention is to provide a handwritten note processing method, a computer device, and a readable storage medium, so as to solve at least one of the problems in the prior art.
In order to achieve the purpose, the invention adopts the following technical scheme:
the invention provides a handwritten note processing method, which is applied to terminal equipment with a touch screen and comprises the following steps:
responding to a first operation of a user on the information presentation interface, and displaying a note input interface;
acquiring relevant parameters of track points input by a user in a handwriting manner on a note input interface, wherein the relevant parameters comprise coordinate parameters and time parameters;
based on the relevant parameters of the track points, inputting the track formed by each continuous track point into a trained deep learning model for carrying out preset type pattern recognition:
if the recognition result is that the track belongs to a preset type of graph, displaying the track on the note input interface by replacing the track with a standard graph and storing the track in a note file;
and if the recognition result is that the track does not belong to the preset type of graph, displaying the track on the note input interface and storing the track into a note file.
Optionally, the inputting a track formed by each continuous track point into a trained deep learning model for preset type pattern recognition includes: inputting a track formed by each continuous track point into a trained deep learning model to identify the probability that the track belongs to each preset type of graph, and identifying the track as the preset type of graph when the probability that the track belongs to one preset type of graph is greater than a preset threshold value.
Optionally, the method further comprises:
obtaining a training track sample set, wherein the training track sample set comprises training track samples of each preset type of graph, and the training track samples of at least one preset type of graph comprise training track samples of multiple track point input sequences;
and training the deep learning model by using the training track sample set to obtain the trained deep learning model.
Optionally, the displaying and storing the standard graph in the note input interface to the note file instead of the track includes: and displaying a standard graph replacing the track based on the position and the size of the track and storing the standard graph into a note file.
Optionally, the method further comprises: and calling the note file and displaying the note file on a note input interface in response to the first operation of the user on the information presentation interface again or the first operation of the user on other information presentation interfaces belonging to the same sequence with the information presentation interface.
Optionally, the method further comprises: and inputting the track which does not belong to the preset type of graph into a character recognition model for character recognition, replacing one or more adjacent tracks recognized as characters with standard characters, displaying on the note input interface, and storing into a note file.
Optionally, the replacing, at the note input interface, one or more adjacent tracks identified as text with standard text and storing the one or more adjacent tracks in a note file includes: and displaying standard texts replacing the one or more adjacent tracks identified as the texts based on the positions and the sizes of the one or more adjacent tracks identified as the texts and storing the standard texts into a note file.
Optionally, the information presentation interface is an electronic book reading interface, and the preset type of graph includes a graph adopted by the thinking guide picture.
A second aspect of the present invention provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the handwritten note processing method provided by the first aspect of the present invention when executing the program.
A third aspect of the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the handwritten note processing method provided in the first aspect of the invention.
The invention has the following beneficial effects:
according to the technical scheme, the note content is stored by adopting the independent note file, the convenience of the user in the subsequent operations of checking notes, modifying notes, continuously writing notes, sharing notes and the like is improved, the beautification of the handwriting input graph can be realized on the basis of the preset handwriting rule on the basis of the note content supporting graph, the preset handwriting rule is embodied as limiting the preset type graph to a track formed by continuous track points, namely the preset type graph is a graph drawn by one stroke, the preset type graph is accurately identified through the trained deep learning model on the basis of the preset type graph, the error identification of characters or other graphs input by the handwriting of the user is avoided, and the user experience is improved.
Drawings
The following describes embodiments of the present invention in further detail with reference to the accompanying drawings.
FIG. 1 illustrates an exemplary system architecture diagram in which an embodiment of the present invention may be applied.
Fig. 2 is a flowchart illustrating a handwritten note processing method according to an embodiment of the present invention.
Fig. 3 shows a schematic diagram of a two-way long-short term memory network model.
Fig. 4 shows a schematic structural diagram of a computer device provided by an embodiment of the present invention.
Detailed Description
In order to more clearly illustrate the present invention, the present invention will be further described with reference to the following examples and the accompanying drawings. Similar parts in the figures are denoted by the same reference numerals. It is to be understood by persons skilled in the art that the following detailed description is illustrative and not restrictive, and is not to be taken as limiting the scope of the invention.
In scenes where a user uses an electronic device such as a tablet computer (Pad) or a conference all-in-one machine to perform document browsing, electronic book reading, conference content viewing, and the like, there is often a demand for recording notes. Taking a note-taking pen for recording when a tablet computer is used to read an electronic book as an example, currently, a common note-taking mode is to record notes in an electronic book reading interface in a labeling mode, and the notes are displayed on a side bar of the electronic book reading interface and stored in electronic book files. The user experience is poor.
In view of this, an embodiment of the present invention provides a method for processing handwritten notes, including the following steps:
responding to a first operation of a user on the information presentation interface, and displaying a note input interface;
acquiring relevant parameters of track points input by a user through handwriting on a note input interface, wherein the relevant parameters comprise coordinate parameters and time parameters;
based on the relevant parameters of the track points, inputting the track formed by each continuous track point into a trained deep learning model for carrying out preset type pattern recognition:
if the recognition result is that the track belongs to a preset type of graph, a standard graph is used for replacing the track to be displayed on the note input interface and stored in a note file;
and if the recognition result is that the track does not belong to the preset type of graph, displaying the track on the note input interface and storing the track into a note file.
According to the handwritten note processing method, a user can conveniently enter a note input interface to record notes when checking information, and an independent note file is adopted to store note contents, so that the convenience of the user in subsequent operations such as note checking, note modifying, note continuation, note sharing and the like is improved, and the beautifying of a handwritten input graph can be realized on the basis of a preset handwriting rule on the basis of a note content support graph, wherein the preset handwriting rule is embodied as limiting a preset graph to a track formed by continuous track points, namely the preset graph is a graph drawn by one stroke, the preset graph is accurately recognized through a trained deep learning model on the basis of the preset rule, the error recognition of characters or other graphs input by the user is avoided, and the user experience is improved.
The handwritten note processing method provided in this embodiment may be implemented by a terminal device with a touch screen, where the terminal device is, for example, a Computer device with data processing capability, and specifically, the Computer device may be a Computer with data processing capability, including a Personal Computer (PC), a mini-Computer, or a mainframe Computer, for example, an electronic device such as a tablet Computer and a conference integrated machine, which is commonly used, and this embodiment is not limited thereto.
In order to facilitate understanding of the technical solution of the present embodiment, a scene of the method provided by the present embodiment in practice is described below with reference to fig. 1. Referring to fig. 1, for example, the scenario includes a training server 101 and a tablet 102. In this embodiment, the training server 101 trains a deep learning model for identifying a trajectory belonging to a preset type of graph among the trajectories input by the user by using the training trajectory sample set to obtain a trained or trained deep learning model. Subsequently, after obtaining the trace points handwritten and input by the user in the note input interface, the tablet pc 102 may identify the trajectory formed by each continuous trace point by using the deep learning model trained by the training server 101, display and store the trajectory belonging to the preset type of graphics in the note input interface by replacing the trajectory with a standard graphics in the note input interface, and display and store the trajectory not belonging to the preset type of graphics, such as characters or other graphics, in the note input interface as it is in the note file.
It should be noted that, instead of the training server 101, the tablet computer 102 may implement the training process of the training server 101. When the training server 101 is set up, communication between the training server 101 and the tablet 102 may be via a network, which may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.
Next, a handwritten note processing method provided by the present embodiment is explained from the perspective of a processing apparatus having a data processing capability.
An embodiment of the present invention provides a handwritten note processing method, as shown in fig. 2, including steps S210-S240, where step S210 belongs to a training phase, and may be executed by a training server 101 or a tablet computer 102, for example, in this embodiment, by the training server 101; step S220 and the following steps belong to the processing stage and are executed by the tablet computer 102. The following is a detailed description.
As shown in fig. 2, the handwritten note processing method provided in this embodiment includes the following steps:
s210, a training track sample set is obtained, and the training track sample set is used for training the deep learning model to obtain the trained deep learning model.
In a possible implementation manner, the training track sample set includes training track samples of each preset type of graph, where the training track samples of at least one preset type of graph include training track samples of multiple track point input sequences.
The preset type graph is a graph drawn by one stroke, namely, the preset type graph is limited to a track formed by a section of continuous track points corresponding to the track points input by a user on the touch screen, so that the preset type graph can be accurately recognized by using a trained deep learning model in the subsequent steps, the error recognition of characters input by handwriting of the user or other graphs is avoided, and the situation that the user cannot input the characters completely when inputting the characters is avoided.
In one possible implementation manner, the preset type of graph includes a graph adopted by the thinking guide graph, that is, the preset type of graph is a graph which can be drawn by one stroke in the graph adopted by the thinking guide graph, such as a circle, a rectangle, a single arrow, a double arrow, a broken line arrow, a five-pointed star, and the like; the deep learning model has corresponding classes (templates) which are well trained; because the preset type graphs are completed in one stroke, the graphs completed in one stroke are adopted as training track samples when the deep learning model is trained, and the training track samples of each preset type graph can comprise training track samples of various track point input sequences, for example, for a rectangle, one training track sample can be completed through continuous horizontal-vertical-horizontal-vertical, and one training track sample can be completed through continuous vertical-horizontal-vertical-horizontal, so that the richness of the training track sample can be ensured, the generalization of the deep learning model is ensured, and the deep learning model after training can adapt to different writing habits of different users. In one example, a five-pointed star may complete a training trajectory sample at different starting points; the single arrow may complete the training trajectory sample by successive "cross-folds", "cross-greater than sign", or "greater than sign-cross". The double-arrow and the broken-line arrow are the same, and the thinking guide graph which can be completed by one stroke can be used as a preset type graph to obtain a training track sample of a plurality of track point input sequences.
In one possible implementation, the deep learning model is a Recurrent Neural Network (RNN) model. Further, the recurrent neural network model adopts a bidirectional long-short term memory network (Bi-LSTM) model.
The recurrent neural network is a recurrent neural network in which sequence data is input, recursion is performed in the evolution direction of the sequence, and all nodes (recurrent units) are connected in a chain manner, and has memory, parameter sharing, and graph completion (training completion), so that the recurrent neural network has certain advantages in learning the nonlinear characteristics of the sequence. Furthermore, the long-term and short-term memory network is a recurrent neural network, which is specially designed for solving the long-term dependence problem of the common recurrent neural network, and all recurrent neural networks have a chain form of repeated neural network modules. Due to the characteristics of the design of the long-term and short-term memory network, the long-term and short-term memory network is very suitable for modeling time sequence data, such as track point data which is composed of continuous track points and carries input sequence information.
In one specific example, as shown in FIG. 3, the LSTM is a function of the coordinates P of the trace points at time t t Cell State C t Temporary cell state
Hidden layer state h t Forgetting door f t Memory door i t Output gate o t Composition, the calculation process of LSTM can be summarized asThe forgetting and the memorizing of new information in the cell state enable information useful for the calculation at the subsequent moment to be transmitted, while useless information is discarded, and a hidden layer state h is output at each time step t Wherein the forgetting, memorizing and outputting are based on the hidden layer state h passing the last moment t-1 And the current input P t Calculated forgetting door f t Memory door i t Output gate o t To control.
In this example, the number of layers of the Long-Short term memory network LSTM network structure may be determined according to the difficulty of the task and the time-consuming situation, such as 1 layer or more, where the parameter bidirectionality of the LSTM should be set to true, that is, the Long-Short term memory network LSTM employs a bidirectional Long-Short term memory (Bi-LSTM) which is formed by combining a forward LSTM and a backward LSTM.
For example, as shown in FIG. 3, in the present example, the two-way long-short term memory network model is formed by combining a plurality of consecutive Bi-LSTMs and a Linear classifier (Linear). The Linear classifier (Linear) is positioned behind the Bi-LSTM and used for classifying the data processed by the Bi-LSTM, outputting a classification result of a specified dimensionality, and inputting a track consisting of continuous track points into a trained bidirectional long-short term memory network model to obtain the probability that the track belongs to each preset type of graph respectively to serve as a recognition result. It can be understood that the deep learning model may also use RNNs with more complex network structures, such as Gated Recirculation Units (GRUs) and the like, according to performance requirements, such as accuracy. In addition, for example, the loss function in the training phase is selected as the classification loss function.
And S220, responding to the first operation of the user on the information presentation interface, and displaying a note input interface.
In one possible implementation, the information presentation interface is an electronic book reading interface.
When a user reads an electronic document on the electronic device and wants to note, the user can call or wake up the note file by, for example, double-clicking the upper left corner of the electronic book reading interface, and in response to the operation, the tablet computer opens the note file, displays the note input interface, and the system calls a pattern recognition algorithm, wherein the pattern recognition algorithm is realized by a trained deep learning model. Therefore, when the user checks the information, the user can conveniently enter the note input interface to record the note, and the note content is stored by adopting the independent note file, so that the convenience of the user in subsequent operations of checking the note, modifying the note, writing the note again, sharing the note and the like is improved.
The implementation principle of the operation of calling or waking up the note file in the upper left corner of the double-click electronic book reading interface is as follows: the system judges whether to call out or create a note file, display a note input interface and start or call a graph recognition algorithm or an online handwriting recognition algorithm according to whether the captured touch information meets a preset rule. The basis of the system for judging double click is as follows: the time stamps of two consecutive touches are less than a specified threshold (e.g. 2 seconds) and the distance between x and y directions of the coordinates generated by each touch does not exceed the specified threshold (e.g. the distance of 1% pixel value of the resolution of the hardware device, the distance should be set lower than the width of the finger relative to the width of the pixel on the screen) to distinguish the double-click wake-up note file operation of the user from other operations. In addition, the note file can be awakened in the following way: drawing a preset symbol such as a letter W on the touch screen; and e.g. long-pressing, clicking n times, drawing a certain preset symbol to wake up and trigger a note file at the upper right corner, the lower left corner, the lower right corner, the middle of the screen and other positions of the touch screen.
S230, acquiring relevant parameters of track points input by a user in a handwriting mode on a note input interface, wherein the relevant parameters comprise coordinate parameters and time parameters;
s240, inputting the track formed by each continuous track point into a trained deep learning model for preset type pattern recognition based on the relevant parameters of the track points:
if the recognition result is that the track belongs to a preset type of graph, displaying the track on the note input interface by replacing the track with a standard graph and storing the track in a note file;
and if the recognition result is that the track does not belong to the preset type of graph, displaying the track on the note input interface and storing the track into a note file.
For example, in the case that the information presentation interface is an electronic book reading interface and the preset type of graph includes a graph adopted by the thought guidance graph, the user can write characters in the note file to record the read feeling and form the thought guidance graph by drawing geometric graphs to help to arrange and record the thought process and the like, so that complete note recording and thought backup are formed. The method comprises the steps of automatically recognizing a stroke of drawn graph as a preset type graph according to a preset handwriting rule, displaying the stroke of drawn graph on a note input interface by replacing the stroke of drawn graph with a standard graph, and storing the stroke of drawn graph into a note file to realize graph beautification.
Different from the existing manner in which the note is displayed on the sidebar of the e-book reading interface and stored in the e-book file, step S240 stores the track in the note file to implement unified note making in one note file, so that the user can conveniently look up and copy the note, and the method has a good effect on improving the reading effect. In addition, in step S240, based on the note that the note content supports graphics and realizes coexistence of text and graphics, beautification of the handwriting input graphics can be realized based on the preset handwriting rule, the preset handwriting rule is embodied as limiting the preset type graphics to a track formed by continuous track points, that is, the preset type graphics is a drawn graphics, and based on this, the preset type graphics is accurately recognized by the trained deep learning model, so that false recognition of text input by handwriting or other graphics of the user is avoided, incomplete input by the user when inputting text, for example, is avoided, and user experience is improved.
In a possible implementation manner, inputting a trajectory composed of each continuous trajectory point into a trained deep learning model for performing preset type pattern recognition includes: inputting a track formed by each continuous track point into a trained deep learning model to identify the probability that the track belongs to each preset type of graph, and identifying the track as the preset type of graph when the probability that the track belongs to one preset type of graph is greater than a preset threshold value.
It can be understood that the trained deep learning model does not have a high probability of recognizing a track to obtain more than two preset types of graphs, but may have a low probability that the probability that a track not belonging to the preset type of graphs is a certain preset type of graphs is not close to 0. In a specific example, a trajectory composed of continuous track points is input into a trained bidirectional long-short term memory network model, after probabilities that the trajectory belongs to each preset type of graph are obtained, whether a probability greater than a preset threshold value, for example, a value of 0.7, exists or not is judged, if yes, the trajectory is considered to belong to a corresponding preset type of graph, and if not, the trajectory is considered not to belong to any preset type of graph.
In one possible implementation, the displaying and storing the standard graph in the note input interface to the note file instead of the track includes: and displaying a standard graph replacing the track based on the position and the size of the track and storing the standard graph in a note file.
For example, if the trace input by the user is recognized as belonging to a rectangle, the standard rectangle in the display and handwriting file coincides with the center of the trace and is the same or similar in size. Therefore, the beautified graph is more in line with the expectation of the user, and the user experience is further improved.
In one possible implementation, the method further includes: and performing character recognition on a track input character recognition model which does not belong to the preset type of graph, replacing one or more adjacent tracks recognized as characters with standard characters, displaying the one or more adjacent tracks on the note input interface, and storing the one or more adjacent tracks into a note file, namely, if the recognition result is that the tracks do not belong to the preset type of graph, performing character recognition on the track input character recognition model, replacing one or more adjacent tracks recognized as characters with standard characters, displaying the one or more adjacent tracks on the note input interface, and storing the one or more adjacent tracks into the note file. The plurality of adjacent tracks are formed by writing characters obtained by recognition possibly in a plurality of strokes or are formed by a plurality of adjacent tracks.
It should be noted that, similar to the deep learning model for identifying whether the trajectory belongs to the preset type of pattern, the character recognition model may also adopt a recurrent neural network model, and further, may adopt a bidirectional long-short term memory network model.
The character recognition model may be another natural language processing model, such as a Transformer model, a Transformer xl model, a Bert model, or a GPT model, and the present embodiment is not limited thereto.
Wherein the dimensions of the natural language processing model are variable, including: the dimensionality of the natural language processing model, such as user settings, hardware resources, etc., can be adjusted according to preset conditions. The natural language processing model may include multiple feature extraction layers to ensure accuracy of feature extraction, e.g., a first feature extraction layer may exist in the form of a TransfomerXL model; the second feature extraction layer may exist in the form of a transform model or a Bert (Bidirectional Encoder reconstruction from transforms) model; the third feature extraction layer may exist in the form of a Transpherer model or a GPT (Generation Pre-Training) model.
In one possible implementation, the replacing one or more adjacent tracks identified as text with standard text in the note input interface and storing into a note file includes: and displaying standard texts replacing the one or more adjacent tracks identified as the texts based on the positions and the sizes of the one or more adjacent tracks identified as the texts and storing the standard texts into a note file.
It can be understood that controls such as an undo control, an eraser control and the like can be further arranged in the note input interface, and the corresponding input tracks are modified in response to the operations such as clicking, dragging and the like of the user on the controls, so that the modification and deletion of the input tracks by the user are realized.
In a possible implementation manner, after step S240, the method for processing written notes provided by this embodiment further includes: and responding to a second operation of the user on the note input interface, and returning to the information presentation interface.
For example, when the user wants to continue reading the electronic book after writing the note, the user closes the note file by, for example, double-clicking the upper left corner of the note input interface and returns to the electronic book reading interface, and in response to the operation, the tablet computer closes and saves the note file, displays the electronic book reading interface, and the system closes the pattern recognition algorithm, wherein the call for the pattern recognition algorithm is released while the note file is closed, so that system resources can be saved. It should be noted that the second operation and the first operation may adopt the same or similar operation modes, or may adopt different operation modes, for example, the note file is moved or awakened by double-clicking the upper left corner of the electronic book reading interface, and the note file is closed by long-pressing the lower right corner of the electronic book reading interface.
In a possible implementation manner, after the returning to the information presentation interface in response to the second operation of the user on the note input interface, the written note processing method provided by this embodiment further includes: and calling the note file and displaying the note file on a note input interface in response to the first operation of the user on the information presentation interface again or the first operation of the user on other information presentation interfaces belonging to the same sequence with the information presentation interface.
The other information presentation interfaces which belong to the same sequence with the information presentation interface are, for example, subsequent electronic book reading interfaces after page turning operation is performed on the previous electronic book reading interface, so that continuous note recording can be performed in the same note file when the same electronic book is read, and the convenience of note recording can be further improved.
Taking a tablet personal computer with a touch screen and used in cooperation with a writing pen as an example, another embodiment of the present invention provides an electronic device, including a tablet personal computer with a touch screen and a writing pen used in cooperation with the tablet personal computer, wherein the tablet personal computer can implement the handwritten note processing method provided by the above embodiment;
the touch screen of the tablet computer is used for responding to a first operation of a user on the information presentation interface and displaying a note input interface;
the tablet computer further comprises an acquisition module and an identification module;
the acquisition module is used for acquiring relevant parameters of track points input by a user in a handwriting manner on the note input interface, wherein the relevant parameters comprise coordinate parameters and time parameters;
and the recognition module is used for inputting the track formed by each continuous track point into a trained deep learning model for preset type pattern recognition based on the relevant parameters of the track points:
if the recognition result is that the track belongs to a preset type of graph, displaying the track on the note input interface by replacing the track with a standard graph and storing the track in a note file;
and if the recognition result is that the track does not belong to the preset type of graph, displaying the track on the note input interface and storing the track into a note file.
In a specific example, an operating system of the tablet computer is an Android (Android) system, an application layer and a bottom layer of the tablet computer cooperate to implement handwritten note processing, and the method specifically includes: when track points input by a user through handwriting on a note input interface are acquired, an application layer transmits coordinate parameters and time parameters of the track points to a bottom layer (an algorithm layer) through a data processing request to identify whether a track formed by each continuous track point belongs to a preset type of graph, and the bottom layer returns a result (track display data) to the application layer after processing and stores the track display data in editable note files such as a text file in a docx format and a picture file in a TIFF format.
It should be noted that the principle and the workflow of the tablet computer in the electronic device provided in this embodiment are similar to those of the handwritten note processing method, and reference may be made to the above description for relevant points, which are not described herein again.
As shown in fig. 4, a computer system suitable for implementing the tablet computer provided in the above embodiments includes a central processing module (CPU) that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) or a program loaded from a storage section into a Random Access Memory (RAM). In the RAM, various programs and data necessary for the operation of the computer system are also stored. The CPU, ROM, and RAM are connected thereto via a bus. An input/output (I/O) interface is also connected to the bus.
An input portion including a touch screen, a keyboard, a mouse, and the like; an output section including a touch panel or the like and a speaker or the like; a storage section including a hard disk and the like; and a communication section including a network interface card such as a LAN card, a modem, or the like. The communication section performs communication processing via a network such as the internet. The drive is also connected to the I/O interface as needed. A removable medium such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive as needed, so that the computer program read out therefrom is mounted into the storage section as needed.
In particular, the processes described in the above flowcharts may be implemented as computer software programs according to the present embodiment. For example, the present embodiments include a computer program product comprising a computer program tangibly embodied on a computer-readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section, and/or installed from a removable medium.
The flowchart and schematic diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to the present embodiments. In this regard, each block in the flowchart or schematic diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the schematic and/or flowchart illustration, and combinations of blocks in the schematic and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
On the other hand, the embodiment also provides a nonvolatile computer storage medium, which may be the nonvolatile computer storage medium included in the apparatus in the above embodiment, or may be a nonvolatile computer storage medium that exists separately and is not installed in a terminal. The non-volatile computer storage medium stores one or more programs that, when executed by a device, cause the device to: responding to a first operation of a user on the information presentation interface, and displaying a note input interface; acquiring relevant parameters of track points input by a user in a handwriting manner on a note input interface, wherein the relevant parameters comprise coordinate parameters and time parameters; based on the relevant parameters of the track points, inputting the track formed by each continuous track point into a trained deep learning model for carrying out preset type pattern recognition: if the recognition result is that the track belongs to a preset type of graph, displaying the track on the note input interface by replacing the track with a standard graph and storing the track in a note file; and if the recognition result is that the track does not belong to the preset type of graph, displaying the track on the note input interface and storing the track into a note file. .
In the description of the present invention, it should be noted that the terms "upper", "lower", and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, which are merely for convenience in describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed in a specific orientation, and operate, and thus, should not be construed as limiting the present invention. Unless expressly stated or limited otherwise, the terms "mounted," "connected," and "coupled" are to be construed broadly and encompass, for example, both fixed and removable coupling as well as integral coupling; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
It is further noted that, in the description of the present invention, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising one of 8230; \8230;" 8230; "does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.
It should be understood that the above-mentioned embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention, and it will be obvious to those skilled in the art that other variations and modifications can be made on the basis of the above description, and all embodiments cannot be exhaustive, and all obvious variations and modifications belonging to the technical scheme of the present invention are within the protection scope of the present invention.
Claims (10)
1. A handwritten note processing method is applied to terminal equipment with a touch screen, and is characterized by comprising the following steps:
responding to a first operation of a user on the information presentation interface, and displaying a note input interface;
acquiring relevant parameters of track points input by a user through handwriting on a note input interface, wherein the relevant parameters comprise coordinate parameters and time parameters;
based on the relevant parameters of the track points, inputting the track formed by each continuous track point into a trained deep learning model for carrying out preset type pattern recognition:
if the recognition result is that the track belongs to a preset type of graph, displaying the track on the note input interface by replacing the track with a standard graph and storing the track in a note file;
and if the recognition result is that the track does not belong to the preset type of graph, displaying the track on the note input interface and storing the track into a note file.
2. The method according to claim 1, wherein the inputting the track composed of each continuous track point into the trained deep learning model for pattern recognition of a preset type comprises: inputting a track formed by each continuous track point into a trained deep learning model to identify and obtain the probability that the track belongs to each preset type of graph, and identifying the track as the preset type of graph when the probability that the track belongs to one preset type of graph is greater than a preset threshold value.
3. The method of claim 1, further comprising:
obtaining a training track sample set, wherein the training track sample set comprises training track samples of each preset type of graph, and the training track samples of at least one preset type of graph comprise training track samples of multiple track point input sequences;
and training the deep learning model by using the training track sample set to obtain the trained deep learning model.
4. The method of claim 1, wherein the displaying and storing the track in the note entry interface in place of the standard graphic into a note file comprises: and displaying a standard graph replacing the track based on the position and the size of the track and storing the standard graph into a note file.
5. The method of claim 1, further comprising: and calling the note file and displaying the note file on a note input interface in response to the first operation of the user on the information presentation interface again or the first operation of the user on other information presentation interfaces belonging to the same sequence with the information presentation interface.
6. The method of claim 1, further comprising: and inputting the track which does not belong to the preset type of graph into a character recognition model for character recognition, replacing one or more adjacent tracks recognized as characters with standard characters, displaying on the note input interface, and storing into a note file.
7. The method of claim 6, wherein replacing the one or more adjacent tracks identified as text with standard text is displayed in the note entry interface and stored in a note file comprises: and displaying standard texts replacing the one or more adjacent tracks identified as the texts based on the positions and the sizes of the one or more adjacent tracks identified as the texts and storing the standard texts into a note file.
8. The method of claim 1, wherein the information presentation interface is an electronic book reading interface, and the preset type of graphics includes graphics used by the abri-dimensional guide chart.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1-8 when executing the program.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210883786.0A CN115167750A (en) | 2022-07-26 | 2022-07-26 | Handwritten note processing method, computer equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210883786.0A CN115167750A (en) | 2022-07-26 | 2022-07-26 | Handwritten note processing method, computer equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115167750A true CN115167750A (en) | 2022-10-11 |
Family
ID=83497763
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210883786.0A Pending CN115167750A (en) | 2022-07-26 | 2022-07-26 | Handwritten note processing method, computer equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115167750A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH07129722A (en) * | 1993-10-29 | 1995-05-19 | Matsushita Electric Ind Co Ltd | Online character recognition device using neural network |
| US20140354562A1 (en) * | 2013-05-30 | 2014-12-04 | Kabushiki Kaisha Toshiba | Shaping device |
| CN112711362A (en) * | 2020-12-24 | 2021-04-27 | 北京华宇信息技术有限公司 | Method and device for generating hand-drawn flow chart icon in standardized manner |
| CN113673432A (en) * | 2021-08-23 | 2021-11-19 | 京东方科技集团股份有限公司 | Handwriting recognition method, touch display device, computer device and storage medium |
| CN114579032A (en) * | 2022-02-15 | 2022-06-03 | 长沙朗源电子科技有限公司 | OCR-based intelligent hand-drawn graphic method, device and equipment for electronic whiteboard |
-
2022
- 2022-07-26 CN CN202210883786.0A patent/CN115167750A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH07129722A (en) * | 1993-10-29 | 1995-05-19 | Matsushita Electric Ind Co Ltd | Online character recognition device using neural network |
| US20140354562A1 (en) * | 2013-05-30 | 2014-12-04 | Kabushiki Kaisha Toshiba | Shaping device |
| CN112711362A (en) * | 2020-12-24 | 2021-04-27 | 北京华宇信息技术有限公司 | Method and device for generating hand-drawn flow chart icon in standardized manner |
| CN113673432A (en) * | 2021-08-23 | 2021-11-19 | 京东方科技集团股份有限公司 | Handwriting recognition method, touch display device, computer device and storage medium |
| CN114579032A (en) * | 2022-02-15 | 2022-06-03 | 长沙朗源电子科技有限公司 | OCR-based intelligent hand-drawn graphic method, device and equipment for electronic whiteboard |
Non-Patent Citations (1)
| Title |
|---|
| APPLE: "使用 iPad 上的"标记"在 App 内绘制", Retrieved from the Internet <URL:httpd://support.apple.com/zh-cn/guide/ipad/ipad6350b8dc/16.0/ipados/16.0> * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Aksan et al. | Deepwriting: Making digital ink editable via deep generative modeling | |
| US20200174658A1 (en) | Managing real-time handwriting recognition | |
| US11402991B2 (en) | System and method for note taking with gestures | |
| US7693842B2 (en) | In situ search for active note taking | |
| US9495620B2 (en) | Multi-script handwriting recognition using a universal recognizer | |
| JP2022116104A (en) | Managing real-time handwriting recognition | |
| KR102473543B1 (en) | Systems and methods for digital ink interaction | |
| CN109564656B (en) | System and method for collaborative ink management | |
| US7925987B2 (en) | Entry and editing of electronic ink | |
| US6952803B1 (en) | Method and system for transcribing and editing using a structured freeform editor | |
| US20170060819A1 (en) | System and method of digital note taking | |
| US20140363082A1 (en) | Integrating stroke-distribution information into spatial feature extraction for automatic handwriting recognition | |
| KR20060049081A (en) | Parsing hierarchicahl lists and outlines | |
| US7970812B2 (en) | Redistribution of space between text segments | |
| EP2075712A2 (en) | Persistent selection marks | |
| WO2019000681A1 (en) | Information layout method, device, apparatus and computer storage medium | |
| Zhelezniakov et al. | Online handwritten mathematical expression recognition and applications: A survey | |
| US7562314B2 (en) | Data processing apparatus and method | |
| CN112711362B (en) | Method and device for generating hand-drawn flow chart icon in standardized manner | |
| CN113673432A (en) | Handwriting recognition method, touch display device, computer device and storage medium | |
| CN113657347A (en) | Written character recognition method and device, terminal equipment and storage medium | |
| CN115167750A (en) | Handwritten note processing method, computer equipment and readable storage medium | |
| WO2024022068A9 (en) | Interaction method based on figure recognition, and display device | |
| US11631263B1 (en) | Gradient boosting tree-based spatial line grouping on digital ink strokes | |
| Aksan et al. | Generative Ink: Data-Driven Computational Models for Digital Ink |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |