WO2024021078A1 - 一种图文交互的方法及显示设备 - Google Patents

一种图文交互的方法及显示设备 Download PDF

Info

Publication number
WO2024021078A1
WO2024021078A1 PCT/CN2022/109143 CN2022109143W WO2024021078A1 WO 2024021078 A1 WO2024021078 A1 WO 2024021078A1 CN 2022109143 W CN2022109143 W CN 2022109143W WO 2024021078 A1 WO2024021078 A1 WO 2024021078A1
Authority
WO
WIPO (PCT)
Prior art keywords
sub
recognition result
text
information
graphic
Prior art date
Application number
PCT/CN2022/109143
Other languages
English (en)
French (fr)
Inventor
郑瑞
赵丽菲
马会广
乔丽娟
杜子龙
刘鑫
陈铁砺
Original Assignee
京东方科技集团股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 京东方科技集团股份有限公司 filed Critical 京东方科技集团股份有限公司
Priority to CN202280002471.9A priority Critical patent/CN117813580A/zh
Priority to PCT/CN2022/109143 priority patent/WO2024021078A1/zh
Priority to CN202310473009.3A priority patent/CN116578224A/zh
Priority to PCT/CN2023/105899 priority patent/WO2024022068A1/zh
Publication of WO2024021078A1 publication Critical patent/WO2024021078A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04845—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range for image manipulation, e.g. dragging, rotation, expansion or change of colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/32—Digital ink
    • G06V30/36—Matching; Classification
    • G06V30/387—Matching; Classification using human interaction, e.g. selection of the best displayed recognition candidate

Definitions

  • the present disclosure relates to the technical field of human-computer interaction, and in particular to a method of graphic and text interaction and a display device.
  • Smart conferences are office products that integrate multiple devices such as electronic whiteboards, computers, and remote conference terminals. They have multiple functions such as interactive smart whiteboards. With the rise of smart conferences, the importance of interactive conference experience in corporate conferences has gradually emerged. come out.
  • the present disclosure provides a graphic-text interaction method and a display device, which are used to perform associated operations on text written by a user through hand-drawn graphics, thereby improving the user's interactive experience.
  • inventions of the present disclosure provide a method for image-text interaction.
  • the method includes:
  • an association operation is performed on the displayed first text information.
  • an embodiment of the present disclosure provides a display device, including a display screen and a processor, wherein:
  • the display screen is used to interact with users and display content
  • the processor is configured to perform the following steps:
  • an association operation is performed on the displayed first text information.
  • embodiments of the present disclosure also provide a device for graphic and text interaction, which device includes:
  • a hand-drawn graphics unit configured to receive the first trajectory information of the user's hand-drawn graphics; wherein the first trajectory information is used to trigger the execution of interactive instructions related to the first text information;
  • a pattern recognition unit configured to perform pattern recognition on the first trajectory information and determine a pattern recognition result
  • An association operation unit configured to perform an association operation on the displayed first text information according to the interactive instruction corresponding to the graphic recognition result.
  • embodiments of the present disclosure also provide a computer storage medium on which a computer program is stored, and when the program is executed by a processor, it is used to implement the steps of the method described in the first aspect.
  • Figure 1 is an implementation flow chart of a graphic-text interaction method provided by an embodiment of the present disclosure
  • Figure 2 is a schematic structural diagram of a pattern recognition model provided by an embodiment of the present disclosure
  • Figure 3 is a schematic diagram of graphic recognition of input strokes provided by an embodiment of the present disclosure.
  • Figure 4 is a schematic diagram of a multi-stroke interaction process provided by an embodiment of the present disclosure.
  • Figures 5A-5B are a display interface diagram for automatically generating meeting minutes provided by an embodiment of the present disclosure
  • Figure 6 is a display interface diagram for inserting pictures provided by an embodiment of the present disclosure.
  • Figures 7A-7D are a display interface diagram for generating meeting minutes provided by an embodiment of the present disclosure.
  • Figure 8 is a text alignment display interface diagram provided by an embodiment of the present disclosure.
  • Figures 9A-9B are diagrams of a display interface for automatically converting text provided by an embodiment of the present disclosure.
  • 10A-10B are a display interface diagram for automatically generating to-do items provided by an embodiment of the present disclosure
  • 11A-11B are schematic scene diagrams of a smart calendar provided by an embodiment of the present disclosure.
  • Figures 12A-12B are schematic diagrams of adding to-do items provided by an embodiment of the present disclosure.
  • Figure 13 is a schematic diagram of a reminder of to-do items provided by an embodiment of the present disclosure.
  • Figure 14 is a schematic diagram of a display device provided by an embodiment of the present disclosure.
  • Figure 15 is a schematic diagram of an image-text interaction device provided by an embodiment of the present disclosure.
  • the term "and/or” describes the association relationship of associated objects, indicating that there can be three relationships, for example, A and/or B, which can mean: A exists alone, A and B exist simultaneously, and B exists alone. these three situations.
  • the character "/” generally indicates that the related objects are in an "or” relationship.
  • a smart conference is an office product that integrates multiple devices such as electronic whiteboards, computers, and remote conference terminals. It has multiple functions such as interactive smart whiteboards.
  • interactive conference experience is becoming more important in corporate conferences. Sexuality gradually emerges.
  • Current smart conferences are applied to large-size display devices.
  • Writing programs such as electronic whiteboards are installed on the display device. Users can write, draw graphics and other operations on the display device, and display the user's writing trajectory and hand-drawn graphics. trajectories, etc., but when the user interacts with the display device by writing or drawing, the interaction result is relatively simple.
  • the content written by the user or the graphics drawn by the user are directly displayed, which cannot provide more interactive experience and achieve more.
  • the interactive functions lead to poor user experience.
  • embodiments of the present disclosure provide a method of graphic-text interaction.
  • the core idea is to use hand-drawn graphics to trigger the execution of interactive instructions on text written by the user, thereby performing associated operations on the written text, and providing a method that uses hand-drawn graphics to trigger interactive instructions.
  • Graphics implement the function of text manipulation and are used in smart meeting scenarios, such as meeting writing scenarios, meeting minutes scenarios, to-do scenarios, smart calendar scenarios, etc., which can provide users with a rich interactive experience through hand-drawn graphics.
  • Step 100 Receive the first trajectory information of the user's hand-drawn graphics; wherein the first trajectory information is used to trigger the execution of interactive instructions related to the first text information;
  • the user-drawn graphics in this embodiment include one or more sub-graphics;
  • the first trajectory information includes but is not limited to the sub-trajectory information of one sub-graphic or the sub-trajectory information of multiple sub-graphics, where each sub-graphic can It is completed by one or more strokes, that is, the sub-trajectory information includes one stroke trajectory or multiple stroke trajectories.
  • the hand-drawn graphics in this embodiment include, but are not limited to, at least one of plane graphics of various shapes, various types of lines, special characters, punctuation marks, and arithmetic symbols.
  • the hand-drawn graphics include, but are not limited to, triangles and rectangles. , at least one of a circle, a diamond, a straight line, a wavy line, a curve, an arrow, a question mark, an exclamation mark, an asterisk, and a plus sign.
  • the first text information in this embodiment includes but is not limited to at least one of text written by the user, printed text, PDF text, WORD text, and text in picture format.
  • the first text information in this embodiment generally includes Regarding the medium of text content, this embodiment does not place too many restrictions on the specific form of the first text information.
  • the handwriting information of the text written by the user is received, and the first text information is determined based on the handwriting information.
  • the first text information in this embodiment may be handwriting information written by the user, or may be information obtained after format conversion of the handwriting information.
  • the display interface does not have the first text information
  • the first trajectory information is directly displayed, or the first trajectory information is graphically recognized, and the graphics are The recognition results are displayed.
  • the first text information in this embodiment can also be any document, such as a PDF document, a WORD document and other documents in various formats.
  • the first text information is determined based on received handwriting information written by the user.
  • the first text information is determined in any of the following ways:
  • Method 1 Receive the handwriting information of the text written by the user, perform trajectory recognition on the handwriting information to determine the trajectory recognition result; replace the handwriting information with the trajectory recognition result, and perform the trajectory recognition result in a preset format Display, and determine the displayed trajectory recognition result as the first text information;
  • Method 2 Receive and display the handwriting information of the text written by the user, and determine the displayed handwriting information as the first text information.
  • Step 101 Perform pattern recognition on the first trajectory information and determine the pattern recognition result
  • the graphics include multiple sub-graphics, and the first trajectory information includes sub-trajectory information of the multiple sub-graphics;
  • Receive sub-trajectory information of multiple sub-figures hand-drawn by the user perform pattern recognition on the sub-trajectory information of each sub-figure, and obtain first recognition results corresponding to the multiple sub-figures; determine the pattern recognition based on the multiple first recognition results result.
  • the order of pattern recognition is not limited.
  • the sub-trajectory information belonging to the same sub-figure can be filtered out from the sub-trajectory information of multiple sub-figures hand-drawn by the user according to the preset time threshold, and then the sub-trajectory information belonging to one sub-figure can be filtered out. sub-trajectory information for identification.
  • pattern recognition can also be performed sequentially in the following manner:
  • the sub-trajectory information of the multiple sub-figures is received in sequence; according to the order in which the multiple sub-trajectory information is received, each sub-trajectory information is sequentially recognized graphically, the corresponding first recognition result is obtained, and all the sub-trajectory information is saved. Describe the first recognition result.
  • the sub-trajectory information of multiple sub-graphics is received in sequence according to the order of multiple sub-graphics drawn by the user; each time the sub-trajectory information of a sub-graphic is received, a pattern recognition is performed on the sub-trajectory information to obtain the corresponding first recognition. results, and saves the first recognition result; and determines the final graphic recognition result based on the first recognition results corresponding to the saved multiple sub-graphics.
  • the first recognition result corresponding to this sub-figure is saved.
  • the next sub-figure of the current sub-figure is determined according to the interaction strategy.
  • the position range of the next sub-graphic of the current sub-graphic is determined according to the interaction strategy; wait within the position range Receive the sub-trajectory information of the next sub-figure. If the sub-trajectory information of the next sub-figure is received within the said position range, then continue to perform pattern recognition on the sub-trajectory information of the next sub-figure. If it is within the said position range, If the sub-track information of the next sub-figure is not received, it stops receiving the sub-track information of the next sub-figure and clears all the saved first recognition results.
  • this embodiment can also determine and execute the interactive instructions corresponding to the figure recognition results in sequence in the following manner:
  • the recognition order of the first recognition results respectively corresponding to the plurality of sub-figures is obtained, and when the recognition order satisfies the preset order, it is determined to execute the interactive instructions corresponding to the plurality of first recognition results.
  • the first recognition results corresponding to multiple sub-figures are obtained in the following manner:
  • the first stroke Recognize the first stroke to obtain a first recognition result, and when the first recognition result is an incomplete figure, save the first stroke; where the incomplete figure means that the original figure consists of multiple strokes, and the incomplete figure Indicates that some strokes in the original graphic are missing;
  • the second stroke is received within the coordinate range, and the first stroke and the second stroke are recognized together to obtain a first recognition result.
  • the sub-trajectory information of the current sub-figure is saved; according to the number of missing strokes and the coordinate range of the incomplete figure, in the Waiting to receive the sub-trajectory information corresponding to the missing stroke within the coordinate range; perform graphic recognition on the saved sub-trajectory information of the current sub-graphic and the sub-trajectory information corresponding to the missing stroke to obtain the incomplete
  • the first recognition result corresponding to the combination of the graphic and the missing stroke.
  • the interaction strategy in this embodiment includes at least one graphic or multiple graphics and the writing order of the multiple graphics; for example, when the interaction strategy is straight line + question mark, when the user draws a "straight line” by hand, the first recognition result is a straight line.
  • the first recognition result is "question mark-like”, and the class is saved.
  • the recognition result of the question mark is waiting to receive the next graphic, that is, a point in the question mark, in the lower position range of the user's hand-drawn "incomplete question mark”.
  • the interactive command will still not be triggered; when the user hand-draws "a point”, the user's hand-drawn "incomplete question mark” will be processed.
  • the combination of "question mark” and “dot” is used for graphic recognition, and the first recognition result is obtained and saved.
  • the graphic recognition result is determined based on each first recognition result.
  • the graphic recognition result satisfies the interaction strategy, the corresponding interaction instruction is executed.
  • the graphics include a first sub-graphic and a second sub-graphic;
  • the first trajectory information includes the first sub-trajectory information of the first sub-graphic and the second sub-trajectory information of the second sub-graphic:
  • Receive the first sub-trajectory information perform graphic recognition on the first sub-trajectory information, and obtain the first recognition result of the first sub-graphic.
  • the first recognition result of the first sub-graphic is the first sub-trajectory graphics, saving the first recognition result of the first sub-graphic;
  • Receive the second sub-trajectory information perform graphic recognition on the second sub-trajectory information, and obtain the first recognition result of the second sub-graphic.
  • the first recognition result of the second sub-graphic is the second sub-trajectory graphics
  • receiving the second sub-trajectory information, performing pattern recognition on the second sub-trajectory information, and obtaining a first recognition result of the second sub-pattern also includes:
  • Step 102 Perform an association operation on the displayed first text information according to the interactive instruction corresponding to the graphic recognition result.
  • the user draws hand-drawn graphics in order to perform related operations on the displayed text.
  • the corresponding interactive instructions are determined, thereby performing corresponding operations on the written text. associated operations.
  • the user after the user has finished writing the meeting content, he or she can mark the meeting content graphically to perform related operations on the marked meeting content.
  • the marked meeting content can also be identified by annotating the meeting content. content and automatically generate meeting minutes based on the recognition results.
  • the text written by the user can also be graphically annotated, the annotated text can be recognized, and to-do items can be automatically generated based on the recognition results.
  • the graphics in this embodiment include a first sub-graphic and a second sub-graphic; wherein the interactive instruction corresponding to the graphic recognition result of the first sub-graphic is used to determine the first text related to the interactive instruction The scope of the information; the interaction instruction corresponding to the graphic recognition result of the second sub-graphic is used to determine the interaction type, where different interaction types perform different association operations on the first text information.
  • the first association operation is performed on the first line of text information; if the user draws a "straight line” + “question mark” under the first line of the first text information; If a "straight line” + “exclamation mark” is hand-drawn below a line, the second association operation is performed on the text information of the first line.
  • this embodiment can also perform the following association operations on the displayed first text information:
  • alignment processing is performed on each row of text and/or each column of text in the displayed first text information.
  • an association operation is performed on the displayed first text information according to the interactive instructions corresponding to the recognition results in the following manner:
  • the first text information related to the information performs an association operation.
  • the first text information may be determined in the following manner:
  • Receive the handwriting information of the text written by the user perform trajectory recognition on the handwriting information to determine the trajectory recognition result; replace the handwriting information with the trajectory recognition result, and display the trajectory recognition result in a preset format, and The displayed trajectory recognition result is determined to be the first text information.
  • the first text information is handwriting information of text written by the user
  • Performing an association operation on the first text information related to the first trajectory information according to the interactive instruction corresponding to the recognition result includes:
  • the first text information related to the first trajectory information is replaced with a format corresponding to any one of a preset title, a preset subtitle, and a preset paragraph.
  • the trajectory recognition results are displayed in a preset format in the following manner:
  • the preset format corresponding to the trajectory recognition result of the handwriting information According to the corresponding relationship between the writing position of the handwriting information of the text and the preset row text position or preset column text position in the current writing area, determine the preset format corresponding to the trajectory recognition result of the handwriting information; The recognition results are displayed according to the corresponding preset format.
  • the trajectory recognition results of the user's handwriting information are displayed in any one or more of the following ways:
  • Method 1 When the writing position of the handwriting information of the text is the first row of text position or the first column of text position in the current writing area, the trajectory recognition result is displayed as a title corresponding to the preset title format;
  • the font of the displayed title is a preset format font, such as bold and bold.
  • the font size of the displayed title is determined based on the largest font size in the line of text written by the user. For example, the largest font size in the first line of text is 20 (px) ⁇ 20 (px), and the font size of the title is point 11.
  • Method 2 When the writing position of the handwriting information of the text is the second row of text position or the second column of text position in the current writing area, the trajectory recognition result is displayed as a subtitle corresponding to the preset subtitle format;
  • the font of the displayed subtitle is a preset format font, such as boldface.
  • the font size of the subtitle is determined based on the largest font size in the line of text written by the user.
  • Method 3 When the writing position of the handwriting information of the text is the row text position after the second line or the column text position after the second column in the current writing area, the trajectory recognition result is displayed as a preset paragraph format. corresponding paragraph content.
  • the font of the displayed paragraph content is a preset format font, such as Song Dynasty.
  • the size of the font in the paragraph content is determined based on the largest font size in the third line of text written by the user.
  • the handwriting information of the text will be automatically converted into titles, subtitles, and paragraph content, thereby automatically formatting the text written by the user, solving the problem that the user's handwritten content is messy and cannot be organized. problems, providing a more convenient way to improve the user experience.
  • the user when performing title recognition, the user opens handwriting software and automatically recognizes the first line of written text as the title of the current content, where the font of the title is a predefined first standard font.
  • the font of the title is a predefined first standard font.
  • subtitle recognition the second line of text written by the user is automatically recognized as a subtitle.
  • the font of the subtitle is the second predefined standard font.
  • the subtitle can also be automatically indented by 2 characters.
  • paragraph recognition the third line written by the user and the text after it are automatically recognized as paragraph content and automatically indented. If the paragraph content is not neat, the distance between adjacent lines will be automatically adjusted according to the preset line spacing to keep the line spacing consistent throughout the paragraph.
  • the trajectory recognition results are displayed in a preset format in the following manner:
  • the preset format corresponding to the trajectory recognition result of the handwriting information According to the corresponding relationship between the font size in the handwriting information of the text and the preset range, determine the preset format corresponding to the trajectory recognition result of the handwriting information; display the trajectory recognition result according to the corresponding preset format .
  • the trajectory recognition result is displayed in the corresponding preset format in any one or more of the following ways:
  • Method 4 When the font size in the handwriting information of the text meets the first preset range, display the trajectory recognition result as a title corresponding to the preset title format;
  • Method 5 When the font size in the handwriting information of the text meets the second preset range, display the trajectory recognition result as a subtitle corresponding to the preset subtitle format;
  • Method 6 When the font size in the handwriting information of the text meets the third preset range, the trajectory recognition result is displayed as the paragraph content corresponding to the preset paragraph format.
  • the first preset range is greater than the second preset range
  • the second preset range is greater than the third preset range
  • the above-mentioned methods 1 and 4 can be implemented in combination
  • the above-mentioned methods 2 and 5 can be implemented in combination
  • the above-mentioned methods 3 and 6 can be implemented in combination.
  • the solutions implemented in combination will not be described again here.
  • the preset format of the trajectory recognition result of the current line of handwriting information can be determined according to the position of the text written by the user; on the other hand, the displayed handwriting information of the current line can be determined based on the size of the font of the current line written by the user.
  • the preset format of the trajectory recognition result; finally, the position and font size of the text written by the user can also be combined to determine the preset format of the trajectory recognition result of the current line of handwriting information;
  • the preset format in this embodiment includes but It is not limited to font type, font size, whether it is bold, whether it is underlined, and other types of formats.
  • this embodiment may also determine the first text information in the following manner:
  • Handwriting information of the text written by the user is received and displayed, and the displayed handwriting information is determined as the first text information.
  • the first text information in this embodiment may be the handwriting information of the text written by the user, or may be the trajectory information obtained by converting the handwriting information, which is not too limited in this embodiment.
  • this embodiment performs pattern recognition on the first trajectory information in the following manner to determine the pattern recognition result:
  • one of the sub-shapes in the user's hand-drawn graphics includes a stroke.
  • pattern recognition is performed on the stroke trajectory to determine the pattern recognition result.
  • the user hand-draws a circle or rectangle containing a stroke, and performs pattern recognition on the stroke to determine that the pattern recognition result is a circle or rectangle.
  • one of the sub-shapes in the user's hand-drawn graphics includes multiple strokes.
  • pattern recognition is performed on the combination of multiple strokes with continuous stroke order, and the pattern recognition result is determined.
  • Multiple strokes are multiple graphics, at least one graphic contains one stroke, and at least one graphic contains multiple strokes.
  • the multiple strokes can be sequentially identified according to the stroke order of the sub-figures hand-drawn by the user.
  • the pattern recognition result of the strokes is a complete figure
  • the next stroke can be continued to be identified.
  • Recognition when the graphic recognition result of the stroke is a incomplete graphic, the stroke is retained first, and the stroke and at least one consecutive stroke after the stroke are recognized as a whole until it is determined that the recognition result is a complete graphic. stop.
  • the pattern recognition result is determined by sequentially performing pattern recognition on the combination of multiple consecutive strokes in the stroke sequence in the following manner:
  • the user hand-draws a question mark "?" as an example.
  • the question mark includes two strokes.
  • the graphic recognition result of the stroke is determined to be a incomplete graphic, it is determined that the missing character is missing.
  • the number and coordinates of the strokes that is, it is determined that a stroke is missing and the coordinates should be located below the first stroke, then according to the number and coordinates, determine a stroke below the first stroke, the stroke and the first stroke
  • To perform pattern recognition in combination that is, to perform pattern recognition on the stroke and the first stroke as a whole, and determine that the pattern recognition result is a question mark.
  • pattern recognition in this embodiment includes but is not limited to OCR (Optical Character Recognition), which refers to an electronic device (such as a scanner or digital camera) checking the characters printed on the paper and detecting dark characters. , the process of determining its shape using a bright pattern, and then using character recognition methods to translate the shape into computer text. Among them, OCR is also used for text recognition, trajectory recognition, etc.
  • OCR Optical Character Recognition
  • OCR recognition can be divided into three stages according to the processing method: pre-processing, text recognition or graphic recognition, and post-processing; among them, pre-processing mainly includes grayscale, binarization, noise removal, tilt correction, etc.; graphic recognition mainly includes CNN +RNN+CTC, or CNN+RNN+Attention (attention model).
  • pre-processing mainly includes grayscale, binarization, noise removal, tilt correction, etc.
  • graphic recognition mainly includes CNN +RNN+CTC, or CNN+RNN+Attention (attention model).
  • the pattern recognition algorithm or trajectory recognition algorithm in this embodiment includes but is not limited to: at least one of EAST, CTPN, SegLink, PixelLink, TextBoxes, TextBoxes++, TextSnake, and MSR.
  • OCR will be called to save the current picture, so that the relevant strokes can be intercepted for subsequent pattern recognition.
  • this embodiment can perform pattern recognition through a LSTM+linear (linear transformation layer) network.
  • the number of layers of the LSTM network structure can be determined according to the difficulty and time-consuming nature of the task. For example, one or more layers are used.
  • the parameter bidirectional of LSTM is set to true, which means using bidirectional LSTM and using the input before and after information; linear is a linear transformation layer, used to output the specified dimension.
  • this embodiment provides a pattern recognition model, including an LSTM network layer 200 and a linear transformation layer 201.
  • An RNN network with additional structures can also be used as needed, and a classification loss function can be used for RNN training.
  • LSTM is used to extract features from the input strokes to obtain stroke features
  • linear is used to classify the input stroke features and determine the graphic type and coordinates of the stroke.
  • this embodiment also provides a schematic diagram for graphic recognition of input strokes, in which each stroke received is determined according to each stroke-up stroke of the user, and the received strokes are determined according to the order of writing.
  • the handwriting point coordinates in each stroke are input to the LSTM + linear transformation layer linear for identification, and the graphic type corresponding to the stroke is obtained.
  • the handwriting points of each stroke can be sparsely processed and then input to LSTM+linear for recognition to improve recognition efficiency.
  • this embodiment provides a multi-stroke interaction process.
  • the specific steps for implementing this process are as follows:
  • Step 400 Receive the first trajectory information of a user-drawn graphic, wherein the graphic includes a straight line and a question mark, the straight line includes one stroke, and the question mark includes two strokes;
  • Step 401 Perform graphic recognition on a stroke related to a straight line in the first trajectory information according to the order of strokes drawn by the user, and obtain that the graphic type of the stroke is a straight line;
  • Step 402 Continue to perform graphic recognition on the first stroke related to the question mark in the first trajectory information, and obtain that the type of the stroke is an incomplete question mark;
  • Step 403 Perform graphic recognition on the combination of the first stroke and the second stroke related to the question mark in the first trajectory information, and the obtained graphic type is a question mark;
  • Step 404 According to the interactive instruction corresponding to the straight line + question mark, perform an association operation on the first text information related to the first trajectory information.
  • this embodiment performs an association operation on the displayed first text information according to the interactive instructions corresponding to the recognition results in the following manner:
  • the first text information related to the information performs an association operation.
  • the first text information related to the first trajectory information is determined, it is first filtered out from the corresponding first text information, and then the associated operation is performed according to the interactive instructions triggered by the user. .
  • the first text information filtered out is information in text format.
  • this embodiment can perform an association operation on part or all of the first text information that has been displayed through the first trajectory information.
  • the first text information related to the first trajectory information is the already displayed At least part of the first text information performs an association operation on the first text information related to the first trajectory information according to the interactive instruction corresponding to the recognition result of the first trajectory information.
  • the relationship between the display positions of the first trajectory information and the first text information in this embodiment includes but is not limited to, the first trajectory information completely surrounds the first text information, or the first trajectory information is located in the first text. At least one of: below the information, or the first track information is located on one side of the first text information.
  • the relationship between the display positions of the first trajectory information and the first text information in this embodiment is only an example, and the purpose is to use the first trajectory information to perform annotation, association operations, etc. on the first text information. Based on Other positional relationships of the inventive concept also fall within the protection scope of the present disclosure.
  • this embodiment performs the following steps to perform an association operation on the first text information related to the first trajectory information according to the interactive instructions corresponding to the recognition results:
  • text recognition is performed on the first text information related to the first trajectory information, and a text recognition result is obtained and displayed.
  • semantic recognition is performed on the first text information related to the first trajectory information according to the interactive instruction of the recognition result, and meeting minutes or to-do items are generated according to the semantic recognition result.
  • the meeting minutes in this embodiment represent the meeting summary related to the meeting content handwritten by the user in the meeting scenario, including but not limited to at least one of the meeting title, meeting content, meeting time, and meeting location.
  • the to-do items in this embodiment represent content related to the user's handwritten to-do items in the to-do item scenario, including but not limited to at least one of schedule topic, schedule content, schedule time, and schedule location.
  • the first trajectory information of the hand-drawn graphics triggers text recognition and semantic recognition of the text written by the user, and the title, content, time, location, etc. in the text are identified. Information, thereby automatically generating meeting minutes or to-do items based on the identified information according to different scenarios such as meeting scenarios or to-do scenarios, improving the user experience.
  • meeting minutes are generated based on at least one of meeting title, meeting content, meeting time, and meeting location in the semantic recognition result.
  • At least one of the meeting title, meeting content, meeting time, and meeting location is added to the corresponding position of the meeting minutes template, that is, the meeting title is added to the title position of the meeting minutes template, Add the time, location, and content to the corresponding locations in the meeting minutes template to automatically generate meeting minutes.
  • this embodiment provides a display interface for automatically generating meeting minutes.
  • the user handwrites the meeting title, meeting content, meeting time and meeting location in the writing software to complete the meeting related text.
  • the user specifies all or part of the conference-related text through hand-drawn graphics. For example, the user surrounds all the written text with a hand-drawn rectangle.
  • the execution of interactive instructions with the entire text is triggered, and semantic recognition of the entire text is performed.
  • the minutes template meeting minutes are generated.
  • this embodiment can also insert pictures, web pages, files, etc. when writing text, and perform the following steps:
  • the inserted content is displayed in a hypertext markup language format.
  • the third trajectory information hand-drawn by the user can also be received, and the inserted pictures can be arranged according to the text position pointed by the third trajectory information. On the adjacent side of the text position.
  • this embodiment provides a display interface for inserting pictures.
  • the user When the user inserts a picture into the writing software, it can also receive the user's hand-drawn arrow, use the arrow to connect the picture and text, and according to the The text position pointed by the arrow will arrange the inserted picture below the text position.
  • the interactive instructions corresponding to the recognition results of the first trajectory information hand-drawn by the user are displayed through HTML5, so that the user can view the inserted pictures, web pages, files, etc. again.
  • meeting minutes are generated based on the semantic recognition results, and all the meeting minutes can also be generated. At least one of the pictures, web pages, and files is added to the meeting minutes in hypertext markup language format. Users can view the pictures, web pages, files and other content inserted by users in the meeting minutes again.
  • this embodiment provides a display interface for generating meeting minutes.
  • the display interface displays the meeting title, the icon of the inserted picture, the URL link of the inserted web page, the inserted The name of the file and the meeting time.
  • the user clicks the icon of the inserted picture as shown in Figure 7B
  • the inserted picture is displayed, and the display position of the content behind the icon is automatically adjusted, as shown in Figure 7C.
  • the corresponding browser interface is opened to display the web page.
  • the file reading program is opened and the content of the file is displayed.
  • the meeting minutes include the meeting time; this embodiment can also respond to the user's click operation on the meeting time, start the calendar program and add the meeting minutes as a to-do item to the calendar program ; Or receive the first trajectory information related to the meeting time, perform graphic recognition on the first trajectory information, and determine the graphic recognition result; start the calendar program according to the interactive instructions corresponding to the graphic recognition result, and convert the Meeting minutes are added to the calendar program as a to-do item.
  • each row of text and/or each column of text in the displayed first text information is aligned according to the interactive instruction corresponding to the graphic recognition result.
  • this embodiment provides a text alignment display interface. When the user draws a straight line on the left side of the written text, the printed text obtained after identifying the text on the right side of the straight line is left aligned.
  • this embodiment also provides a display interface for automatically converting text.
  • the text written by the user is converted , as shown in Figure 9B, the picture is inserted below the text connected by the arrow drawn by the user, and the web page and file are displayed in HTML5 format to facilitate the user's viewing.
  • a to-do item is generated based on at least one of schedule topic, schedule content, schedule time, and schedule location in the semantic recognition result.
  • the schedule topic, schedule content, schedule time, and schedule location are added to the corresponding positions of the to-do template respectively, thereby automatically generating to-do items.
  • this embodiment provides a display interface for automatically generating to-do items.
  • the user hand-writes the schedule topic, schedule content, schedule time, and schedule location of the to-do item, and then the user The hand-drawn rectangle places all the text of the handwritten to-do item inside the rectangle, triggering the execution of semantic recognition of all the text of the to-do item.
  • the schedule subject, schedule content, schedule time, and schedule obtained by semantic recognition are The schedule location is added to the corresponding location in the preset to-do template to automatically generate to-do items.
  • this embodiment generates a to-do item based on the semantic recognition result, and can also determine the application program associated with the to-do item based on the application program associated with at least one keyword in the semantic recognition result; At the schedule time of the to-do item, an application associated with the to-do item is launched.
  • the to-do item is associated with the video conferencing application; when the semantic recognition results include the keyword “Transfer”, “Copy”, “File”, then associate the to-do item to the file quick transfer application; when the semantic recognition results include the keywords "writing”, “handwriting”, “record”, “minutes", “table” “, the to-do item is associated with the workbench application; when the keywords “picture”, “drawing”, “drawing”, and “sketch” exist in the semantic recognition results, the to-do item is associated with the drawing pad; when If there are keywords “retrieval", “search”, “query”, and “webpage” in the semantic recognition results, the to-do item will be associated with the browser. When the schedule time of the to-do item is reached, the application associated with the to-do item is launched.
  • this embodiment also provides a smart calendar scenario.
  • the calendar program is displayed in the form of a window on the home page of the desktop launcher, and in the calendar program Show schedules for different dates.
  • the floating window of the calendar program displays the dates of the current month by default, and dates with agendas (to-do items) are marked with reminders. Users can view the added schedules and display the added schedules in the order of their start time. When the start times of different schedules are consistent or conflict, the conflicting schedules will be displayed in the order of the time when the schedules were added. , when the content of the schedule exceeds the display range of the display area, it can be displayed through pull-down and slide-down operations.
  • this embodiment performs semantic recognition on the text recognition results through the following steps, and generates to-do items based on the semantic recognition results:
  • Add the text recognition result to the first editing area of the schedule editing interface perform semantic recognition on the text recognition result in the editing area, and add the semantic recognition result to at least one second editing area; according to at least one second editing area Semantic recognition results in the area generate to-do items.
  • this embodiment provides a schematic diagram for adding to-do items.
  • the text recognition result is displayed in the first editing area of the schedule editing interface.
  • the user can continue to edit the text recognition result, or can directly perform semantic recognition without editing, as shown in Figure 12B , when the semantic recognition results include schedule subject, schedule content, schedule time, and schedule location, the schedule subject, schedule content, schedule time, and schedule location are respectively added to the corresponding second editing area for display.
  • the user can also continue to edit the second You can edit the content displayed in the editing area, or you can directly generate to-do items based on the content in each second editing area.
  • the to-do items generated based on the semantic recognition results can also be added to a calendar program for display; wherein, the calendar program Display as a window on the homepage of the desktop launcher.
  • the user writes text in the writing software, and the user specifies the first text information of the relevant text by drawing the first trajectory information of the graphic, and performs text recognition on the first text information related to the first trajectory information to obtain the text.
  • Recognition results perform semantic recognition on the text recognition results, and generate to-do items based on the semantic recognition results.
  • the text recognition results in this embodiment include text in standard fonts and/or standard formats, and the semantic recognition results include but are not limited to: at least one of topic, content, time, location, and preset keywords.
  • the user can also set a reminder time. For example, the user can select 10 minutes, 15 minutes, 20 minutes, or 30 minutes before the to-do item's schedule time to be reminded.
  • this embodiment provides a schematic diagram for reminding the to-do item.
  • the to-do item has an associated application
  • the user can be prompted with a global pop-up window before the start of the to-do item's schedule time.
  • the application is about to start, and other operations except in the pop-up window will be invalid. For example, you can use a countdown reminder. If the countdown is 0, the pop-up window will automatically close and the relevant application will be opened.
  • the embodiment of the present disclosure also provides a display device. Since this device is the device in the method in the embodiment of the present disclosure, and the principle of solving the problem of the device is similar to that of the method, therefore The implementation of the device can be found in the implementation of the method, and repeated details will not be repeated.
  • the display device includes a display screen 1400 and a processor 1401, where:
  • the display screen 1400 is used to interact with the user and display content
  • the processor 1401 is configured to perform the following steps:
  • an association operation is performed on the displayed first text information.
  • the first text information is determined based on the received handwriting information written by the user.
  • the processor is specifically configured to determine the first text information in the following manner:
  • Receive the handwriting information of the text written by the user perform trajectory recognition on the handwriting information to determine the trajectory recognition result; replace the handwriting information with the trajectory recognition result, and display the trajectory recognition result in a preset format, and The displayed trajectory recognition result is determined to be the first text information; or,
  • Handwriting information of the text written by the user is received and displayed, and the displayed handwriting information is determined as the first text information.
  • the graphics include multiple sub-graphics, and the first trajectory information includes sub-trajectory information of multiple sub-graphics; the processor is specifically configured to execute:
  • Receive sub-trajectory information of multiple sub-figures hand-drawn by the user perform graphic recognition on the sub-trajectory information of the sub-figures, and obtain first recognition results corresponding to the multiple sub-figures respectively;
  • the pattern recognition result is determined according to a plurality of first recognition results.
  • the processor is specifically configured to execute:
  • graphic recognition is performed on each sub-trajectory information in turn, a corresponding first recognition result is obtained, and the first recognition result is saved.
  • the processor is specifically configured to execute:
  • the processor is specifically configured to execute:
  • the processor is specifically configured to determine and execute the interactive instructions corresponding to the figure recognition results in the following manner:
  • the recognition order of the first recognition results respectively corresponding to the plurality of sub-figures is obtained, and when the recognition order satisfies the preset order, it is determined to execute the interactive instructions corresponding to the plurality of first recognition results.
  • the sub-graphic includes at least a first stroke and a second stroke; the processor is specifically configured to execute:
  • the second stroke is received within the coordinate range, and the first stroke and the second stroke are recognized together to obtain a first recognition result.
  • the graphics include a first sub-graphic and a second sub-graphic;
  • the first trajectory information includes the first sub-trajectory information of the first sub-graphic and the second sub-trajectory of the second sub-graphic Information;
  • the processor is specifically configured to execute the interactive instructions in the following manner:
  • Receive the first sub-trajectory information perform graphic recognition on the first sub-trajectory information, and obtain the first recognition result of the first sub-graphic.
  • the first recognition result of the first sub-graphic is the first sub-trajectory graphics, saving the first recognition result of the first sub-graphic;
  • Receive the second sub-trajectory information perform graphic recognition on the second sub-trajectory information, and obtain the first recognition result of the second sub-graphic.
  • the first recognition result of the second sub-graphic is the second sub-trajectory graphics
  • the graphics include a first sub-graphic and a second sub-graphic
  • the interactive instruction corresponding to the graphic recognition result of the first sub-graph is used to determine the range of the first text information related to the interactive instruction;
  • the interaction instruction corresponding to the graphic recognition result of the second sub-graphic is used to determine the interaction type, where different interaction types perform different association operations on the first text information.
  • the processor is specifically configured to execute:
  • alignment processing is performed on each row of text and/or each column of text in the displayed first text information.
  • the processor is specifically configured to execute:
  • an association operation is performed on the first text information related to the first trajectory information.
  • the first text information is handwriting information of text written by the user;
  • the processor is specifically configured to execute:
  • the first text information related to the first trajectory information is replaced with a format corresponding to any one of a preset title, a preset subtitle, and a preset paragraph.
  • the processor is specifically configured to execute:
  • text recognition is performed on the first text information related to the first trajectory information, and a text recognition result is obtained and displayed.
  • the processor is specifically configured to execute:
  • semantic recognition is performed on the first text information related to the first trajectory information, and meeting minutes or to-do items are generated according to the semantic recognition result.
  • the processor is specifically configured to execute:
  • Meeting minutes are generated based on at least one of the meeting title, meeting content, meeting time, and meeting location in the semantic recognition result.
  • the first text information includes meeting minutes; the meeting minutes include meeting time; the processor is specifically configured to:
  • a calendar program is started, and the meeting minutes are added to the calendar program as a to-do item.
  • the processor is specifically configured to execute:
  • an application associated with the to-do item is launched.
  • the processor is specifically configured to execute:
  • a to-do item is generated based on the semantic recognition result in at least one second editing area.
  • the processor is further configured to execute:
  • the calendar program is displayed on the homepage of the desktop launcher in the form of a window.
  • the embodiment of the present disclosure also provides a device for picture-text interaction, because this device is the device in the method in the embodiment of the present disclosure, and the principle of solving the problem of the device is the same as that of the method. are similar, so the implementation of the device can be referred to the implementation of the method, and repeated details will not be repeated.
  • the device includes:
  • the hand-drawn graphics unit 1500 is used to receive the first trajectory information of the user's hand-drawn graphics; wherein the first trajectory information is used to trigger the execution of interactive instructions related to the first text information;
  • the pattern recognition unit 1501 is used to perform pattern recognition on the first trajectory information and determine the pattern recognition result
  • the association operation unit 1502 is configured to perform an association operation on the displayed first text information according to the interactive instruction corresponding to the graphic recognition result.
  • embodiments of the present disclosure also provide a computer storage medium on which a computer program is stored.
  • the program is used to implement the following steps when executed by a processor:
  • an association operation is performed on the displayed first text information.
  • embodiments of the present disclosure may be provided as methods, systems, or computer program products. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, optical storage, and the like) embodying computer-usable program code therein.
  • a computer-usable storage media including, but not limited to, magnetic disk storage, optical storage, and the like
  • These computer program instructions may also be stored in a computer-readable memory that causes a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instructed device, the instructions
  • the equipment implements the functions specified in a process or processes in the flow diagram and/or in a block or blocks in the block diagram.
  • These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device.
  • Instructions provide steps for implementing the functions specified in a process or processes of a flowchart diagram and/or a block or blocks of a block diagram.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • User Interface Of Digital Computer (AREA)
  • Character Discrimination (AREA)

Abstract

本公开提出了一种图文交互的方法及显示设备，用于通过手绘图形对用户书写的文本进行关联操作，提高用户的交互体验。该方法包括：接收用户手绘的图形的第一轨迹信息；其中所述第一轨迹信息用于触发执行与第一文本信息相关的交互指令；对所述第一轨迹信息进行图形识别，确定图形识别结果；根据所述图形识别结果对应的交互指令，对显示的第一文本信息执行关联操作。

Description

一种图文交互的方法及显示设备 技术领域
本公开涉及人机交互技术领域，特别涉及一种图文交互的方法及显示设备。
背景技术
智能会议是集成了电子白板、电脑和远程会议终端等多种设备的办公产品，具备交互式智能白板等多种功能，随着智能会议的兴起，会议交互体验在企业会议中的重要性逐渐显现出来。
目前的智能会议应用于大尺寸的显示设备，在该显示设备上安装有电子白板等书写程序，用户可以在该显示设备上进行书写、绘制图形等操作，并显示用户书写的轨迹、手绘的图形轨迹等，但是用户通过书写或绘制的方式与显示设备进行交互时，交互结果较为单一，通常只是将用户书写的内容或绘制的图形直接进行显示，无法提供更多的交互体验，无法实现更多的交互功能，导致用户的体验感较差。
发明内容
本公开提供一种图文交互的方法及显示设备，用于通过手绘图形对用户书写的文本进行关联操作，提高用户的交互体验。
第一方面，本公开实施例提供的一种图文交互的方法，该方法包括：
接收用户手绘的图形的第一轨迹信息；其中所述第一轨迹信息用于触发执行与第一文本信息相关的交互指令；
对所述第一轨迹信息进行图形识别，确定图形识别结果；
根据所述图形识别结果对应的交互指令，对显示的第一文本信息执行关联操作。
第二方面，本公开实施例提供的一种显示设备，包括显示屏和处理器， 其中：
所述显示屏用于与用户进行交互，并进行内容的显示；
所述处理器被配置为执行如下步骤：
接收用户手绘的图形的第一轨迹信息；其中所述第一轨迹信息用于触发执行与第一文本信息相关的交互指令；
对所述第一轨迹信息进行图形识别，确定图形识别结果；
根据所述图形识别结果对应的交互指令，对显示的第一文本信息执行关联操作。
第三方面，本公开实施例还提供一种图文交互的装置，该装置包括：
手绘图形单元，用于接收用户手绘的图形的第一轨迹信息；其中所述第一轨迹信息用于触发执行与第一文本信息相关的交互指令；
图形识别单元，用于对所述第一轨迹信息进行图形识别，确定图形识别结果；
关联操作单元，用于根据所述图形识别结果对应的交互指令，对显示的第一文本信息执行关联操作。
第四方面，本公开实施例还提供计算机存储介质，其上存储有计算机程序，该程序被处理器执行时用于实现上述第一方面所述方法的步骤。
本公开的这些方面或其他方面在以下的实施例的描述中会更加简明易懂。
附图说明
为了更清楚地说明本公开实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简要介绍，显而易见地，下面描述中的附图仅仅是本公开的一些实施例，对于本领域的普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为本公开实施例提供的一种图文交互的方法的实施流程图；
图2为本公开实施例提供的一种图形识别模型的结构示意图；
图3为本公开实施例提供的一种对输入的笔画进行图形识别的示意图；
图4为本公开实施例提供的一种多笔画的交互流程示意图；
图5A-图5B为本公开实施例提供的一种自动生成会议纪要的显示界面图；
图6为本公开实施例提供的一种插入图片的显示界面图；
图7A-图7D为本公开实施例提供的一种生成会议纪要的显示界面图；
图8为本公开实施例提供的一种文本对齐的显示界面图；
图9A-图9B为本公开实施例提供的一种自动转换文本的显示界面图；
图10A-图10B为本公开实施例提供的一种自动生成待办事项的显示界面图；
图11A-图11B为本公开实施例提供的一种智能日历的场景示意图；
图12A-图12B为本公开实施例提供的一种添加待办事项的示意图；
图13为本公开实施例提供的一种提醒待办事项的示意图；
图14为本公开实施例提供的一种显示设备的示意图；
图15为本公开实施例提供的一种图文交互的装置的示意图。
具体实施方式
为了使本公开的目的、技术方案和优点更加清楚，下面将结合附图对本公开作进一步地详细描述，显然，所描述的实施例仅仅是本公开一部分实施例，而不是全部的实施例。基于本公开中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其它实施例，都属于本公开保护的范围。
本公开实施例中术语“和/或”，描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。字符“/”一般表示前后关联对象是一种“或”的关系。
本公开实施例描述的应用场景是为了更加清楚的说明本公开实施例的技术方案，并不构成对于本公开实施例提供的技术方案的限定，本领域普通技术人员可知，随着新应用场景的出现，本公开实施例提供的技术方案对于类似的技术问题，同样适用。其中，在本公开的描述中，除非另有说明，“多个” 的含义是两个或两个以上。
示例的，智能会议是集成了电子白板、电脑和远程会议终端等多种设备的办公产品，具备交互式智能白板等多种功能，随着智能会议的兴起，会议交互体验在企业会议中的重要性逐渐显现出来。目前的智能会议应用于大尺寸的显示设备，在该显示设备上安装有电子白板等书写程序，用户可以在该显示设备上进行书写、绘制图形等操作，并显示用户书写的轨迹、手绘的图形轨迹等，但是用户通过书写或绘制的方式与显示设备进行交互时，交互结果较为单一，通常只是将用户书写的内容或绘制的图形直接进行显示，无法提供更多的交互体验，无法实现更多的交互功能，导致用户的体验感较差。
为了克服上述缺陷，本公开实施例提供了一种图文交互的方法，核心思想是利用手绘的图形触发对用户书写的文本执行交互指令，从而对书写的文本进行关联操作，提供一种利用手绘图形实现对文本操作的功能，应用于智能会议场景中，例如应用于会议书写场景、会议纪要场景、待办事项场景、智能日历场景等，可以通过手绘图形提供用户丰富的交互体验。
如图1所示，本实施例提供的一种图文交互的方法的实施流程如下所示：
步骤100、接收用户手绘的图形的第一轨迹信息；其中所述第一轨迹信息用于触发执行与第一文本信息相关的交互指令；
需要说明的是，本实施例中的用户手绘的图形包括一个或多个子图形；第一轨迹信息包括但不限于一个子图形的子轨迹信息或多个子图形的子轨迹信息，其中每个子图形可以通过一个或多个笔画完成，即子轨迹信息包括一个笔画轨迹或多个笔画轨迹。
本实施例中的手绘的图形包括但不限于各种形状的平面图形、各种类型的线条、特殊字符、标点符号、运算符号中的至少一种，例如手绘的图形包括但不限于三角形、矩形、圆形、菱形、直线、波浪线、曲线、箭头、问号、感叹号、星号、加号中的至少一种。
本实施例中的第一文本信息包括但不限于用户手写的文本、印刷体文本、PDF文本、WORD文本、图片格式的文本中的至少一种，本实施例中的第一 文本信息泛指包含文本内容的媒介，本实施例对第一文本信息的具体形式不作过多限定。
在一些实施例中，在接收用户手绘的图形的第一轨迹信息之前，接收用户书写的文本的笔迹信息，并根据所述笔迹信息确定第一文本信息。本实施例中的第一文本信息可以是用户手写的笔迹信息，也可以是对笔迹信息进行格式转换后得到的信息。实施中，当显示界面没有第一文本信息时，接收用户手绘的图形的第一轨迹信息之后，将所述第一轨迹信息直接进行显示，或将所述第一轨迹信息进行图形识别，将图形识别结果进行显示。可选的，本实施例中的第一文本信息还可以是任何的文档，例如PDF文档、WORD文档等多种格式的文档。
在一些实施例中，所述第一文本信息是根据接收的用户手写的笔迹信息确定的。通过如下任一方式确定所述第一文本信息：
方式1、接收用户书写的文本的笔迹信息，对所述笔迹信息进行轨迹识别确定轨迹识别结果；将所述笔迹信息替换为所述轨迹识别结果，并将所述轨迹识别结果按预设格式进行显示，将显示的轨迹识别结果确定为所述第一文本信息；
方式2、接收并显示用户书写的文本的笔迹信息，将显示的笔迹信息确定为所述第一文本信息。
步骤101、对所述第一轨迹信息进行图形识别，确定图形识别结果；
在一些实施例中，所述图形包括多个子图形，所述第一轨迹信息包括多个子图形的子轨迹信息；
通过如下方式对所述第一轨迹信息进行图形识别，确定图形识别结果：
接收用户手绘的多个子图形的子轨迹信息，对每个子图形的子轨迹信息都进行图形识别，得到多个子图形分别对应的第一识别结果；根据多个第一识别结果，确定所述图形识别结果。
可选的，不限定图形识别的顺序，可以根据预先设定的时间阈值，从用户手绘的多个子图形的子轨迹信息中筛选出属于同一个子图形的子轨迹信息， 然后对属于一个子图形的子轨迹信息进行识别。
在一些实施例中，还可以通过如下方式按顺序进行图形识别：
按照用户手绘的多个子图形的顺序，依次接收多个子图形的子轨迹信息；根据多个子轨迹信息的接收顺序，依次对每个子轨迹信息进行图形识别，得到对应的第一识别结果，并保存所述第一识别结果。
实施中，按照用户手绘的多个子图形的顺序，依次接收多个子图形的子轨迹信息；每接收一个子图形的子轨迹信息，对所述子轨迹信息进行一次图形识别，得到对应的第一识别结果，并保存所述第一识别结果；根据保存的多个子图形对应的第一识别结果，确定最终的所述图形识别结果。
在一些实施例中，当接收的本次子图形的顺序以及本次子图形对应的第一识别结果，满足预先设定的交互策略时，保存本次子图形对应的第一识别结果。
在一些实施例中，保存所述第一识别结果之后，当所述本次子图形不是所述交互策略中最后一个子图形时，根据所述交互策略确定所述本次子图形的下一次子图形的位置范围；当在所述位置范围内未接收到下一次子图形时，清空保存的本次子图形对应的第一识别结果。
实施中，当所述本次子图形不是所述交互策略中最后一个子图形时，根据所述交互策略确定所述本次子图形的下一次子图形的位置范围；在所述位置范围内等待接收下一个子图形的子轨迹信息，若在所述位置范围内接收到下一个子图形的子轨迹信息，则继续对下一个子图形的子轨迹信息进行图形识别，若在所述位置范围内未接收到下一个子图形的子轨迹信息，则停止接收下一个子图形的子轨迹信息并清空保存的所有第一识别结果。
在一些实施例中，本实施例在得到多个子图形分别对应的第一识别结果之后，还可以通过如下方式按顺序确定执行所述图形识别结果对应的交互指令：
获取多个子图形分别对应的第一识别结果的识别顺序，当所述识别顺序满足预设顺序时，确定执行所述多个第一识别结果对应的交互指令。
在一些实施例中，以子图形至少包含第一笔画和第二笔画为例，通过如下方式得到多个子图形分别对应的第一识别结果：
对所述第一笔画进行识别得到第一识别结果，当所述第一识别结果为残缺图形时，保存所述第一笔画；其中，残缺图形是指原始图形包括多个笔画组成，而残缺图形表示缺少原始图形中的部分笔画；
根据缺失的第二笔画的坐标范围，在所述坐标范围内接收所述第二笔画，对所述第一笔画和所述第二笔画一起进行识别得到第一识别结果。
在一些实施例中，当接收的本次子图形的第一识别结果为残缺图形时，保存所述本次子图形的子轨迹信息；根据所述残缺图形缺失的笔画数量和坐标范围，在所述坐标范围内等待接收所述缺失的笔画对应的子轨迹信息；对保存的所述本次子图形的子轨迹信息和所述缺失的笔画对应的子轨迹信息一起进行图形识别，得到所述残缺图形和所述缺失笔画的组合对应的第一识别结果。
实施中，本实施例中的交互策略包括至少一个图形或多个图形以及多个图形的书写顺序；例如，当交互策略为直线+问号时，当用户手绘“直线”，第一识别结果为直线时，保存直线的识别结果并等待接收下一个图形，此时并不会触发交互指令；当用户手绘“残缺问号即缺少问号中的一点”，第一识别结果为“类问号”，则保存类问号的识别结果并在用户手绘“残缺问号”的下面位置范围等待接收下一个图形即问号中的一点，此时仍不会触发交互指令；当用户手绘“一点”后，对用户手绘的“残缺问号”和“一点”的组合进行图形识别，得到第一识别结果，并保存，根据各个第一识别结果确定图形识别结果，当图形识别结果满足交互策略时，执行对应的交互指令。
在一些实施例中，当所述第一识别结果以及所述第一识别结果对应的子图形的顺序，不满足预先设定的交互策略时，清空保存的所有第一识别结果并结束图形识别。
在一些实施例中，以所述图形包括第一子图形和第二子图形；所述第一轨迹信息包括第一子图形的第一子轨迹信息和第二子图形的第二子轨迹信息 为例，本实施例中的执行交互指令的过程如下所示：
接收所述第一子轨迹信息，对所述第一子轨迹信息进行图形识别，得到所述第一子图形的第一识别结果，当所述第一子图形的第一识别结果为第一子图形时，保存所述第一子图形的第一识别结果；
接收所述第二子轨迹信息，对所述第二子轨迹信息进行图形识别，得到所述第二子图形的第一识别结果，当所述第二子图形的第一识别结果为第二子图形时，执行所述第一子图形和所述第二子图形对应的交互指令。
在一些实施例中，所述接收所述第二子轨迹信息，对所述第二子轨迹信息进行图形识别，得到所述第二子图形的第一识别结果，还包括：
当所述第二子图形的第一识别结果不是所述第二子图形时，清空保存的所述第一子图形的第一识别结果。
步骤102、根据所述图形识别结果对应的交互指令，对显示的第一文本信息执行关联操作。
实施中，本实施例中用户手绘图形是为了对已经显示的文本进行相关操作，通过对手绘的图形的第一轨迹信息进行识别的方式，确定对应的交互指令，从而对书写的文本执行的对应的关联操作。例如，在会议场景下，当用户书写完会议内容之后，可以通过在会议内容上进行图形标记，实现对标记的会议内容的相关操作，还可以通过将会议内容进行标注的方式，识别标注的会议内容并根据识别结果自动生成会议纪要。还可以将用户书写的文本进行图形标注，将标注的文本进行识别，并根据识别结果自动生成待办事项。
在一些实施例中，本实施例中的图形包括第一子图形和第二子图形；其中，第一子图形的图形识别结果对应的交互指令，用于确定所述交互指令相关的第一文本信息的范围；所述第二子图形的图形识别结果对应的交互指令，用于确定交互类型，其中不同的交互类型对第一文本信息执行不同的关联操作。
例如，若用户在第一文本信息的第一行的下面手绘“直线”+“问号”，则对所述第一行的文本信息进行第一种关联操作；若用户在第一文本信息的 第一行的下面手绘“直线”+“叹号”，则对所述第一行的文本信息进行第二种关联操作。
在一些实施例中，本实施例还可以对显示的第一文本信息执行如下关联操作：
根据所述图形识别结果对应的交互指令，对显示的第一文本信息中的各行文本和/或各列文本进行对齐处理。
在一些实施例中，通过如下方式根据所述识别结果对应的交互指令，对显示的第一文本信息执行关联操作：
根据所述第一轨迹信息和第一文本信息的显示位置的关系，确定与所述第一轨迹信息相关的第一文本信息；根据所述识别结果对应的交互指令，对与所述第一轨迹信息相关的第一文本信息执行关联操作。
在一些实施例中，可以通过如下方式确定所述第一文本信息：
接收用户书写的文本的笔迹信息，对所述笔迹信息进行轨迹识别确定轨迹识别结果；将所述笔迹信息替换为所述轨迹识别结果，并将所述轨迹识别结果按预设格式进行显示，将显示的轨迹识别结果确定为所述第一文本信息。
在一些实施例中，所述第一文本信息为用户书写的文本的笔迹信息；
所述根据所述识别结果对应的交互指令，对与所述第一轨迹信息相关的第一文本信息执行关联操作，包括：
根据所述识别结果对应的交互指令，将与所述第一轨迹信息相关的第一文本信息替换为与预设标题、预设副标题、预设段落中的任一种对应的格式。
在一些实施例中，通过如下方式将所述轨迹识别结果按预设格式进行显示：
根据所述文本的笔迹信息的落笔位置与当前书写区域中的预设行文本位置或预设列文本位置的对应关系，确定所述笔迹信息的轨迹识别结果对应的预设格式；将所述轨迹识别结果按所述对应的预设格式进行显示。
在一些实施例中，根据笔迹信息和当前书写区域的相对位置关系，通过如下任一或任多种方式显示用户书写的笔迹信息的轨迹识别结果：
方式1、当所述文本的笔迹信息的落笔位置为当前书写区域中的首行文本位置或首列文本位置时，将所述轨迹识别结果显示为预设标题格式对应的标题；
在一些实施例中，显示的标题的字体为预设格式的字体，如黑体、加粗，可选的，显示的标题的字体大小是根据用户书写的该行文本中最大的字体大小确定的，例如首行文本中最大的字体大小为20(px)×20(px)，则标题的字体大小为11号字。
方式2、当所述文本的笔迹信息的落笔位置为当前书写区域中的第二行文本位置或第二列文本位置时，将所述轨迹识别结果显示为预设副标题格式对应的副标题；
在一些实施例中，显示的副标题的字体为预设格式的字体，如黑体，可选的，副标题的字体大小是根据用户书写的该行文本中的最大的字体大小确定的。
方式3、当所述文本的笔迹信息的落笔位置为当前书写区域中的第二行之后的行文本位置或第二列之后的列文本位置时，将所述轨迹识别结果显示为预设段落格式对应的段落内容。
在一些实施例中，显示的段落内容的字体为预设格式的字体，如宋体，可选的，段落内容中字体的大小是根据用户书写的第三行文本中的最大的字体大小确定的。
实施中，在会议纪要场景下，当用户书写文本后，会自动将文本的笔迹信息转换为标题、副标题、段落内容，从而对用户书写的文本进行自动排版，解决了用户手写内容乱，无法整理的问题，提供更加便捷的方式，提高用户的使用体验。
在一些实施例中，在进行标题识别时，用户打开手写软件，将书写的第一行文本自动识别为当前内容的标题，其中标题的字体是预先定义的第一种标准字体，进行文本保存时，也可以按照该标题命名保存。在进行副标题识别时，将用户写下的第二行文本自动识别为副标题，其中副标题的字体是预 先定义的第二种标准字体，还可以自动将副标题向内缩进2个字符。在进行段落识别时，将用户写下的第三行及其之后的文本自动识别为段落内容，并自动缩进。如果段落内容不整齐，按预设行间距自动调整相邻行间的距离，保持整个段落的行间距一致。
在一些实施例中，通过如下方式将所述轨迹识别结果按预设格式进行显示：
根据所述文本的笔迹信息中的字体大小与预设范围的对应关系，确定所述笔迹信息的轨迹识别结果对应的预设格式；将所述轨迹识别结果按所述对应的预设格式进行显示。
在一些实施例中，通过如下任一或任多种方式将所述轨迹识别结果按所述对应的预设格式进行显示：
方式4、当所述文本的笔迹信息中的字体大小符合第一预设范围时，将所述轨迹识别结果显示为预设标题格式对应的标题；
方式5、当所述文本的笔迹信息中的字体大小符合第二预设范围时，将所述轨迹识别结果显示为预设副标题格式对应的副标题；
方式6、当所述文本的笔迹信息中的字体大小符合第三预设范围时，将所述轨迹识别结果显示为预设段落格式对应的段落内容。
其中，所述第一预设范围大于所述第二预设范围，所述第二预设范围大于所述第三预设范围。
在一些实施例中，本实施例中上述方式1和方式4可以组合实施，上述方式2和方式5可以组合实施，上述方式3和方式6可以组合实施。其中组合实施的方案此处不再赘述。实施中，一方面可以根据用户书写的文本位置确定当前行的笔迹信息的轨迹识别结果的预设格式；另一方面可以根据用户书写的当前行的字体的大小确定显示的当前行的笔迹信息的轨迹识别结果的预设格式；最后一方面，还可以结合用户书写的文本位置和字体大小，确定当前行的笔迹信息的轨迹识别结果的预设格式；其中本实施例中的预设格式包括但不限于字体种类、字体大小、是否加粗、是否有下划线等各类型的格 式。
在一些实施例中，本实施例还可以通过如下方式确定所述第一文本信息：
接收并显示用户书写的文本的笔迹信息，将显示的笔迹信息确定为所述第一文本信息。
需要说明的是，本实施例中的第一文本信息可以是用户书写的文本的笔迹信息，也可以是对该笔迹信息进行转换后得到的轨迹信息，本实施例对此不作过多限定。
在一些实施例中，本实施例通过如下方式对所述第一轨迹信息进行图形识别，确定图形识别结果：
第一种方式、用户手绘的图形中的其中一个子图形包括一个笔画。
实施中，对所述笔画轨迹进行图形识别，确定图形识别结果。
例如，用户手绘一个包含一个笔画的圆形或矩形，对该笔画进行图形识别确定图形识别结果为一个圆形或矩形。
第二种方式，用户手绘的图形中的其中一个子图形包括多个笔画。
实施中，按照用户手绘的子图形的笔画顺序，依次对所述笔画顺序连续的多个笔画的组合进行图形识别，确定图形识别结果。
当包括多个笔画时，存在多种情况如下所示：
情况1)多个笔画为多个图形，每个图形包含一个笔画；
情况2)多个笔画为一个图形；
情况3)多个笔画为多个图形，每个图形包含多个笔画；
情况4)多个笔画为多个图形，至少一个图形包含一个笔画，至少一个图形包含多个笔画。
实施中，上述情况中都可以先按照用户手绘的子图形的笔画顺序，依次对所述多个笔画进行图形识别，当所述笔画的图形识别结果为一个完整图形时，继续对下一个笔画进行识别；当所述笔画的图形识别结果为一个残缺图形时，则先保留该笔画，将该笔画和该笔画之后的连续至少一个笔画作为一个整体进行识别，直至确定该识别结果是一个完整图形时停止。
在一些实施例中，通过如下方式依次对所述笔画顺序连续的多个笔画的组合进行图形识别，确定图形识别结果：
(1)当所述笔画的图形识别结果为一个残缺图形时，确定所述残缺图形缺失的笔画的数量和坐标范围；
(2)按照所述数量和坐标范围，确定所述笔画之后的连续至少一个笔画；
(3)将所述笔画以及所述笔画之后的连续至少一个笔画的组合进行图形识别，确定图形识别结果。
实施中，以用户手绘一个问号“？”为例，其中问号包括两个笔画，当对用户手绘的第一个笔画进行图形识别后，确定该笔画的图形识别结果为一个残缺图形时，确定缺失的笔画的数量和坐标，即确定缺失一个笔画且坐标应位于第一个笔画的下方，则按照该数量和坐标，确定该第一个笔画下面的一个笔画，对该笔画和第一个笔画的组合进行图形识别，即将该笔画和第一个笔画作为一个整体进行图形识别，确定图形识别结果为问号。
在一些实施例中，本实施例中的图形识别包括但不限于OCR(Optical Character Recognition，光学字符识别)，是指电子设备(例如扫描仪或数码相机)检查纸上打印的字符，通过检测暗、亮的模式确定其形状，然后用字符识别方法将形状翻译成计算机文字的过程。其中，OCR还用于文本识别、轨迹识别等。
OCR识别根据处理方法可分为三个阶段：预处理、文本识别或图形识别、后处理；其中，预处理主要包括灰度化，二值化，噪声去除，倾斜矫正等；图形识别主要包括CNN+RNN+CTC，或者，CNN+RNN+Attention(注意力模型)。
本实施例中的图形识别算法或轨迹识别算法包括但不限于：EAST、CTPN、SegLink、PixelLink、TextBoxes、TextBoxes++、TextSnake、MSR中的至少一种。
可选的，用户每写完一个笔画，会调用OCR将当前画面保存，便于后续进行图形识别时，截取出相关笔画进行图形识别。
实施中，本实施例可以通过LSTM+linear(线性变换层)的网络进行图形识别，其中LSTM网络结构的层数可以根据任务的难易程度以及耗时情况确定，如使用一层或者多层，其中LSTM的参数bidirectional设置为true，表示使用双向LSTM，使用输入的前后信息；linear是线性变换层，用于输出指定维度。如图2所示，本实施例提供一种图形识别模型，包括LSTM网络层200和线性变换层201，也可根据需要使用附加结构的RNN网络，可以使用分类损失函数进行RNN的训练。其中，LSTM用于对输入的笔画进行特征提取得到笔画特征，linear用于对输入的笔画特征进行分类，确定该笔画的图形类型和坐标。
如图3所示，本实施例还提供一种对输入的笔画进行图形识别的示意图，其中，根据用户每次的下笔-抬笔，确定接收的每个笔画，按照书写的顺序即接收的笔画顺序，依次将每个笔画中的笔迹点坐标输入到LSTM+线性变换层linear进行识别，得到该笔画对应的图形类型。其中可以对每个笔画的笔迹点进行稀疏处理后，输入到LSTM+linear进行识别，以提高识别效率。
如图4所示，本实施例提供一种多笔画的交互流程，该流程实施的具体步骤如下所示：
步骤400、接收用户手绘的图形的第一轨迹信息，其中所述图形包括直线和问号，直线包括一个笔画，问号包括两个笔画；
步骤401、按照对用户手绘的笔画顺序，对第一轨迹信息中与直线相关的一个笔画进行图形识别，得到该笔画的图形类型为直线；
步骤402、继续第一轨迹信息中与问号相关的第一个笔画进行图形识别，得到该笔画的类型为不完整的问号；
步骤403、对第一轨迹信息中与问号相关的第一个笔画和第二个笔画的组合进行图形识别，得到的图形类型为问号；
步骤404、根据直线+问号对应的交互指令，对与第一轨迹信息相关的第一文本信息执行关联操作。
在一些实施例中，本实施例通过如下方式根据所述识别结果对应的交互 指令，对显示的第一文本信息执行关联操作：
根据所述第一轨迹信息和第一文本信息的显示位置的关系，确定与所述第一轨迹信息相关的第一文本信息；根据所述识别结果对应的交互指令，对与所述第一轨迹信息相关的第一文本信息执行关联操作。可选的，本实施例中确定出与述第一轨迹信息相关的第一文本信息后，首先将其从对应的第一文本信息中筛选出来，然后再根据用户触发的交互指令执行与关联操作。其中筛选出的第一文本信息是文本格式的信息。
实施中，本实施例可以通过第一轨迹信息对已经显示的部分或全部的第一文本信息执行关联操作，本实施例中的与所述第一轨迹信息相关的第一文本信息为已经显示的至少部分第一文本信息，根据第一轨迹信息的识别结果对应的交互指令，对与第一轨迹信息相关的第一文本信息执行关联操作。
可选的，本实施例中的第一轨迹信息和第一文本信息的显示位置的关系包括但不限于，第一轨迹信息将第一文本信息完全包围，或，第一轨迹信息位于第一文本信息下方，或第一轨迹信息位于第一文本信息一侧中的至少一种。需要说明的是，本实施例中的第一轨迹信息和第一文本信息的显示位置的关系仅为一种示例，目的是利用第一轨迹信息对第一文本信息进行标注、关联操作等，基于该发明构思的其他位置关系也属于本公开的保护范围。
在一些实施例中，本实施例通过如下步骤根据所述识别结果对应的交互指令，对与所述第一轨迹信息相关的第一文本信息执行关联操作：
根据所述识别结果的交互指令，对与所述第一轨迹信息相关的第一文本信息进行文本识别，得到并显示文本识别结果。
在一些实施例中，根据所述识别结果的交互指令，对与所述第一轨迹信息相关的第一文本信息进行语义识别，根据所述语义识别结果生成会议纪要或待办事项。
其中，本实施例中的会议纪要表征在会议场景下，与用户手写的会议内容相关的会议概要，包括但不限于会议标题、会议内容、会议时间、会议地点中的至少一种。本实施例中的待办事项表征在待办事项场景下，与用户手 写的待办事项相关的内容，包括但不限于日程主题、日程内容、日程时间、日程地点中的至少一种。
实施中，用户在书写软件中书写文本内容后，通过手绘的图形的第一轨迹信息，触发对用户书写的文本进行文本识别和语义识别，识别出该文本中的标题、内容、时间、地点等信息，从而根据不同的场景如会议场景或待办事项场景，根据识别出的信息自动生成会议纪要或待办事项，提高用户的使用体验。
在一些实施例中，根据所述语义识别结果中的会议标题、会议内容、会议时间、会议地点中的至少一种，生成会议纪要。
实施中，根据预先定义的会议纪要模板，将会议标题、会议内容、会议时间、会议地点中的至少一种，添加到会议纪要模板的对应位置，即将会议标题添加到会议纪要模板的标题位置，将时间、地点、内容分别添加到会议纪要模板的对应位置，从而自动生成会议纪要。
如图5A-图5B所示，本实施例提供一种自动生成会议纪要的显示界面，在图5A中，用户在书写软件中手写会议标题、会议内容、会议时间和会议地点，完成会议相关文本的书写后，用户通过手绘图形指定会议相关文本中的全部或部分内容，如用户通过手绘矩形将书写的全部文本包围，此时触发执行与该全部文本的交互指令，对该全部文本进行语义识别，得到会议标题、会议内容、会议时间和会议地点；然后如图5B所示，根据预先设定的会议纪要模板，将语义识别得到的会议标题、会议内容、会议时间和会议地点分别添加到会议纪要模板中，生成会议纪要。
在一些实施例中，本实施例在书写文本时还可以插入图片、网页、文件等内容，执行如下步骤：
插入与所述第一文本信息相关的图片、网页、文件中的至少一种内容；
根据所述识别结果对应的交互指令，通过超文本标记语言格式显示插入的内容。
在一些实施例中，在插入与所述第一文本信息相关的图片之后，还可以 接收用户手绘的第三轨迹信息，根据所述第三轨迹信息指向的文本位置，将所述插入的图片排列在所述文本位置相邻一侧。
实施中，如图6所示，本实施例提供了一种插入图片的显示界面，当用户插入图片到书写软件中，还可以接收用户手绘的箭头，利用该箭头连接图片和文本，并根据该箭头指向的文本位置，将插入的图片排列在该文本位置的下面。根据用户手绘的第一轨迹信息的识别结果对应的交互指令，通过HTML5的方式进行显示，使得用户可以再次查看插入的图片、网页、文件等。
在一些实施例中，当所述第一文本信息还包括与所述第一文本信息相关的图片、网页、文件中的至少一种时，根据所述语义识别结果生成会议纪要，还可以将所述图片、网页、文件中的至少一种，以超文本标记语言格式添加到所述会议纪要中。用户可以在会议纪要中再次查看用户插入的图片、网页、文件等内容。
实施中，当将插入图片、网页、文件的文本信息进行识别并生成会议纪要时，通过HTML5的方式显示插入的内容，并添加到会议纪要中的相应位置。如图7A-图7D所示，本实施例提供了一种生成会议纪要的显示界面，在图7A中，该显示界面中显示会议标题、插入的图片的图标、插入的网页的网址链接、插入的文件的名称以及会议时间，当用户点击插入的图片的图标后，如图7B所示，显示插入的图片，并自动调整该图标后的内容的显示位置，如图7C所示，当用户点击网址链接后，打开对应的浏览器界面显示该网页，当用户点击文件的名称后，打开文件阅读程序并显示该文件内容。
在一些实施例中，所述会议纪要包括会议时间；本实施例还可以响应于用户对会议时间的点击操作，启动日历程序并将所述会议纪要作为一个待办事项添加到所述日历程序中；或者接收与所述会议时间相关的第一轨迹信息，对所述第一轨迹信息进行图形识别，确定图形识别结果；根据所述图形识别结果对应的交互指令，启动日历程序，并将所述会议纪要作为一个待办事项添加到所述日历程序中。
实施中，如图7D所示，当用户点击会议时间，则打开日历程序，将该会 议时间添加到对应日期，将该会议生成一个待办事项添加到日历程序中，作为提醒日程。
可选的，根据所述图形识别结果对应的交互指令，对显示的第一文本信息中的各行文本和/或各列文本进行对齐处理。如图8所示，本实施例提供一种文本对齐的显示界面，当用户在书写的文本左侧手绘一条直线，则将该直线右侧的文本进行识别后得到的印刷体文本左对齐。
如图9A-图9B所示，本实施例还提供一种自动转换文本的显示界面，如图9A所示，当用户在书写文本中插入了图片、网页和文件，将用户书写的文本转换后，如图9B所示，将图片插入到用户手绘的箭头连接的文本的下方，将网页和文件以HTML5的格式进行显示，便于用户进行查看。
在一些实施例中，根据所述语义识别结果中的日程主题、日程内容、日程时间、日程地点中的至少一种生成待办事项。
实施中，根据预先定义的待办事项模板，将日程主题、日程内容、日程时间、日程地点分别添加到待办事项模板的对应位置，从而自动生成待办事项。
如图10A-图10B所示，本实施例提供一种自动生成待办事项的显示界面，其中在图10A中，用户手写待办事项的日程主题、日程内容、日程时间、日程地点，然后用户手绘矩形将手写的待办事项的全部文本置于矩形内部，触发执行对该待办事项的全部文本的语义识别，如图10B所示，将语义识别得到的日程主题、日程内容、日程时间、日程地点，添加到预设待办事项模板中的对应位置，从而自动生成待办事项。
在一些实施例中，本实施例根据所述语义识别结果生成待办事项，还可以根据所述语义识别结果中的至少一个关键词关联的应用程序，确定所述待办事项关联的应用程序；在所述待办事项的日程时间，启动所述待办事项关联的应用程序。
实施中，例如当语义识别结果中存在关键词“会议”、“开会”、“讨论”、“远程”等，则将该待办事项关联到视频会议应用；当语义识别结果中存在 关键词“传输”、“拷贝”、“文件”，则将该待办事项关联到文件快传应用；当语义识别结果中存在关键词“书写”、“笔迹”、“记录”、“纪要”、“表格”，则将该待办事项关联到工作台应用；当语义识别结果中存在关键词“图”、“画”、“绘制”、“素描”，则将该待办事项关联到绘图板；当语义识别结果中存在关键词“检索”、“搜索”、“查询”、“网页”，则将该待办事项关联到浏览器。当达到待办事项的日程时间时，启动该待办事项关联的应用程序。
在一些实施例中，如图11A-图11B所示，本实施例还提供一种智能日历的场景，该场景下，将日历程序以窗口形式显示在桌面启动器的首页，并且在日历程序中显示不同日期的日程。实施中，日历程序的浮窗默认显示当月的日期，有日程(待办事项)的日期用提醒标识进行标记。用户可以对已经添加的日程进行查看，将添加的日程按照起始时间的先后顺序进行显示，当不同日程的起始时间一致或发生冲突，则按照添加日程的时间的先后顺序，显示冲突的日程，当日程的内容超出显示区域的显示范围，则可以通过下拉、下滑的操作进行显示。当用户未添加当日的日程时，则提示“当日无日程”。用户还可以手动开启或关闭日历程序，当待办事项关联的应用程序在待办事项的日程时间已经打开时，不再弹出弹窗提醒用户，当待办事项关联的应用程序在待办事项的日程时间还未打开时，按照待办事项的设置提醒用户。当已经出现对某个应用程序的提醒弹窗，在提示该应用程序打开的倒计有效时间时内又弹出新的应用程序的提醒弹窗，则多个弹窗错位显示。用户还可以对任一日程进行编辑、修改，还可以添加新的日程。
在一些实施例中，本实施例通过如下步骤对所述文本识别结果进行语义识别，根据所述语义识别结果生成待办事项：
将所述文本识别结果添加到日程编辑界面的第一编辑区域；对所述编辑区域中的文本识别结果进行语义识别，将语义识别结果添加到至少一个第二编辑区域；根据至少一个第二编辑区域中的语义识别结果，生成待办事项。
实施中，如图12A-图12B所示，本实施例提供一种添加待办事项的示意图，其中，用户书写文本后，通过手绘矩形将该书写的文本内容全部指定， 对该文本内容进行文本识别，如图12A所示，将文本识别结果在日程编辑界面的第一编辑区域进行显示，用户可以继续对文本识别结果进行编辑，也可以不进行编辑，直接进行语义识别，如图12B所示，当语义识别结果包括日程主题、日程内容、日程时间、日程地点时，分别将日程主题、日程内容、日程时间、日程地点添加到对应的第二编辑区域进行显示，用户还可以继续对第二编辑区域显示的内容进行编辑，也可以直接根据各个第二编辑区域中的内容，生成待办事项。
在一些实施例中，本实施例根据所述语义识别结果生成待办事项之后，还可以将根据所述语义识别结果生成的待办事项，添加到日历程序中进行显示；其中，所述日历程序以窗口形式在桌面启动器的首页进行显示。
实施中，用户在书写软件中书写文本，用户通过绘制图形的第一轨迹信息指定相关的文本的第一文本信息，对与所述第一轨迹信息相关的第一文本信息进行文本识别，得到文本识别结果；对所述文本识别结果进行语义识别，根据所述语义识别结果生成待办事项。其中，本实施例中的文本识别结果包括标准字体和/或标准格式的文本，语义识别结果包括但不限于：主题、内容、时间、地点、预设关键词中的至少一种。
可选的，添加待办事项后，用户还可以设置提醒时间，例如可以选择待办事项的日程时间开始之前的10分钟、15分钟、20分钟、30分钟进行提醒。如图13所示，本实施例提供一种提醒待办事项的示意图，当待办事项存在关联的应用程序时，可以在该待办事项的日程时间开始之前以全局弹窗形式提示用户关联的应用程序即将启动，除弹窗内其余地方操作失效，例如可以使用倒计时提醒，若倒计时为0，则弹窗自动关闭，打开相关应用程序，当用户点击弹窗显示的提示语中的“打开”，则直接打开关联的应用程序，当用户点击弹窗显示的提示语中的“取消打开”，则弹窗消失且应用程序不打开，当用户点击弹窗显示的提示语中的“知道了”，则弹窗消失，到了预约的日程时间自动打开关联的应用程序。
示例的，基于相同的发明构思，本公开实施例还提供了一种显示设备，由于该设备即是本公开实施例中的方法中的设备，并且该设备解决问题的原理与该方法相似，因此该设备的实施可以参见方法的实施，重复之处不再赘述。
如图14所示，该显示设备包括显示屏1400和处理器1401，其中：
所述显示屏1400用于与用户进行交互，并进行内容的显示；
所述处理器1401被配置为执行如下步骤：
接收用户手绘的图形的第一轨迹信息；其中所述第一轨迹信息用于触发执行与第一文本信息相关的交互指令；
对所述第一轨迹信息进行图形识别，确定图形识别结果；
根据所述图形识别结果对应的交互指令，对显示的第一文本信息执行关联操作。
作为一种可选的实施方式，所述第一文本信息是根据接收的用户手写的笔迹信息确定的。
作为一种可选的实施方式，所述处理器具体被配置为通过如下方式确定所述第一文本信息：
接收用户书写的文本的笔迹信息，对所述笔迹信息进行轨迹识别确定轨迹识别结果；将所述笔迹信息替换为所述轨迹识别结果，并将所述轨迹识别结果按预设格式进行显示，将显示的轨迹识别结果确定为所述第一文本信息；或，
接收并显示用户书写的文本的笔迹信息，将显示的笔迹信息确定为所述第一文本信息。
作为一种可选的实施方式，所述图形包括多个子图形，所述第一轨迹信息包括多个子图形的子轨迹信息；所述处理器具体被配置为执行：
接收用户手绘的多个子图形的子轨迹信息，对所述子图形的子轨迹信息进行图形识别，得到多个子图形分别对应的第一识别结果；
根据多个第一识别结果，确定所述图形识别结果。
作为一种可选的实施方式，所述处理器具体被配置为执行：
按照用户手绘的多个子图形的顺序，依次接收多个子图形的子轨迹信息；
根据多个子轨迹信息的接收顺序，依次对每个子轨迹信息进行图形识别，得到对应的第一识别结果，并保存所述第一识别结果。
作为一种可选的实施方式，所述处理器具体被配置为执行：
当接收的本次子图形的顺序以及本次子图形对应的第一识别结果，满足预先设定的交互策略时，保存本次子图形对应的第一识别结果。
作为一种可选的实施方式，所述保存本次子图形对应的第一识别结果之后，所述处理器具体还被配置为执行：
当所述本次子图形不是所述交互策略中最后一个子图形时，根据所述交互策略确定所述本次子图形的下一次子图形的位置范围；
当在所述位置范围内未接收到下一次子图形时，清空保存的本次子图形对应的第一识别结果。
作为一种可选的实施方式，所述得到多个子图形分别对应的第一识别结果之后，所述处理器具体被配置为通过如下方式确定执行所述图形识别结果对应的交互指令：
获取多个子图形分别对应的第一识别结果的识别顺序，当所述识别顺序满足预设顺序时，确定执行所述多个第一识别结果对应的交互指令。
作为一种可选的实施方式，所述子图形至少包含第一笔画和第二笔画；所述处理器具体被配置为执行：
对所述第一笔画进行识别得到第一识别结果，当所述第一识别结果为残缺图形时，保存所述第一笔画；
根据缺失的第二笔画的坐标范围，在所述坐标范围内接收所述第二笔画，对所述第一笔画和所述第二笔画一起进行识别得到第一识别结果。
作为一种可选的实施方式，所述图形包括第一子图形和第二子图形；所述第一轨迹信息包括第一子图形的第一子轨迹信息和第二子图形的第二子轨迹信息；所述处理器具体被配置为通过如下方式执行所述交互指令：
接收所述第一子轨迹信息，对所述第一子轨迹信息进行图形识别，得到所述第一子图形的第一识别结果，当所述第一子图形的第一识别结果为第一子图形时，保存所述第一子图形的第一识别结果；
接收所述第二子轨迹信息，对所述第二子轨迹信息进行图形识别，得到所述第二子图形的第一识别结果，当所述第二子图形的第一识别结果为第二子图形时，执行所述第一子图形和所述第二子图形对应的交互指令。
作为一种可选的实施方式，所述图形包括第一子图形和第二子图形；
所述第一子图形的图形识别结果对应的交互指令，用于确定所述交互指令相关的第一文本信息的范围；
所述第二子图形的图形识别结果对应的交互指令，用于确定交互类型，其中不同的交互类型对第一文本信息执行不同的关联操作。
作为一种可选的实施方式，所述处理器具体被配置为执行：
根据所述图形识别结果对应的交互指令，对显示的第一文本信息中的各行文本和/或各列文本进行对齐处理。
作为一种可选的实施方式，所述处理器具体被配置为执行：
根据所述第一轨迹信息和第一文本信息的显示位置的关系，确定与所述第一轨迹信息相关的第一文本信息；
根据所述识别结果对应的交互指令，对与所述第一轨迹信息相关的第一文本信息执行关联操作。
作为一种可选的实施方式，所述第一文本信息为用户书写的文本的笔迹信息；所述处理器具体被配置为执行：
根据所述识别结果对应的交互指令，将与所述第一轨迹信息相关的第一文本信息替换为与预设标题、预设副标题、预设段落中的任一种对应的格式。
作为一种可选的实施方式，所述处理器具体被配置为执行：
根据所述识别结果的交互指令，对与所述第一轨迹信息相关的第一文本信息进行文本识别，得到并显示文本识别结果。
作为一种可选的实施方式，所述处理器具体被配置为执行：
根据所述识别结果的交互指令，对与所述第一轨迹信息相关的第一文本信息进行语义识别，根据所述语义识别结果生成会议纪要或待办事项。
作为一种可选的实施方式，所述处理器具体被配置为执行：
根据所述语义识别结果中的会议标题、会议内容、会议时间、会议地点中的至少一种，生成会议纪要。
作为一种可选的实施方式，所述第一文本信息包括会议纪要；所述会议纪要包括会议时间；所述处理器具体还被配置为：
接收与所述会议时间相关的第一轨迹信息，对所述第一轨迹信息进行图形识别，确定图形识别结果；
根据所述图形识别结果对应的交互指令，启动日历程序，并将所述会议纪要作为一个待办事项添加到所述日历程序中。
作为一种可选的实施方式，所述处理器具体还被配置为执行：
根据所述语义识别结果中的至少一个关键词关联的应用程序，确定所述待办事项关联的应用程序；
在所述待办事项的日程时间，启动所述待办事项关联的应用程序。
作为一种可选的实施方式，所述处理器具体被配置为执行：
将所述文本识别结果添加到日程编辑界面的第一编辑区域；
对所述编辑区域中的文本识别结果进行语义识别，将语义识别结果添加到至少一个第二编辑区域；
根据至少一个第二编辑区域中的语义识别结果，生成待办事项。
作为一种可选的实施方式，所述根据所述语义识别结果生成待办事项之后，所述处理器具体还被配置为执行：
将根据所述语义识别结果生成的待办事项，添加到日历程序中进行显示；
其中，所述日历程序以窗口形式在桌面启动器的首页进行显示。
示例的，基于相同的发明构思，本公开实施例还提供了一种图文交互的装置，由于该装置即是本公开实施例中的方法中的装置，并且该装置解决问 题的原理与该方法相似，因此该装置的实施可以参见方法的实施，重复之处不再赘述。
如图15所示，该装置包括：
手绘图形单元1500，用于接收用户手绘的图形的第一轨迹信息；其中所述第一轨迹信息用于触发执行与第一文本信息相关的交互指令；
图形识别单元1501，用于对所述第一轨迹信息进行图形识别，确定图形识别结果；
关联操作单元1502，用于根据所述图形识别结果对应的交互指令，对显示的第一文本信息执行关联操作。
基于相同的发明构思，本公开实施例还提供了一种计算机存储介质，其上存储有计算机程序，该程序被处理器执行时用于实现如下步骤：
接收用户手绘的图形的第一轨迹信息；其中所述第一轨迹信息用于触发执行与第一文本信息相关的交互指令；
对所述第一轨迹信息进行图形识别，确定图形识别结果；
根据所述图形识别结果对应的交互指令，对显示的第一文本信息执行关联操作。
本领域内的技术人员应明白，本公开的实施例可提供为方法、系统、或计算机程序产品。因此，本公开可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本公开可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器和光学存储器等)上实施的计算机程序产品的形式。
本公开是参照根据本公开实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流 程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的设备。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令设备的制造品，该指令设备实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
显然，本领域的技术人员可以对本公开进行各种改动和变型而不脱离本公开的精神和范围。这样，倘若本公开的这些修改和变型属于本公开权利要求及其等同技术的范围之内，则本公开也意图包含这些改动和变型在内。

Claims (23)

  1. 一种图文交互的方法，其中，该方法包括：
    接收用户手绘的图形的第一轨迹信息；其中所述第一轨迹信息用于触发执行与第一文本信息相关的交互指令；
    对所述第一轨迹信息进行图形识别，确定图形识别结果；
    根据所述图形识别结果对应的交互指令，对显示的第一文本信息执行关联操作。
  2. 根据权利要求1所述的方法，其中，所述第一文本信息是根据接收的用户手写的笔迹信息确定的。
  3. 根据权利要求2所述的方法，其中，通过如下方式确定所述第一文本信息：
    接收用户书写的文本的笔迹信息，对所述笔迹信息进行轨迹识别确定轨迹识别结果；将所述笔迹信息替换为所述轨迹识别结果，并将所述轨迹识别结果按预设格式进行显示，将显示的轨迹识别结果确定为所述第一文本信息；或，
    接收并显示用户书写的文本的笔迹信息，将显示的笔迹信息确定为所述第一文本信息。
  4. 根据权利要求1所述的方法，其中，所述图形包括多个子图形，所述第一轨迹信息包括多个子图形的子轨迹信息；
    所述对所述第一轨迹信息进行图形识别，确定图形识别结果，包括：
    接收用户手绘的多个子图形的子轨迹信息，对所述子图形的子轨迹信息进行图形识别，得到多个子图形分别对应的第一识别结果；
    根据多个第一识别结果，确定所述图形识别结果。
  5. 根据权利要求4所述的方法，其中，所述对每个子图形的子轨迹信息都进行图形识别，得到多个子图形分别对应的第一识别结果，包括：
    按照用户手绘的多个子图形的顺序，依次接收多个子图形的子轨迹信息；
    根据多个子轨迹信息的接收顺序，依次对每个子轨迹信息进行图形识别，得到对应的第一识别结果，并保存所述第一识别结果。
  6. 根据权利要求5所述的方法，其中，所述保存所述第一识别结果，包括：
    当接收的本次子图形的顺序以及本次子图形对应的第一识别结果，满足预先设定的交互策略时，保存本次子图形对应的第一识别结果。
  7. 根据权利要求6所述的方法，其中，所述保存本次子图形对应的第一识别结果之后，还包括：
    当所述本次子图形不是所述交互策略中最后一个子图形时，根据所述交互策略确定所述本次子图形的下一次子图形的位置范围；
    当在所述位置范围内未接收到下一次子图形时，清空保存的本次子图形对应的第一识别结果。
  8. 根据权利要求4所述的方法，其中，所述得到多个子图形分别对应的第一识别结果之后，通过如下方式确定执行所述图形识别结果对应的交互指令：
    获取多个子图形分别对应的第一识别结果的识别顺序，当所述识别顺序满足预设顺序时，确定执行所述多个第一识别结果对应的交互指令。
  9. 根据权利要求4所述的方法，其中，所述子图形至少包含第一笔画和第二笔画；
    所述对每个子图形的子轨迹信息都进行图形识别，得到多个子图形分别对应的第一识别结果，包括：
    对所述第一笔画进行识别得到第一识别结果，当所述第一识别结果为残缺图形时，保存所述第一笔画；
    根据缺失的第二笔画的坐标范围，在所述坐标范围内接收所述第二笔画，对所述第一笔画和所述第二笔画一起进行识别得到第一识别结果。
  10. 根据权利要求1～9任一所述的方法，其中，所述图形包括第一子图形和第二子图形；所述第一轨迹信息包括第一子图形的第一子轨迹信息和第 二子图形的第二子轨迹信息；通过如下方式执行所述交互指令：
    接收所述第一子轨迹信息，对所述第一子轨迹信息进行图形识别，得到所述第一子图形的第一识别结果，当所述第一子图形的第一识别结果为第一子图形时，保存所述第一子图形的第一识别结果；
    接收所述第二子轨迹信息，对所述第二子轨迹信息进行图形识别，得到所述第二子图形的第一识别结果，当所述第二子图形的第一识别结果为第二子图形时，执行所述第一子图形和所述第二子图形对应的交互指令。
  11. 根据权利要求1～9任一所述的方法，其中，所述图形包括第一子图形和第二子图形；
    所述第一子图形的图形识别结果对应的交互指令，用于确定所述交互指令相关的第一文本信息的范围；
    所述第二子图形的图形识别结果对应的交互指令，用于确定交互类型，其中不同的交互类型对第一文本信息执行不同的关联操作。
  12. 根据权利要求1～9任一所述的方法，其中，根据所述图形识别结果对应的交互指令，对显示的第一文本信息执行关联操作，包括：
    根据所述图形识别结果对应的交互指令，对显示的第一文本信息中的各行文本和/或各列文本进行对齐处理。
  13. 根据权利要求1～9任一所述的方法，其中，所述根据所述识别结果对应的交互指令，对显示的第一文本信息执行关联操作，包括：
    根据所述第一轨迹信息和第一文本信息的显示位置的关系，确定与所述第一轨迹信息相关的第一文本信息；
    根据所述识别结果对应的交互指令，对与所述第一轨迹信息相关的第一文本信息执行关联操作。
  14. 根据权利要求13所述的方法，其中，所述第一文本信息为用户书写的文本的笔迹信息；
    所述根据所述识别结果对应的交互指令，对与所述第一轨迹信息相关的第一文本信息执行关联操作，包括：
    根据所述识别结果对应的交互指令，将与所述第一轨迹信息相关的第一文本信息替换为与预设标题、预设副标题、预设段落中的任一种对应的格式。
  15. 根据权利要求1～9任一所述的方法，其中，所述根据所述识别结果对应的交互指令，对显示的第一文本信息执行关联操作，包括：
    根据所述识别结果的交互指令，对与所述第一轨迹信息相关的第一文本信息进行文本识别，得到并显示文本识别结果。
  16. 根据权利要求1～9任一所述的方法，其中，所述根据所述识别结果对应的交互指令，对与所述第一轨迹信息相关的第一文本信息执行关联操作，包括：
    根据所述识别结果的交互指令，对与所述第一轨迹信息相关的第一文本信息进行语义识别，根据所述语义识别结果生成会议纪要或待办事项。
  17. 根据权利要求16所述的方法，其中，所述根据所述语义识别结果生成会议纪要，包括：
    根据所述语义识别结果中的会议标题、会议内容、会议时间、会议地点中的至少一种，生成会议纪要。
  18. 根据权利要求16所述的方法，其中，所述第一文本信息包括会议纪要；所述会议纪要包括会议时间；该方法还包括：
    接收与所述会议时间相关的第一轨迹信息，对所述第一轨迹信息进行图形识别，确定图形识别结果；
    根据所述图形识别结果对应的交互指令，启动日历程序，并将所述会议纪要作为一个待办事项添加到所述日历程序中。
  19. 根据权利要求16所述的方法，其中，所述根据所述语义识别结果生成待办事项，还包括：
    根据所述语义识别结果中的至少一个关键词关联的应用程序，确定所述待办事项关联的应用程序；
    在所述待办事项的日程时间，启动所述待办事项关联的应用程序。
  20. 根据权利要求16所述的方法，其中，所述对所述文本识别结果进行 语义识别，根据所述语义识别结果生成待办事项，包括：
    将所述文本识别结果添加到日程编辑界面的第一编辑区域；
    对所述编辑区域中的文本识别结果进行语义识别，将语义识别结果添加到至少一个第二编辑区域；
    根据至少一个第二编辑区域中的语义识别结果，生成待办事项。
  21. 根据权利要求16所述的方法，其中，所述根据所述语义识别结果生成待办事项之后，还包括：
    将根据所述语义识别结果生成的待办事项，添加到日历程序中进行显示；
    其中，所述日历程序以窗口形式在桌面启动器的首页进行显示。
  22. 一种显示设备，其中，包括显示屏和处理器，其中：
    所述显示屏用于与用户进行交互，并进行内容的显示；
    所述处理器被配置为执行权利要求1～21任一所述方法的步骤。
  23. 一种计算机存储介质，其上存储有计算机程序，其中，该程序被处理器执行时实现如权利要求1～21任一所述方法的步骤。
PCT/CN2022/109143 2022-07-29 2022-07-29 一种图文交互的方法及显示设备 WO2024021078A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
CN202280002471.9A CN117813580A (zh) 2022-07-29 2022-07-29 一种图文交互的方法及显示设备
PCT/CN2022/109143 WO2024021078A1 (zh) 2022-07-29 2022-07-29 一种图文交互的方法及显示设备
CN202310473009.3A CN116578224A (zh) 2022-07-29 2023-04-27 一种基于图形识别交互的方法及显示设备
PCT/CN2023/105899 WO2024022068A1 (zh) 2022-07-29 2023-07-05 一种基于图形识别交互的方法及显示设备

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2022/109143 WO2024021078A1 (zh) 2022-07-29 2022-07-29 一种图文交互的方法及显示设备

Publications (1)

Publication Number Publication Date
WO2024021078A1 true WO2024021078A1 (zh) 2024-02-01

Family

ID=87542426

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/CN2022/109143 WO2024021078A1 (zh) 2022-07-29 2022-07-29 一种图文交互的方法及显示设备
PCT/CN2023/105899 WO2024022068A1 (zh) 2022-07-29 2023-07-05 一种基于图形识别交互的方法及显示设备

Family Applications After (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/105899 WO2024022068A1 (zh) 2022-07-29 2023-07-05 一种基于图形识别交互的方法及显示设备

Country Status (2)

Country Link
CN (2) CN117813580A (zh)
WO (2) WO2024021078A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101667074A (zh) * 2008-09-03 2010-03-10 联想(北京)有限公司 一种通过图形输入完成操作的装置及其方法
CN104360808A (zh) * 2014-12-04 2015-02-18 李方 一种利用符号手势指令进行文档编辑的方法及装置
CN106326343A (zh) * 2016-08-05 2017-01-11 重庆锐畅科技有限公司 一种基于音视频数据关联同步的电子白板数据共享系统
CN106600223A (zh) * 2016-12-09 2017-04-26 奇酷互联网络科技（深圳）有限公司 一种日程创建的方法及装置
CN107483747A (zh) * 2017-09-26 2017-12-15 维沃移动通信有限公司 一种事件提醒方法及移动终端

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101339489A (zh) * 2008-08-14 2009-01-07 炬才微电子（深圳）有限公司 人机交互方法、装置和系统
CN102156577B (zh) * 2011-03-28 2013-05-29 安徽科大讯飞信息科技股份有限公司 实现连续手写识别输入的方法及系统
CN105988567B (zh) * 2015-02-12 2023-03-28 北京三星通信技术研究有限公司 手写信息的识别方法和装置
CN112115936A (zh) * 2020-10-10 2020-12-22 京东方科技集团股份有限公司 一种用于文本的识别方法、装置、存储介质以及电子设备
CN114690930A (zh) * 2021-04-22 2022-07-01 广州创知科技有限公司 一种书写笔迹处理方法、装置、交互平板及存储介质
CN113673432A (zh) * 2021-08-23 2021-11-19 京东方科技集团股份有限公司 手写识别方法、触摸显示设备、计算机设备及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101667074A (zh) * 2008-09-03 2010-03-10 联想(北京)有限公司 一种通过图形输入完成操作的装置及其方法
CN104360808A (zh) * 2014-12-04 2015-02-18 李方 一种利用符号手势指令进行文档编辑的方法及装置
CN106326343A (zh) * 2016-08-05 2017-01-11 重庆锐畅科技有限公司 一种基于音视频数据关联同步的电子白板数据共享系统
CN106600223A (zh) * 2016-12-09 2017-04-26 奇酷互联网络科技（深圳）有限公司 一种日程创建的方法及装置
CN107483747A (zh) * 2017-09-26 2017-12-15 维沃移动通信有限公司 一种事件提醒方法及移动终端

Also Published As

Publication number Publication date
WO2024022068A1 (zh) 2024-02-01
CN117813580A (zh) 2024-04-02
WO2024022068A9 (zh) 2024-02-29
CN116578224A (zh) 2023-08-11

Similar Documents

Publication Publication Date Title
US11922712B2 (en) Technologies for content analysis
US7966352B2 (en) Context harvesting from selected content
US10778928B2 (en) Device and method for inputting note information into image of photographed object
CN108780381B (zh) 利用手势进行笔记记录的装置和方法
US10664650B2 (en) Slide tagging and filtering
US6952803B1 (en) Method and system for transcribing and editing using a structured freeform editor
US7793230B2 (en) Search term location graph
US7962846B2 (en) Organization of annotated clipping views
US7966558B2 (en) Snipping tool
US9659279B2 (en) Method and system for enhanced inferred mode user interface operations
US7970763B2 (en) Searching and indexing of photos based on ink annotations
US9311338B2 (en) Method and apparatus for analyzing and associating behaviors to image content
KR20150132444A (ko) 클리핑 기능을 이용하는 협업 저작을 위한 방법, 장치 및 컴퓨터 판독가능 매체
KR20150087405A (ko) E-리더에 콘텐츠의 노트 기반 주석을 제공하는 방법
US20150033102A1 (en) Direct presentations from content collections
US20150121202A1 (en) System and method for transmitting mixed content type messages
US10296570B2 (en) Reflow narrative text objects in a document having text objects and graphical objects, wherein text object are classified as either narrative text object or annotative text object based on the distance from a left edge of a canvas of display
JP4081056B2 (ja) 情報処理装置、情報処理方法及びプログラム
US20150121179A1 (en) System and method for creating graphically rich messages incorporating shared docments
US20150121203A1 (en) System and method for generating uniform format pages for a system for composing messages
US8572480B1 (en) Editing the sequential flow of a page
US7519901B2 (en) Methods and systems for selecting objects by grouping annotations on the objects
US20160180167A1 (en) System and method for managing and reviewing document integration and updates
WO2024021078A1 (zh) 一种图文交互的方法及显示设备
JP6500341B2 (ja) 文書管理プログラム、文書管理方法及び文書管理システム

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 202280002471.9

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22952525

Country of ref document: EP

Kind code of ref document: A1