CN113434059A

CN113434059A - Written document processing method and device, electronic equipment and computer readable medium

Info

Publication number: CN113434059A
Application number: CN202011116909.5A
Authority: CN
Inventors: 杨宏伟; 李蕊
Original assignee: Individual
Current assignee: Individual
Priority date: 2020-10-19
Filing date: 2020-10-19
Publication date: 2021-09-24

Abstract

The disclosure relates to a written document processing method, a written document processing device, an electronic device and a computer readable medium. The method comprises the following steps: acquiring a picture and text control identifier in a preset area; determining a target interaction instruction and a target interaction object area based on the image-text control identifier; acquiring a document image of the written document in the target interaction object area; and processing the document image based on the target interaction instruction. The written document processing method, the written document processing device, the electronic equipment and the computer readable medium can realize the multimedia interaction function of the written document of a user under the condition of not depending on a display screen, and the user can call the multimedia resource of the computer to interact.

Description

Written document processing method and device, electronic equipment and computer readable medium

Technical Field

The present disclosure relates to the field of computer information processing, and in particular, to a method and an apparatus for processing a written document, an electronic device, and a computer-readable medium.

Background

The online operation modes such as multimedia, online lessons and remote offices are more and more widely used, and the traditional online operation mode extremely depends on the display screen, so that the influence on the vision of a user is large, and the traditional online operation mode is particularly not favorable for the eye health of children. Products such as a point-and-read pen and an intelligent pen using a dot matrix code technology can be operated on paper written documents to acquire multimedia interactive information, but the products need to be matched with specially printed paper books to use, and higher printing cost is needed.

Therefore, a new written document processing method, apparatus, electronic device, and computer readable medium are needed.

The above information disclosed in this background section is only for enhancement of understanding of the background of the disclosure and therefore it may contain information that does not constitute prior art that is already known to a person of ordinary skill in the art.

Disclosure of Invention

In view of the above, the present disclosure provides a method, an apparatus, an electronic device, and a computer-readable medium for processing a written document, which can implement a multimedia interaction function of the written document of a user without depending on a display screen, and the user can invoke a computer multimedia resource for interaction.

Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.

According to an aspect of the present disclosure, there is provided a written document processing method including: acquiring a picture and text control identifier in a preset area; determining a target interaction instruction and a target interaction object area based on the image-text control identifier; acquiring a document image of the written document in the target interaction object area; and processing the document image based on the target interaction instruction.

In an exemplary embodiment of the present disclosure, further comprising: generating an interactive instruction mapping rule based on one or more image-text control identifications and interactive instructions corresponding to the image-text control identifications; and/or generating an auxiliary control instruction mapping rule based on one or more image-text control identifications and corresponding auxiliary control instructions.

In an exemplary embodiment of the present disclosure, further comprising: acquiring a picture and text control identifier in a preset area; extracting a target auxiliary control instruction from an auxiliary control instruction mapping rule based on the image-text control identifier; and controlling the processing process or the multimedia equipment participating in the processing process based on the target auxiliary control instruction.

In an exemplary embodiment of the present disclosure, generating a target auxiliary control instruction based on the teletext control identifier and an auxiliary control instruction mapping rule further includes: and when the image-text control identification comprises parameters to be input, inputting parameter information according to a preset mode, wherein the parameters comprise at least one of voice data, character data, image data and gesture data.

In an exemplary embodiment of the present disclosure, acquiring the graphic control identifier in the preset area includes: acquiring an object image of a marked object placed in a preset area, wherein the preset area is positioned in an imaging range of an imaging device; and carrying out image recognition on the object image of the identification object to generate the image-text control identification.

In an exemplary embodiment of the present disclosure, image recognition of a recognition object image of the recognition object to generate the teletext control mark comprises: carrying out image recognition on the object image of the identification object to generate a recognition result; and eliminating invalid identification objects in the identification result to generate the image-text control identification.

In an exemplary embodiment of the present disclosure, determining a target interaction instruction and a target interaction object region based on the teletext control identifier includes: extracting a target interactive instruction from the interactive instruction mapping rule according to the image-text control identifier; and determining the target interaction object area based on the target interaction instruction.

In an exemplary embodiment of the present disclosure, processing the document image based on the target interactive instruction includes: performing image recognition on the document image based on the target interaction instruction to generate the document information; and carrying out data processing on the document information.

In an exemplary embodiment of the present disclosure, performing image recognition on the document image based on the target interaction instruction to generate the document information includes: determining a target area based on the target interaction instruction; and performing image recognition on an image in a target area in the document image to generate the document information.

In an exemplary embodiment of the present disclosure, processing the document image based on the target interactive instruction further includes: and when the image-text control identifier needs to input parameters, inputting parameter information according to a preset mode, wherein the parameters comprise at least one of voice data, character data, image data and gesture data.

In an exemplary embodiment of the present disclosure, further comprising: generating multimedia information based on the processing result; and performing multimedia operation based on the multimedia information.

In an exemplary embodiment of the present disclosure, performing a multimedia operation based on the multimedia information includes: determining a display area based on the image-text control identification; and projecting the processing result in the display area.

In an exemplary embodiment of the present disclosure, further comprising: and correcting the document image and/or the projection image projected in the display area.

In an exemplary embodiment of the present disclosure, rectifying the document image and/or the projection image projected on the display area includes: acquiring a first relative position of a known point on a written document corresponding to the document image; acquiring a second relative position of the known point in the document image; determining an aberration correction mapping between the written document and the document image based on the first relative position and the second relative position; rectifying the document image based on the distortion rectification mapping.

In an exemplary embodiment of the present disclosure, rectifying the document image and/or the projection image projected on the display area further includes: superposing the projection image on the corrected document image to generate a superposed image; determining a deviation adjustment mapping of the projected image according to the superimposed image; rectifying the projected image based on the deviation adjustment map.

According to an aspect of the present disclosure, there is provided a written document processing method including: acquiring a picture-text control identifier from the picture-text control identifier area; acquiring a document image of the written document from the interactive object area; determining a target interaction instruction based on the image-text control identifier; and processing the document image based on the target interaction instruction.

In an exemplary embodiment of the present disclosure, further comprising: generating at least one interactive instruction mapping rule based on one or more image-text control identifications, at least one image-text control identification area and corresponding interactive instructions; and/or generating at least one auxiliary control instruction mapping rule based on one or more teletext control identifications, at least one teletext control auxiliary region and its corresponding auxiliary control instruction.

In an exemplary embodiment of the present disclosure, further comprising: acquiring a picture-text control identifier from the picture-text control auxiliary area; extracting a target auxiliary control instruction from the at least one auxiliary control instruction mapping rule based on the image-text control identifier and the number of the image-text control auxiliary area corresponding to the image-text control identifier; and controlling the processing process or the multimedia equipment participating in the processing process based on the target auxiliary control instruction.

In an exemplary embodiment of the present disclosure, extracting a target auxiliary control instruction in the at least one auxiliary control instruction mapping rule based on the teletext control identifier and the number of the teletext control auxiliary area corresponding to the teletext control identifier further includes: and when the image-text control identification comprises parameters to be input, inputting parameter information according to a preset mode, wherein the parameters comprise at least one of voice data, character data, image data and gesture data.

In an exemplary embodiment of the present disclosure, further comprising: adjusting the area range of the image-text control identification area and/or the interactive object area and/or the image-text control auxiliary area; the image-text control identification area, the interactive object area and the image-text control auxiliary area are positioned in the imaging range of an imaging device.

In an exemplary embodiment of the present disclosure, adjusting the area range of the teletext control identification area and/or the interactive object area and/or the teletext control assistance area includes: acquiring an object image of a marking object placed in an imaging range and used for marking the area range; carrying out image recognition on the object image to generate a graphic control identification for marking the area range; determining a target area range according to the image-text control identifier for marking the area range; and adjusting the area range of the interactive object area and/or the image-text control identification area and/or the image-text control auxiliary area based on the target area range.

In an exemplary embodiment of the present disclosure, acquiring the teletext control identifier from the teletext control identifier region includes: the number of the graphic control identification areas is set and determined by the system; and acquiring the image-text control identification from all the image-text control identification areas.

In an exemplary embodiment of the present disclosure, acquiring the teletext control identifier from all the teletext control identifier areas includes: acquiring an object image of a marking object placed in all the image-text control marking areas; and carrying out image recognition on the object image of the identification object to generate the image-text control identification.

In an exemplary embodiment of the present disclosure, determining a target interactive instruction based on the teletext control identifier includes: and determining a target interactive instruction based on the image-text control identification and the number of the image-text control identification area corresponding to the image-text control identification.

In an exemplary embodiment of the present disclosure, determining a target interactive instruction based on the number of the teletext control identifier and the corresponding teletext control identifier region includes: determining a target interactive instruction mapping rule from the at least one interactive instruction mapping rule based on the number of the image-text control identification area; and extracting a target interactive instruction from the target interactive instruction mapping rule according to the image-text control identifier.

According to an aspect of the present disclosure, there is provided a written document processing system, including: the identification object is used for displaying the image-text control identification; the imaging device is used for acquiring the image-text control identification in a preset area and/or acquiring a document image of a written document in a target interaction object area; the processing device is used for determining a target interaction instruction and the target interaction object area based on the image-text control identification; processing the document image based on the target interaction instruction; and/or determining a target auxiliary control instruction based on the image-text control identifier; and controlling the processing process or the multimedia equipment participating in the processing process based on the target auxiliary control instruction.

In an exemplary embodiment of the present disclosure, further comprising: multimedia means for generating multimedia information based on the processing result; and performing multimedia operation based on the multimedia information.

According to an aspect of the present disclosure, there is provided a written document processing system, including: the identification object is used for displaying the image-text control identification; the imaging device is used for acquiring the image-text control identification from the image-text control identification area, and/or acquiring the image-text control identification from the image-text control auxiliary area, and/or acquiring a document image of a written document from the interactive object area; the processing device is used for determining a target interaction instruction based on the image-text control identifier; processing the document image based on the target interaction instruction; and/or determining a target auxiliary control instruction based on the image-text control identification, and controlling a processing process or multimedia equipment participating in the processing process based on the target auxiliary control instruction.

According to an aspect of the present disclosure, an electronic device is provided, the electronic device including: one or more processors; storage means for storing one or more programs; when executed by one or more processors, cause the one or more processors to implement a method as above.

According to an aspect of the disclosure, a computer-readable medium is proposed, on which a computer program is stored, which program, when being executed by a processor, carries out the method as above.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings. The drawings described below are merely some embodiments of the present disclosure, and other drawings may be derived from those drawings by those of ordinary skill in the art without inventive effort.

FIG. 1 is a system block diagram illustrating a method and apparatus for processing a written document according to an example embodiment.

FIG. 2 is a flow diagram illustrating a method of processing a written document according to an example embodiment.

FIG. 3 is a diagram illustrating a method of processing a written document, according to an example embodiment.

FIG. 4 is a flow chart illustrating a method of written document processing according to another exemplary embodiment.

FIG. 5 is a diagram illustrating a method of processing a written document, according to an example embodiment.

FIG. 6 is a flow chart illustrating a method of written document processing according to another exemplary embodiment.

FIG. 7 is a flowchart illustrating a method of processing a written document according to another exemplary embodiment.

FIG. 8 is a block diagram illustrating a written document processing device, according to an example embodiment.

FIG. 9 is a block diagram illustrating a written document processing device, according to another example embodiment.

FIG. 10 is a block diagram illustrating an electronic device in accordance with an example embodiment.

FIG. 11 is a block diagram illustrating a computer-readable medium in accordance with an example embodiment.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals denote the same or similar parts in the drawings, and thus, a repetitive description thereof will be omitted.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the subject matter of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known methods, devices, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the disclosure.

The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.

The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.

It will be understood that, although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one element from another. Thus, a first component discussed below may be termed a second component without departing from the teachings of the disclosed concept. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

It is to be understood by those skilled in the art that the drawings are merely schematic representations of exemplary embodiments, and that the blocks or processes shown in the drawings are not necessarily required to practice the present disclosure and are, therefore, not intended to limit the scope of the present disclosure.

FIG. 1 is a system block diagram illustrating a method, apparatus, electronic device, and computer-readable medium for written document processing according to an example embodiment.

As shown in fig. 1, the system architecture 10 may include an identification object 101, an imaging device 102, a network 103, and a processing device 104. The system architecture 10 may also include several multimedia devices 105. A medium over which the network 103 provides a communication link between the imaging apparatus 102 and the processing apparatus 104; the network 103 is also used to provide a medium for communication links between the multimedia device 105 and the processing device 104. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The identification object 101 is used for displaying the graphic control identification, the identification object 101 can be one-dimensional and/or two-dimensional and/or three-dimensional, and one or more of points, lines, shading, characters, symbols, numbers, two-dimensional codes, bar codes, pictures, graphs, charts, two-dimensional or three-dimensional identifiers or other graphic information which can be visually identified through a computer can be selected to form the identification object 101;

furthermore, one and/or a plurality of identification objects 101 are combined according to a certain rule in time and/or space to form a new identification object 101, and when the identification objects 101 are combined, only the original identification object 101 may be used, or the identification object 101 used for representing the combination relationship between the identification objects 101 may be added to the original identification object 101.

Any one of the identification objects 101 is in a different position, and/or a combination of a plurality of the identification objects 101, and/or a different position and/or relative position between a plurality of the identification objects 101, and/or a relative position of one or more of the identification objects 101 with the imaging device 102, and/or a different sequence of appearance and/or disappearance of the plurality of the identification objects 101, and/or a different motion trajectory and/or velocity and/or acceleration and/or angular velocity and/or angular acceleration and/or relative position change and/or relative velocity and/or relative acceleration and/or relative angular velocity and/or relative angular acceleration and/or vibration frequency and/or vibration amplitude of one or more of the identification objects 101, and/or a discoloration and/or deformation of one or more of the identification objects 101, different interaction commands or auxiliary control commands can be corresponded.

The display graphic control mark can be displayed in a preset area by one or more of writing, drawing, printing, pasting, displaying, placing, splicing or other modes capable of displaying the mark object 101, and can be arranged on the written document or at the proper position of the periphery of the written document.

The mapping rule or the storage address of the identification object 101 and the interactive instruction or the auxiliary control instruction, the program or the storage address of the execution identification task of the graphic control identification, and the program or the storage address of the conversion interactive instruction can be stored in a memory of a client or a server or a cloud, and can also be stored in a visually identified information storage mode (such as a two-dimensional code) on the graphic control identification. When the image-text control mark is stored in a visual identification information storage mode (such as a two-dimensional code), the image-text control mark is identified, and the content stored in the visual identification information storage mode on the image-text control mark can be obtained.

The imaging device 102 is configured to obtain the image-text control identifier in a preset area, and/or obtain a document image of a written document in a target interaction object area; the image forming apparatus 102 interacts with the processing apparatus 104 through the network 103 to receive or transmit messages and the like. The imaging device 102 may be a smart camera, a monitoring apparatus, etc. that includes camera functionality.

The processing device 104 is configured to determine a target interaction instruction and the target interaction object area based on the teletext control identification; processing the document image based on the target interaction instruction; and/or determining a target auxiliary control instruction based on the image-text control identifier; and controlling the processing process or the multimedia equipment participating in the processing process based on the target auxiliary control instruction. Various data transmission type applications, such as image recognition type applications, education platform type applications, multimedia type applications, etc., may be installed on the processing device 104. The processing device 104 may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablets, laptop portable computers, desktop computers, and the like.

It should be noted that the processing device 104 may be a physical device, a remote server, or a cloud virtual device, which is not limited in this disclosure.

The multimedia device 105 is used for generating multimedia information based on the processing result; and performing multimedia operation based on the multimedia information. The multimedia device 105 may include various types of settings, which may include, for example, a smart light, a smart sound, a multimedia player, a projector, and so forth.

In one particular use scenario, the written document and the identifying object 101 are placed on a plane, which may be a desk or other platform, which serves as a predetermined area. The imaging device 102 may, for example, obtain a graphic control identifier within a preset area; the processing device 104 may determine a target interaction instruction and a target interaction object area, for example, based on the teletext control identification; the imaging device 102 may, for example, acquire a document image of a written document within the target interaction object region; the processing device 104 may process the document image, for example, based on the target interaction instruction.

Before the above processing, the processing device 104 may also generate an interactive instruction mapping rule based on one or more teletext control identifications and their corresponding interactive instructions, for example; and/or generating an auxiliary control instruction mapping rule based on one or more image-text control identifications and corresponding auxiliary control instructions.

In the above processing, it is also possible that, for example, the identification object 101 is placed on a desktop, and the imaging apparatus 102 may, for example, acquire the graphic control identification within a preset area; the processing device 104 may extract a target auxiliary control instruction in an auxiliary control instruction mapping rule, e.g. based on the teletext control identification; the processing device 104 may control the process or the multimedia devices 105 participating in the process, for example, based on the target secondary control instructions.

The technical content of the usage scenario will be described in detail in the embodiment corresponding to fig. 2 and fig. 3.

In another specific use scenario, the written document and the identification object 101 are placed on a plane, which may be a desk or other platform, which serves as a preset area. The imaging device 102 may obtain the teletext control identification, for example, from a teletext control identification area; the imaging device 102 may determine a target interaction instruction, for example, based on the teletext control identification; the imaging device 102 may acquire a document image of a written document, for example, from the interaction object area; the processing device 104 may process the document image, for example, based on the target interaction instruction.

Prior to the above-mentioned processing, the processing device 104 may also generate at least one interactive instruction mapping rule, for example, based on one or more teletext control identifications, at least one teletext control identification region and its corresponding interactive instruction; and/or generating at least one auxiliary control instruction mapping rule based on one or more teletext control identifications, at least one teletext control auxiliary region and its corresponding auxiliary control instruction.

In the above process, it is also possible that, for example, the identification object 101 is placed on a desktop, and the imaging apparatus 102 acquires the teletext control identification from the teletext control auxiliary area, for example; the processing means 104 may extract a target auxiliary control instruction in the at least one auxiliary control instruction mapping rule, e.g. based on the teletext control identification and the number of its corresponding teletext control auxiliary area; the process or the multimedia devices 105 participating in the process are controlled based on the target secondary control instruction.

The technical contents of the usage scenario will be described in detail in the embodiments corresponding to fig. 4 and 5.

FIG. 2 is a flow diagram illustrating a method of processing a written document according to an example embodiment. The written document processing method 20 is a written document processing method performed for the scenario described in fig. 3. In the corresponding schematic diagram of fig. 3, the written document is placed on a flat desk, and the user previously places a logo object (graphic control logo) to be processed at any position on the desk. The written document processing method 20 includes at least steps S202 to S208.

As shown in fig. 2, in S202, a graphic control identifier in a preset area is obtained. An object image of a marker object placed in a preset area, which is located within an imaging range of an imaging device, may be acquired, for example; wherein, performing image recognition on the identification object image of the identification object to generate the image-text control identification comprises: carrying out image recognition on the object image of the identification object to generate a recognition result; and eliminating invalid identification objects in the identification result to generate the image-text control identification.

In one embodiment, before obtaining the graphic control identifier in the preset area, the method further includes: generating an interactive instruction mapping rule based on one or more image-text control identifications and interactive instructions corresponding to the image-text control identifications; and/or generating an auxiliary control instruction mapping rule based on one or more image-text control identifications and corresponding auxiliary control instructions.

The interactive instruction mapping rule and the auxiliary control instruction mapping rule specifically comprise; setting one or more graphic control marks; setting an identification method or an identification program of the image-text control identification, and identifying according to all or part of the components of the image-text control identification when identifying the image-text control identification; setting an instruction mapping rule for converting the image-text control identifier into a corresponding operation instruction; the method can also comprise the following steps: the instruction mapping rule supplementing or modifying method for setting the image-text control identification supplements or modifies the operation instruction corresponding to a certain image-text control identification by recognizing the voice information or the image-text information.

The content specifically included in the interactive command and the auxiliary control command will be described in detail later.

In S204, a target interactive instruction and a target interactive object area are determined based on the teletext control identifier. Target interactive instructions can be extracted from the interactive instruction mapping rules, for example, according to the graphic control identification; and determining the target interaction object area based on the target interaction instruction.

The interactive object area refers to a specific area in the written document image, and the image in the specific area or the elements which can be visually recognized in the specific area participate in specific interactive operation. The interactive object area can be all of the images taken by the imaging device or one or more local areas thereof. When the interactive object area is calibrated, the description can be described according to the fixed position of the interactive object area in the image shot by the imaging device, and also can be described according to the relative position of the interactive object area in the image shot by the imaging device and one or more graphic control marks.

Furthermore, the interactive object area refers to a range that can be captured by the imaging device, and can perform interactive operation on the written document image in the range, the interactive object area may be one or multiple, the interactive object area may be all or part of the written document, multiple interactive object areas may be in the same written document or in multiple written documents, respectively, and the interactive object area may be all or part of the image that can be captured by the imaging device.

In S206, a document image of the written document in the target interaction object region is acquired. The obtained written document image can be a written document and can also be a dynamic or static image of a working process on the written document.

In S208, the document image is processed based on the target interactive instruction. Can include the following steps: performing image recognition on the document image based on the target interaction instruction to generate the document information; and carrying out data processing on the document information.

Wherein performing image recognition on the document image based on the target interaction instruction to generate the document information comprises: determining a target area based on the target interaction instruction; and performing image recognition on an image in a target area in the document image to generate the document information.

Wherein, processing the document image based on the target interactive instruction may further be: and when the image-text control identifier needs to input parameters, inputting parameter information according to a preset mode, wherein the parameters comprise at least one of voice data, character data, image data and gesture data.

According to the written document processing method, the image-text control identification in the preset area is obtained; determining a target interaction instruction and a target interaction object area based on the image-text control identifier; acquiring a document image of the written document in the target interaction object area; the method for processing the document image based on the target interaction instruction can realize the multimedia interaction function of the written document of the user under the condition of not depending on a display screen, and the user can also call computer multimedia resources to carry out interaction.

It should be clearly understood that this disclosure describes how to make and use particular examples, but the principles of this disclosure are not limited to any details of these examples. Rather, these principles can be applied to many other embodiments based on the teachings of the present disclosure.

FIG. 4 is a flow diagram illustrating a method of processing a written document, according to an example embodiment. The written document processing method 40 is a written document processing method performed for the scenario as described in fig. 5. In the corresponding schematic diagram of fig. 5, the written document is placed on a flat desk, and the user previously places the identification object (graphic control identification) to be processed in one or more preset areas on the desk. The written document processing method 40 includes at least steps S402 to S408.

As shown in fig. 4, in S402, the teletext control indicator is obtained from the teletext control indicator area. The method comprises the following steps: the number of the graphic control identification areas is set and determined by the system; and acquiring the image-text control identification from all the image-text control identification areas.

One or more graphic control identification areas are set in the shooting range of the imaging device, and the graphic control identification areas can be all images shot by the imaging device or one or more areas in the images. And after the image-text control identification area is set, identifying the image-text control identification in the image-text control identification area according to a mapping rule corresponding to the image-text control identification area. The recognizable teletext control marks may be the same or different in different teletext control mark areas. In different graphic control identification areas, the interactive instructions corresponding to the same graphic control identification can be the same or different.

Wherein, the picture and text control identification is obtained from all the picture and text control identification areas, and the method comprises the following steps: acquiring an object image of a marking object placed in all the image-text control marking areas; and carrying out image recognition on the object image of the identification object to generate the image-text control identification.

In one embodiment, before the obtaining of the teletext control identifier from the teletext control identifier region, the method further includes: generating at least one interactive instruction mapping rule based on one or more image-text control identifications, at least one image-text control identification area and corresponding interactive instructions; and/or generating at least one auxiliary control instruction mapping rule based on one or more teletext control identifications, at least one teletext control auxiliary region and its corresponding auxiliary control instruction.

The interactive instruction mapping rule and the auxiliary control instruction mapping rule specifically comprise; setting one or more graphic control marks; setting an identification method or an identification program of the image-text control identification, and identifying according to all or part of the components of the image-text control identification when identifying the image-text control identification; setting an instruction mapping rule for converting the image-text control identifier into a corresponding operation instruction; the method can also comprise the following steps: the instruction mapping rule supplementing or modifying method for setting the image-text control identification supplements or modifies the operation instruction corresponding to a certain image-text control identification by recognizing the voice information or the image-text information. When the existing image-text data is supplemented or modified according to the information of the image-text control identification, one or more image-text control auxiliary areas can be set in the shooting range of the imaging device, and the image-text information can be added or changed in the image-text control auxiliary areas.

In S404, a document image of the written document is acquired by the interaction object area.

In S406, a target interactive instruction is determined based on the teletext control identification. The method comprises the following steps: and determining a target interactive instruction based on the image-text control identification and the number of the image-text control identification area corresponding to the image-text control identification.

Determining a target interactive instruction based on the graphics context control identifier and the number of the graphics context control identifier area corresponding to the graphics context control identifier, wherein the determining of the target interactive instruction comprises the following steps: determining a target interactive instruction mapping rule from the at least one interactive instruction mapping rule based on the number of the image-text control identification area; and extracting a target interactive instruction from the target interactive instruction mapping rule according to the image-text control identifier.

In S408, the document image is processed based on the target interaction instruction. The method comprises the following steps: performing image recognition on the document image based on the target interaction instruction to generate the document information; and carrying out data processing on the document information.

In one embodiment, further comprising: adjusting the area range of the image-text control identification area and/or the interactive object area and/or the image-text control auxiliary area; the image-text control identification area, the interactive object area and the image-text control auxiliary area are positioned in the imaging range of an imaging device.

Adjusting the area range of the image-text control identification area and/or the interactive object area and/or the image-text control auxiliary area, comprising: acquiring an object image of a marking object placed in an imaging range and used for marking the area range; carrying out image recognition on the object image to generate a graphic control identification for marking the area range; determining a target area range according to the image-text control identifier for marking the area range; and adjusting the area range of the interactive object area and/or the image-text control identification area and/or the image-text control auxiliary area based on the target area range.

The image-text control identifier for performing the interactive operation which can be realized by a computer or other intelligent equipment on all or part of the written document image and the interactive instruction corresponding to the image-text control identifier can comprise:

marking a graph-text control identifier of the written document interaction range and an interaction instruction corresponding to the graph-text control identifier; marking the graphic control marks of one or more interactive object areas in the interactive range of the written document and the interactive instructions corresponding to the graphic control marks; selecting the image-text control identification of one or more interactive object areas in the interactive range of the written document and the interactive instruction corresponding to the image-text control identification for certain interactive operation; the picture and text control marks mark the recognition range of the picture and text control marks and the interactive instructions corresponding to the picture and text control marks; the image-text control identifier is used for combining the image-text control identifiers into a new image-text control identifier and an interactive instruction corresponding to the image-text control identifier; the picture and text control marks and the corresponding interactive instructions are used for explaining the identification method of one or more picture and text control marks or the interactive instruction program mapping rule corresponding to the picture and text control marks; identifying the graphic control identifier of the program for interactive operation of the graphic control identifier of the written document and an interactive instruction corresponding to the graphic control identifier; converting the identified image-text control identification into an image-text control identification of a program of the interactive instruction and an interactive instruction corresponding to the image-text control identification; executing the graphic control identification of the interactive instruction program and the interactive instruction corresponding to the graphic control identification; providing a graphic control identification of relevant parameters or data of an interactive instruction of interactive operation and an interactive instruction corresponding to the graphic control identification, wherein the parameters or data required by the interactive operation can come from one or more data sources; providing a graphic control identification of an address of an executive program and/or a data source of interactive operation and an interactive instruction corresponding to the graphic control identification; setting an image-text control identifier of an executive program operation authority and/or a data source operation authority of interactive operation and an interactive instruction corresponding to the image-text control identifier; the interactive system comprises a graphic control identifier for annotating or explaining the operation process on the interactive written document or the written document and an interactive instruction corresponding to the graphic control identifier; a graphic control mark for describing or assigning the attribute of the mark object and an interactive instruction corresponding to the graphic control mark, wherein the attribute of the mark object comprises the size information of the writing medium and/or all or part of the visually recognizable element printed or printed on the writing medium and/or the position of the visually recognizable element on the medium and/or the relative position between the visually recognizable elements; the picture and text control mark for accurately positioning one or more points on or around the written document refers to one or a group of picture and text control marks which are arranged for accurately calibrating the relative position between the internal or peripheral points of the written document, the relative position between one or more points in or around the written document and an imaging device, and the relative position between one or more points in or around the written document and a projection device; carrying out corrected or deformed image-text control identification on the obtained written document image and an interactive instruction corresponding to the image-text control identification; marking the graphic control identification of the projection feedback display area and the interactive instruction corresponding to the graphic control identification; the image-text control identification for correcting or deforming or moving or reversing or rotating the projection feedback image and an interactive instruction corresponding to the image-text control identification; the image-text control identification and the corresponding interactive instruction are used for controlling equipment participating in interactive operation or equipment influencing interactive surrounding environment; the picture and text control mark adjusts the volume, illumination or projection brightness and the corresponding interactive instruction.

The graphic control identifier and the corresponding interactive instruction are only used for describing the interactive operation more conveniently, in an actual situation, according to the complexity of the interactive operation, the various interactive operations can be independent of each other, can be combined randomly, can exist totally, can only have any one or more types of interactive operations, and can be a one-to-one and/or one-to-many and/or many-to-one correspondence relationship between the interactive operation and the graphic control identifier.

FIG. 6 is a flow chart illustrating a method of written document processing according to another exemplary embodiment. The flow shown in fig. 6 describes the detailed processing of the teletext control identification.

As shown in fig. 6, in S602, a graphic control identifier in a preset area is obtained. It is worth mentioning that in the usage scenario of fig. 4, the imaging device obtains the teletext control indicator from the teletext control assistance area.

In S604, a target auxiliary control command is extracted from the auxiliary control command mapping rule based on the teletext control identification. It is worth mentioning that, in the usage scenario of fig. 4, the processing device extracts a target auxiliary control command in the at least one auxiliary control command mapping rule based on the teletext control identifier and the number of the teletext control auxiliary area corresponding thereto.

In S606, the processing procedure or the multimedia device participating in the processing procedure is controlled based on the target auxiliary control instruction.

When the image-text control identifier needs to input parameters, parameter information can be input according to a preset mode, wherein the parameters comprise at least one of voice data, character data, image data and gesture data.

The identified interactive instruction can be determined according to the preset image-text control identifier and the auxiliary control instruction. In the process of identifying the image-text control identification, whether the image-text control identification needs to supplement or change an interactive instruction corresponding to a certain image-text control identification by identifying voice information or image-text information can be judged, and if so, the voice information or the image-text information is collected and identified to supplement or modify the interactive instruction.

And if the interactive object area exists, obtaining an image of the interactive object area or a visually recognizable element in the interactive object area, and using the image or the visually recognizable element as an interactive object of an operation instruction and/or an instruction set.

FIG. 7 is a flowchart illustrating a method of processing a written document according to another exemplary embodiment. The flow shown in fig. 7 describes the subsequent processing steps of document processing.

As shown in fig. 7, in S702, based on the processing result, multimedia information is generated.

In S704, a multimedia operation is performed based on the multimedia information.

In S706, when the multimedia operation is a projection operation, a display area is determined based on the graphic control identifier.

In S708, the document image and/or the projection image projected on the display area are corrected.

In one embodiment, a first relative position of a known point on a written document corresponding to the document image may be obtained; acquiring a second relative position of the known point in the document image; determining an aberration correction mapping between the written document and the document image based on the first relative position and the second relative position; rectifying the document image based on the distortion rectification mapping. Specifically, for example, the relative position between multiple points inside or around the written document and/or the relative position between one or more points inside or around the written document and the imaging device can be obtained; acquiring images of one or more points inside or around the written document; and obtaining distortion correction mapping of one or more points inside or around the written document according to the images and the relative positions. The image is distortion-corrected so that the relative position between the image of one or more points inside or around the written document and the multiple points inside or around the written document and/or the relative position between one or more points inside or around the written document and the imaging device matches.

In one embodiment, the subsequent processing step of document processing further comprises: superposing the projection image on the corrected document image to generate a superposed image; determining a deviation adjustment mapping of the projected image according to the superimposed image; rectifying the projected image based on the deviation adjustment map. An expected relative position of the feedback image or calibration image and a point or points within or surrounding the written document at a known, precisely located position on the overlay image is obtained. Specifically, for example, a feedback image or a calibration image is projected, and a superimposed image of the projection of the feedback image or the calibration image and the written document image is acquired; determining a deviation adjustment mapping of the feedback image or the calibration image according to the actual relative position of the projection of the feedback image or the calibration image and the superimposed image of the image of one or more points inside the known accurately positioned written document or at the periphery of the written document; adjusting the feedback image according to the deviation adjustment map such that the projection of the feedback image is at a desired relative position on the overlay image to one or more points within or surrounding the known precisely positioned written document.

Furthermore, referring to the positioning result of the graphic control mark for accurately positioning one or more points on or around the written document, the positioning is performed by using the auxiliary bar or the auxiliary graph on the display medium of the written document, which specifically comprises: obtaining the accurate positioning result of one or more points on the written document or on the periphery of the written document; obtaining an auxiliary bar or an auxiliary graphic on the written document display medium; the relative position of one or more points on the auxiliary bar or auxiliary graphic on the display medium of the written document and one or more points on the written document or on the periphery of the written document with known accurate positioning results is obtained.

In S710, the processing result is projected on the display area.

Those skilled in the art will appreciate that all or part of the steps implementing the above embodiments are implemented as computer programs executed by a CPU. When executed by the CPU, performs the functions defined by the above-described methods provided by the present disclosure. The program may be stored in a computer readable storage medium, which may be a read-only memory, a magnetic or optical disk, or the like.

Furthermore, it should be noted that the above-mentioned figures are only schematic illustrations of the processes involved in the methods according to exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.

The following are embodiments of the disclosed apparatus that may be used to perform embodiments of the disclosed methods. For details not disclosed in the embodiments of the apparatus of the present disclosure, refer to the embodiments of the method of the present disclosure.

FIG. 8 is a block diagram illustrating a written document processing device, according to an example embodiment. As shown in fig. 8, the written document processing device 80 can be applied to the scenario shown in fig. 2, and includes: the identification object 802, the imaging device 804, the processing device 806, and the written document processing device 80 may further include: a multimedia device 808.

The identification object 802 is used for displaying the graphic control identification;

the imaging device 804 is used for acquiring the image-text control identifier in a preset area and/or acquiring a document image of a written document in a target interaction object area;

the processing device 806 is configured to determine a target interaction instruction and the target interaction object area based on the teletext control identification; processing the document image based on the target interaction instruction; and/or determining a target auxiliary control instruction based on the image-text control identifier; controlling a processing process or multimedia equipment participating in the processing process based on the target auxiliary control instruction;

the multimedia device 808 is configured to generate multimedia information based on the processing result; and performing multimedia operation based on the multimedia information.

FIG. 9 is a block diagram illustrating a written document processing device, according to another example embodiment. As shown in fig. 9, the written document processing device 90 can be applied to the scenario shown in fig. 4, and includes: the marking object 902, the imaging device 904, the processing device 906, and the written document processing device 90 may further include: a multimedia device 908.

The identification object 902 is used for displaying the graphic control identification;

the imaging device 904 is used for acquiring the image-text control identification from the image-text control identification area, and/or acquiring the image-text control identification from the image-text control auxiliary area, and/or acquiring a document image of a written document from the interactive object area;

the processing device 906 is configured to determine a target interactive instruction based on the teletext control identifier and the number of the teletext control identifier region corresponding to the teletext control identifier; processing the document image based on the target interaction instruction; and/or determining a target auxiliary control instruction based on the image-text control identification and the number of the image-text control auxiliary area corresponding to the image-text control identification, and controlling a processing process or multimedia equipment participating in the processing process based on the target auxiliary control instruction;

the multimedia device 908 is used for generating multimedia information based on the processing result; and performing multimedia operation based on the multimedia information.

According to the written document processing device, the image-text control identification in the preset area is obtained; determining a target interaction instruction and a target interaction object area based on the image-text control identifier; acquiring a document image of the written document in the target interaction object area; the method for processing the document image based on the target interaction instruction can realize the multimedia interaction function of the written document of the user under the condition of not depending on a display screen, and the user can also call computer multimedia resources to carry out interaction.

An electronic device 1000 according to this embodiment of the disclosure is described below with reference to fig. 10. The electronic device 1000 shown in fig. 10 is only an example and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 10, the electronic device 1000 is embodied in the form of a general purpose computing device. The components of the electronic device 1000 may include, but are not limited to: at least one processing unit 1010, at least one memory unit 1020, a bus 1030 that couples various system components including the memory unit 1020 and the processing unit 1010, a display unit 1040, and the like.

Wherein the storage unit stores program code executable by the processing unit 1010 to cause the processing unit 1010 to perform steps according to various exemplary embodiments of the present disclosure described in this specification. For example, the processing unit 1010 may perform the steps shown in fig. 2, 4, 6, and 7.

The memory unit 1020 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)10201 and/or a cache memory unit 10202, and may further include a read only memory unit (ROM) 10203.

The memory unit 1020 may also include a program/utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.

Bus 1030 may be any one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, and a local bus using any of a variety of bus architectures.

The electronic device 1000 may also communicate with one or more external devices 1000' (e.g., keyboard, pointing device, bluetooth device, etc.) such that a user can communicate with devices with which the electronic device 1000 interacts, and/or any devices (e.g., router, modem, etc.) with which the electronic device 1000 can communicate with one or more other computing devices. Such communication may occur through input/output (I/O) interfaces 1050. Also, the electronic device 1000 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the internet) via the network adapter 1060. A network adapter 1060 may communicate with other modules of the electronic device 1000 via the bus 1030. It should be appreciated that although not shown, other hardware and/or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RA id systems, tape drives, and data backup storage systems, among others.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, as shown in fig. 11, the technical solution according to the embodiment of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present disclosure.

The software product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

The computer readable medium carries one or more programs which, when executed by a device, cause the computer readable medium to perform the functions of: acquiring a picture and text control identifier in a preset area; determining a target interaction instruction and a target interaction object area based on the image-text control identifier; acquiring a document image of the written document in the target interaction object area; and processing the document image based on the target interaction instruction. The computer readable medium may also implement the following functions: acquiring a picture-text control identifier from the picture-text control identifier area; acquiring a document image of the written document from the interactive object area; determining a target interaction instruction based on the image-text control identifier; and processing the document image based on the target interaction instruction.

Those skilled in the art will appreciate that the modules described above may be distributed in the apparatus according to the description of the embodiments, or may be modified accordingly in one or more apparatuses unique from the embodiments. The modules of the above embodiments may be combined into one module, or further split into multiple sub-modules.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

Exemplary embodiments of the present disclosure are specifically illustrated and described above. It is to be understood that the present disclosure is not limited to the precise arrangements, instrumentalities, or instrumentalities described herein; on the contrary, the disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method of processing a written document, comprising:

acquiring a picture and text control identifier in a preset area;

determining a target interaction instruction and a target interaction object area based on the image-text control identifier;

acquiring a document image of the written document in the target interaction object area;

and processing the document image based on the target interaction instruction.

2. The method of claim 1, further comprising:

generating an interactive instruction mapping rule based on one or more image-text control identifications and interactive instructions corresponding to the image-text control identifications; and/or

And generating an auxiliary control instruction mapping rule based on one or more image-text control identifications and the corresponding auxiliary control instructions.

3. The method of claim 2, further comprising:

acquiring a picture and text control identifier in a preset area;

extracting a target auxiliary control instruction from an auxiliary control instruction mapping rule based on the image-text control identifier;

and controlling the processing process or the multimedia equipment participating in the processing process based on the target auxiliary control instruction.

4. The method of claim 1, wherein obtaining the teletext control identifier in the predetermined area comprises:

acquiring an object image of a marked object placed in a preset area, wherein the preset area is positioned in an imaging range of an imaging device;

and carrying out image recognition on the object image of the identification object to generate the image-text control identification.

5. The method of claim 2, wherein determining a target interaction instruction and a target interaction object area based on the teletext control identification comprises:

extracting a target interactive instruction from the interactive instruction mapping rule according to the image-text control identifier;

and determining the target interaction object area based on the target interaction instruction.

6. The method of claim 1, wherein processing the document image based on the target interaction instruction comprises:

performing image recognition on the document image based on the target interaction instruction to generate the document information;

and carrying out data processing on the document information.

7. The method of claim 6, wherein processing the document image based on the target interaction instruction further comprises:

and when the image-text control identifier needs to input parameters, inputting parameter information according to a preset mode, wherein the parameters comprise at least one of voice data, character data, image data and gesture data.

8. A document information processing system, characterized by comprising:

the identification object is used for displaying the image-text control identification;

the imaging device is used for acquiring the image-text control identification in a preset area and/or acquiring a document image of a written document in a target interaction object area;

the processing device is used for determining a target interaction instruction and the target interaction object area based on the image-text control identification; processing the document image based on the target interaction instruction; and/or determining a target auxiliary control instruction based on the image-text control identifier; and controlling the processing process or the multimedia equipment participating in the processing process based on the target auxiliary control instruction.

9. The system of claim 8, further comprising:

multimedia means for generating multimedia information based on the processing result; and performing multimedia operation based on the multimedia information.

10. A written document processing system, comprising:

the imaging device is used for acquiring the image-text control identification from the image-text control identification area, and/or acquiring the image-text control identification from the image-text control auxiliary area, and/or acquiring a document image of a written document from the interactive object area;

the processing device is used for determining a target interaction instruction based on the image-text control identifier; processing the document image based on the target interaction instruction; and/or determining a target auxiliary control instruction based on the image-text control identification, and controlling a processing process or multimedia equipment participating in the processing process based on the target auxiliary control instruction.

11. The system of claim 10, further comprising:

12. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs;

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.

13. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-7.