CN112506398A

CN112506398A - Image-text display method, device and computer readable medium for same

Info

Publication number: CN112506398A
Application number: CN202011337296.8A
Authority: CN
Inventors: 杜明利
Original assignee: Flying Carpet Information Technology Co ltd
Current assignee: Flying Carpet Information Technology Co ltd
Priority date: 2020-11-25
Filing date: 2020-11-25
Publication date: 2021-03-16
Anticipated expiration: 2040-11-25
Also published as: CN112506398B

Abstract

The invention relates to an image-text display method, an image-text display device and a computer readable medium. The image-text display method applied to AI assisted reading comprises the following steps: s100, covering a user terminal on a target reading material, wherein the user terminal is provided with a display part adopting OLED self-luminous transparent display; s200, acquiring image-text data of an area covered by a display part on the target reading material, and recording the image-text data as a shielded image-text; s300, generating associated graphics and texts matched with the shielded graphics and texts on a cloud server and/or a user terminal and/or a local server according to a preset matching relation between the graphics and text data and the associated data; and S400, displaying the associated graphics through a display part. The invention utilizes the characteristic that the OLED self-luminous transparent display screen can be used for a user to observe the lower image-text through the screen, ensures that at least the image-text information observed by the user does not come from the electronic display screen, and particularly for the students of the low ages, the character recognition mode can reduce the visual impairment of the reading to the students of the low ages.

Description

Image-text display method, device and computer readable medium for same

Technical Field

The invention relates to the technical field of OLED self-luminous transparent display, in particular to an image-text display method applied to AI auxiliary reading and instant data sharing, a device and a computer readable medium for the same.

Background

At present, along with the popularization of the internet, for students who cannot learn under the face of teachers or guardians, such as children who cannot be attended to or guardians to conduct courseware tutoring, students who cannot be school-study due to the influence of invariance, and the like, remote education becomes the first choice gradually, but the existing remote education mode can only select the mode of desktop computers, notebook computers, smart phones and smart tablet computers, and the prior art cannot provide other choices.

In the application terminal, the intelligent tablet computer is always in the position of unique chelating head, and due to the advantages of large-screen display and limited volume, the intelligent tablet computer is always used as a remote education application terminal recommended to students in all large, medium and small schools. However, the size of the smart tablet computer and the disadvantage that the smart tablet computer cannot replace the application of the mobile phone in the communication field determine that the smart tablet computer cannot be the first choice for most families in the presence of the smart phone which has more various functions, is more convenient for communication application and has a smaller size.

For the application of smart phones in remote education, the remote education system (see the prior patent CN 2016105919350-online real-time answering system and method, CN 2016106003318-assisted recitation method and device, CN 2016108023655-expression processing method, device and terminal, CN 2016106489491-graphical drawing method and device in online answering, etc.) pushed by simian tutoring has been the object pursued in the internet education industry in recent years, as well as the remote education system (see the prior patent CN 2017105043897-verse recitation evaluating method and system, a terminal and storage medium, CN 2018110255452-online method and device, computing device and storage medium, CN 2020102867168-dictation method and device, etc.), and the remote education system (see the prior patent CN 20220201044057-remote teaching questioning method, device, etc.), and byte jumping promotion remote education system (see the prior patent CN 20201044057-remote teaching, Questioning terminal and readable medium, CN 2020105831295-a method and device for displaying work correcting result, CN 2020105830729-a method and device for correcting work, etc.). However, the remote education system proposed by the leadership in the internet education industry is more focused on the application level, that is, the APP of the existing smart phone, and for the LED rear projection lighting technology adopted by the existing smart phone, if the LED rear projection lighting technology is used for a long time, the glasses of the students are seriously affected, and especially for the young children in the growth period, the eyesight of the students is seriously reduced if the device is used for a long time.

In addition, the above applications are not good enough for remote teaching, specifically, once a job is completed on a paper surface, students can only watch teaching videos and/or pictures on a smart phone, and cannot be associated with the writing content on the paper surface, so that compared with the existing live teaching and offline teaching videos, the interactivity is improved, but the improvement effect is limited, and the interactivity cannot be comparable to the effects of paper answering and blackboard answering, and therefore, a larger improvement space still exists in the interactivity of the teaching effect.

Disclosure of Invention

The invention aims to provide an image-text display method which is more beneficial to protecting eyesight, in particular to an image-text display method applied to AI auxiliary reading, and the technical scheme is as follows:

the image-text display method applied to AI assisted reading comprises the following steps:

s100, covering a user terminal on a target reading material, wherein the user terminal is provided with a display part adopting OLED self-luminous transparent display;

s200, acquiring image-text data of an area covered by a display part on the target reading material, and recording the image-text data as a shielded image-text;

s300, generating associated graphics and texts matched with the shielded graphics and texts on a cloud server and/or a user terminal and/or a local server according to a preset matching relation between the graphics and text data and the associated data;

and S400, displaying the associated graphics through a display part.

Further, the method for acquiring the image-text data in step S200 includes:

s210, acquiring a page image of the target reading material;

s220, acquiring the position of the display part on the target reading material, and recording the position as a drop point position;

s230, obtaining the area of the target reading material covered by the display part according to the position relation between the position of the drop point and the display part, and recording the area as an actual covered area;

s240, obtaining a region on the page image, which is associated with the actual shielded region, according to the corresponding relation between the target reading material and the page image, and recording the region as the shielded region of the image;

and S250, generating the image-text data from the image shaded area of the page image based on an OCR recognition technology.

Further, the method for acquiring the position of the falling point in step S220 may be selected from the following manners:

3.1. single scene matching method

S221, the user terminal is placed on the target reading material, and a camera arranged on the user terminal is used for shooting a local paper surface image on the target reading material;

s222, retrieving the image unit associated with the partial paper surface image in the page image in the step S210, and judging whether the image unit is unique in the page image or not,

if so, recording the position of the image unit in the page image as a first position, and obtaining a second position on the target reading material, which is associated with the first position, according to the position relationship between the page image and the target reading material, and taking the second position as the drop point position; or

If not, translating the user terminal on the target reading material, and repeating S221 and step 222; and/or

3.2. Planar trajectory tracking method

s222, retrieving the image unit associated with the partial paper surface image in the page image in the step S210, judging whether the image unit is unique in the page image or not,

if so, recording the position of the image unit in the page image as a first position, and obtaining a second position on the target reading material, which is related to the first position, according to the position relationship between the page image and the target reading material, and taking the second position as the position of the drop point;

if not, the following steps are continued:

s223, translating the user terminal on the target reading material;

s224, repeating the step 221 and recording the moving track of the user terminal;

s225, splicing all partial paper surface images based on the moving track to obtain a paper surface image combination;

s226, in step S210, the page image is searched for the image combination unit associated with the paper image combination, and whether the image combination unit is unique in the page image is judged,

if so, recording the position of the image combination unit in the page image as a third position, obtaining a fourth position on the target reading material, which is related to the third position, according to the position relationship between the page image and the target reading material, and generating the position of the drop point by using the fourth position; or

If not, repeating steps S223 to S226; and/or

3.3. Calibration method

S221, moving the user terminal in a direction close to the target reading material, enabling at least part of the target reading material to be always in a camera view field range arranged on the user terminal, and continuously projecting a coding pattern to the surface of the target reading material through a projection part arranged on the user terminal;

s222, acquiring a composite page image of the target reading material with the coding pattern through a camera;

s223, according to the matching relation between preset distortion reference data of the coding pattern and the camera position and posture data, obtaining camera position and posture data matched with the distortion data of the coding pattern in the composite page image;

s224, obtaining a camera motion track according to the sequence of each composite page image and the camera position and posture data matched with each composite page image;

s225, calculating the terminal point space coordinate of the motion trail of the user terminal, and taking the target reading material position matched with the terminal space coordinate as the drop point position; and/or

3.4. Binocular vision method

S221, moving the user terminal towards the direction close to the target reading material, and acquiring a page image of the target reading material through a camera arranged on the user terminal;

s222, calculating camera pose data matched with each page image based on binocular vision configuration of the camera;

s223, obtaining a camera motion track according to the sequence of each page image and the camera position and posture data matched with each page image;

s224, calculating the terminal point space coordinate of the motion trail of the user terminal, and taking the target reading material position matched with the terminal point space coordinate as the falling point position; and/or

3.5. External reference method

S221, calculating pose data of a certain position of the user terminal according to the mode 3.3 and/or the mode 3.4;

s222, moving the user terminal to the direction close to the target reading material;

s223, recording the real-time attitude data of the user terminal obtained through the gyroscope and the moving distance of the user terminal obtained through the accelerometer to obtain the motion track of the user terminal from the initial space coordinate;

and S224, calculating the end point space coordinate of the motion trail of the user terminal, and taking the target reading material position matched with the end point space coordinate as the drop point position.

Further, the position of the target reading material covered by the first display part is defined as a covered area, and the display area of the image-text data on the second display part in the step S310 of data sharing is an original text display area;

the step S320 of data sharing further includes:

s321, selecting and/or matching the position of the second user instruction information in the original text display area, and recording as a second user comment setting position;

s322, obtaining the corresponding position of the second user annotation setting position on the masked area according to the corresponding relation between the original text display area and the masked area, and recording the position as the second user annotation associated position;

and S323, selecting a position corresponding to the position related to the second user annotation on the second display part according to the position relation between the blocked area and the second display part, and displaying the second user instruction information at the position.

Further, the data sharing further includes:

s330, first user instruction information input by the first user on the first terminal is acquired, and the first user instruction information is received by the second terminal and displayed on the second display part.

the step S330 of data sharing is defined as:

s331, after the second user instruction information is displayed on the first display unit, starting input of the first user instruction information;

s332, displaying first user instruction information on the first display part, and selecting and/or matching the position of the first user instruction information on the first display part, and recording the position as a first user comment setting position;

s333, obtaining the position of the annotation position set by the first user on the covered area according to the position relation between the first display part and the covered area, and recording the position as the annotation associated position of the first user;

and S334, selecting a position corresponding to the position associated with the first user annotation on the second display part according to the corresponding relation between the blocked area and the original text display area, and displaying second user instruction information at the position.

Further, step S100 further includes:

s110, acquiring a page image of the target reading material, and receiving and displaying the page image on a second display part by a second terminal;

s120, selecting partial or all areas in the page image by the second user, and recording the partial or all areas as set target areas;

s130, obtaining a position which needs to be covered by the first display part on the target reading material according to the position of the set target area on the page image and the corresponding relation between the page image and the target reading material, and recording the position as a position to be fallen;

and S140, acquiring the relative position relationship between the target reading material and the current position of the first terminal, and displaying navigation guide information close to the position to be landed from the current position of the first terminal on the first display part.

Another objective of the present invention is to provide a method for displaying graphics and text for instant data sharing, which adopts the following technical solutions:

the image-text display method applied to instant data sharing comprises the following steps:

s100, covering a first terminal associated with a first user on a target reading material, wherein the first terminal is provided with a first display part adopting OLED self-luminous transparent display;

s200, acquiring image-text data of an area covered by the first display part on the target reading material;

s300, selecting and/or matching a second user associated with the first user, and establishing data sharing between the first terminal and the second terminal associated with the second user, wherein the data sharing comprises:

s310, the image-text data is received by the second terminal and displayed on a second display part arranged on the second terminal;

and S320, acquiring second user instruction information input by a second user on the second terminal, and receiving and displaying the second user instruction information on the first display part by the first terminal.

Further, the method for acquiring the image-text data in step S200 includes:

s210, acquiring a page image of the target reading material;

3.1. single scene matching method

3.2. Planar trajectory tracking method

if not, the following steps are continued:

s223, translating the user terminal on the target reading material;

If not, repeating steps S223 to S226; and/or

3.3. Calibration method

3.4. Binocular vision method

3.5. External reference method

the step S320 of data sharing further includes:

Further, the data sharing further includes:

the step S330 of data sharing is defined as:

Further, step S100 further includes:

Another objective of the present invention is to provide a terminal dedicated for a text display method, especially for a text display device for AI assisted reading, which has the following technical solutions:

be applied to supplementary reading's of AI picture and text display device includes:

the terminal body is used for covering the target reading material; and

the camera is used for acquiring image-text data of an area covered by the display part on the target reading material; and

the display part is arranged on the terminal main body and adopts OLED self-luminous transparent display, and the display part is used for displaying associated pictures and texts matched with the picture and text data of the area covered by the display part; and

and pre-storing a cloud service area matched with the image-text data and the associated data and/or a processor and/or a local server arranged on the terminal body.

Further, the camera is mounted on the terminal body.

The terminal body is provided with a track positioning module which is arranged on the back surface of the terminal body and is used for recording the moving track of the terminal body when the terminal body translates on the target reading material.

Further, the track positioning module is an optical positioning module and/or a grating positioning module and/or a laser positioning module and/or a track ball positioning module.

Another objective of the present invention is to provide a graphic display device for instant data sharing, which has the following technical solutions:

a graphic display device applied to instant data sharing comprises:

a first user terminal body for covering a target reading; and

a second user terminal body for pairing with the first user terminal body; and

the camera is matched with the first user terminal equipment and used for acquiring image-text data of an area covered by the display part on the target reading material; and

and the display part is arranged on the first user terminal main body and adopts OLED self-luminous transparent display, and the display part is used for displaying associated pictures and texts matched with the picture and text data of the area covered by the display part.

Further, the camera is mounted on the first user terminal body.

The mobile terminal further comprises a track positioning module arranged on the back surface of the first user terminal main body and used for recording the moving track of the terminal main body when the first user terminal main body translates on the target reading material.

It is a further object of the present invention to provide a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the teletext display method described above.

The invention has the beneficial effects that:

the invention utilizes OLED self-luminous technology to directly explain on paper reading materials or test paper, and can carry out targeted identification and data matching according to the content written on the paper reading materials or test paper by a user in the explaining process, thereby utilizing the characteristic that the OLED self-luminous transparent display screen can be used for the user to observe the lower image-text through the screen, ensuring that at least the image-text information observed by the user is not from an electronic display screen, and particularly for the students of the low ages, the character identification mode can reduce the visual impairment of reading to the students of the low ages, and is more beneficial to protecting the eyesight of the students in remote education.

Furthermore, the content displayed on the OLED self-luminous transparent display screen depends on the target reading material, and the triggering of the playing function of the OLED self-luminous transparent display screen depends on the picture of the target reading material, so that misuse of students on the electronic product can be reduced, and the using time of the students on the electronic product can be better controlled.

Further, if the first user is a student, the second user is a teacher or a parent, the first user and the second user share the display content on the OLED self-luminous transparent display screen, and under the condition of assisting the voice device carried on the terminal device, the method is equivalent to that the teacher or the parent directly remotely annotates and explains the student on the test paper site, so that the interactivity of remote teaching work is improved, and the teaching quality is improved.

Furthermore, the terminal position is positioned by a plane track tracking method, hardware depended by the plane track tracking method is similar to that applied to the existing hardware such as an intelligent dot matrix pen and a mouse, and the hardware is low in cost, so that the method has a high application prospect and a market prospect.

Drawings

Fig. 1 is a schematic back appearance diagram of an image-text display device according to embodiment 1 of the present invention;

FIG. 2 is a schematic diagram of the hardware configuration of the image-text display device of FIG. 1;

fig. 3 is a flowchart of an image-text display method according to embodiment 1 of the present invention;

fig. 4 is a schematic diagram of the hardware configuration of the image-text display device according to embodiment 4 of the present invention;

fig. 5 is an external view of an image-text display device according to embodiment 5 of the present invention;

FIG. 6 is a schematic diagram showing a hardware configuration of the graphic display device of FIG. 5;

fig. 7 is a schematic diagram of a hardware configuration of an image-text display device according to embodiment 6 of the present invention.

Detailed Description

Example 1:

the image-text display device of the embodiment belongs to a single-machine terminal device, can be used offline and accessed to the Internet, and is mainly suitable for the conditions of inconvenient or limited online, such as students in remote areas, lack of effective supervision of guardians, large network addiction and the like.

Referring to fig. 1 and 2, the graphic display device is mainly used for AI reading assistance, and is particularly suitable for stand-alone teaching, and the structure and the working principle of the graphic display device are explained in detail below.

The teletext display arrangement comprises only a user terminal 100, which user terminal 100 comprises a flat-plate-shaped terminal body 110 and a memory 120 arranged thereon, a communication bus 130, a processor 140, an optical positioning element 150, a rear camera 160 and a display 170. Wherein the content of the first and second substances,

the display 170 employs an OLED self-luminous transparent display screen.

A communication bus 130 is used to enable communications among these components.

The rear camera 160 is used for exposure under the control of a user instruction or other program instructions to capture an image frame under the terminal body 110.

The optical positioning element 150 is a light emitting diode or a laser positioning element, and is consistent with the light emitting diode positioning element and the laser positioning element on the mouse, specifically, a light emitting diode is arranged inside the terminal body 110, and the bottom surface of the terminal body 110 is illuminated by light emitted by the light emitting diode; then, a part of light reflected by the surface of the target reading matter at the bottom of the terminal body 110 is transmitted to a light sensing device (micro imager) through a group of optical lenses for imaging; thus, when the terminal body 110 moves, its moving track is recorded as a set of consecutive images photographed at a high speed; finally, a series of images captured on the movement track are analyzed and processed by using a dedicated image analysis chip (DSP, i.e. digital microprocessor 140) inside the terminal body 110, and the movement direction and the movement distance of the terminal body 110 are determined by analyzing the change of the positions of the feature points on the images, thereby completing the positioning of the terminal body 110. The difference between laser positioning and led positioning is that laser is used to replace the ordinary light emitted by the led.

Memory 120 includes high speed RAM memory 120 and stable memory 120 (e.g., disk memory 120), although memory 120 may alternatively be separate from the aforementioned storage for processor 140.

The processor 140 may be configured to call the AI assist reading teletext display program stored in the memory 120, and execute the teletext display method, see fig. 3, including the following steps:

and S100, covering the user terminal on the target reading material.

When the user terminal falls to the target reading material, the user terminal needs to fall from the right top of the target reading material and cannot fall from the side or the oblique top, so that the picture of the target reading material can be ensured to be in the field of view of the rear camera. Meanwhile, the rear camera is required to face the target reading material all the time; after the object reading material falls to the surface of the object reading material, the user terminal can be directly attached to the surface of the object reading material or keep a certain distance with the object reading material, the posture of the object reading material is not limited greatly, namely, the object reading material can keep a bending state, and the user terminal is required to be in a flexible structure in at least one embodiment, namely, the display, the printed board and the like are in a flexible design.

The target reading materials include but are not limited to book type exercise books, single examination papers, self-writing contents, calligraphy and painting works and the like. Of course, if the target reading material cannot be horizontally placed, the hanging state can also be maintained, and the user terminal does not fall onto the surface of the target reading material, but moves to be close to the target reading material on the plane in a horizontal pushing mode.

S200, acquiring image-text data of the area covered by the display part on the target reading material, and recording the image-text data as the shielded image-text.

The method for acquiring the image-text data comprises the following steps:

s210, acquiring a page image of the target reading material.

The acquisition mode is that when the user terminal is at a position far away from the target reading material, the user terminal takes a picture through the rear camera, and the shooting mode can be single shooting or multiple shooting. As for the image processing method obtained by photographing, refer to the prior art CN 101577783A-image processing method and electronic device thereof, the method comprises first defining a reference direction, and then obtaining the current usage direction of the electronic device. When the using direction does not accord with the reference direction, the image picture is adjusted to be displayed in the forward direction according to the reference direction and the using direction, and the processed image picture is stored. Therefore, the stored image frame is a forward display frame for the user no matter the direction of the user holding the electronic device when the user captures the image frame. Therefore, the rotation operation of the image frame which is not displayed in the forward direction when the image frame is browsed is avoided, and the convenience and the fluency of browsing the image frame are greatly improved.

Specifically, the method for acquiring the page image of the target reading material by the user terminal in this embodiment is to start the photographing function of the rear camera in one or two of the following ways in the process from the far end to the near end of the user terminal:

a, manually triggering, namely manually triggering a photographing key or a virtual key arranged on a terminal main body, and selecting a photographing time in a manual mode; and/or

The AI assists the reading image-text display program to automatically identify, the rear camera continuously takes a picture of the target reading matter, the machine vision distance measurement technology and/or the binocular vision distance measurement technology and/or the monocular vision distance measurement technology are applied to measure the distance between the rear camera and the target reading matter in real time, the photographing function is started when the set distance is reached, and the rear camera is applied to take a picture of the target reading matter;

after the photographing is started, the rear camera identifies characters and patterns on the target reading material and the boundary of the target reading material, and the photographed effective picture is stored in a memory in the terminal main body and is called by the processor.

S220, acquiring the position of the display part on the target reading material and recording the position as a landing point position.

The position of a drop point is obtained by adopting a calibration method, and reference can be made to a linear light source grating projection three-dimensional measurement system and a method-2019108214904 based on a phase shift method in the prior patent, wherein the system comprises a fringe projection module, a driving module, an image acquisition module, an image processing module, a display module and a data storage and transmission module; the method comprises the following steps: calibrating system parameters to obtain an internal parameter matrix and a distortion parameter matrix of the industrial camera; the driving module controls the m rows of LEDs to sequentially emit light to irradiate the grating to generate m phase-shift stripes and project the m phase-shift stripes onto the surface of a measured object; acquiring m phase-shift fringe images; carrying out distortion correction on the m stripe images through a distortion parameter matrix; calculating the phase distribution of the measured object in the view field according to the distorted and corrected fringe image; unwrapping the phase distribution to obtain continuous real phase distribution; and calculating the three-dimensional coordinates of the measured object in the world coordinate system to generate three-dimensional point cloud of the measured object. In this embodiment, the positions of the target and the camera need to be reversed, that is, the data that must be obtained finally is the point cloud of the camera and not the target.

The method comprises the following specific steps:

s221, moving the user terminal to the direction close to the target reading matter, enabling at least part of the target reading matter to be always in the field range of the rear camera arranged on the user terminal, and continuously projecting the coding pattern to the surface of the target reading matter through a projection part 180 arranged on the user terminal.

In order to perform this operation in this embodiment, an optical projection device needs to be installed on the back surface of the terminal body, but the code pattern projected by the projection device may be a projection grating as described in the above patent (2019108214904), or may be a pattern of a code pattern such as a grating or other pattern that increases interest or information amount, as projected by LiDAR (laser radar) mounted on a new generation mobile phone (model: iPhone 12 Pro) introduced by apple inc. Of course, the optical projection may project the coding pattern as described above, or may be used only for measuring coordinate information.

S222, acquiring a composite page image of the target reading material with the coding pattern through a rear camera.

And S223, obtaining rear camera position and posture data matched with the distortion data of the coding pattern in the composite page image according to the matching relation between the preset coding pattern distortion reference data and the camera position and posture data.

And S224, obtaining the motion trail of the rear camera according to the sequence of each composite page image and the camera position and posture data matched with each composite page image.

And S225, calculating the end point space coordinate of the motion trail of the user terminal, and taking the target reading material position matched with the terminal space coordinate as the falling point position.

The motion trail of the user terminal is calculated from the motion trail of the rear camera, the setting position of the rear camera on the terminal main body belongs to the internal parameters of the user terminal, and the motion trail of the user terminal can be conveniently converted from the motion trail of the rear camera through the internal parameters and the gyroscope arranged in the user terminal. At this time, of course, the user terminal is still a certain distance away from the target reading, and if the user terminal continues to be kept at the position, the landing position is the actual landing position, and is also the accurate landing position. If the user terminal needs to be close to the target terminal from the drop point position, on one hand, the size of the residual distance can be reduced as much as possible by replacing the rear camera or deploying the rear camera, so that a practical equivalent mode is adopted, and the residual distance is ignored; on the other hand, the remaining distance can be further optimized by the following external reference method:

s226, using the location of the user terminal as a reference location;

s227, moving the user terminal to the direction close to the target reading material;

s228, recording the real-time attitude data of the user terminal obtained through the gyroscope and the moving distance of the user terminal obtained through the accelerometer to obtain the motion track of the user terminal from the initial space coordinate;

and S229, calculating the end point space coordinate of the motion trail of the user terminal, and taking the target reading material position matched with the end point space coordinate as the drop point position.

That is, the position of the drop point without the photographing condition obtained by the prior calibration method is used as a reference drop point, and then the external reference method is applied to finally obtain the position of the user terminal completely falling on the target reading material by using the reference drop point as a starting point. In the process, the problem solved by the external reference method is the positioning of the gesture and the motion trail of the user terminal when no photographing condition exists.

And S230, obtaining the area covered by the display part of the target reading material according to the position relation between the position of the drop point and the display part, and recording the area as an actual covered area.

When the location of the landing point of the user terminal is determined in the above steps, the relative geometric relationship between the display part and the terminal body is determined by combining the internal parameter of the user terminal, namely the location of the display part on the user terminal, and the location of the display part on the target reading material is also determined, so that the area covered by the display part of the target reading material can be determined from the location, and the area is actually the location of the display part.

S240, obtaining a region related to the actual covered region on the page image according to the corresponding relation between the target reading material and the page image, and recording the region as the covered region of the image.

In the above step, the position of the display part on the target reading material is determined, but the user terminal records the page image of the target reading material, and the actual covered area reflects only the position relation between the display part and the target reading material in the world coordinate axis, so that the "corresponding relation" is needed to be combined to convert the world coordinate axis into the internal reference coordinate system on the page image, and the "corresponding relation" refers to the internal parameter of the rear camera, that is, the distortion of the page image relative to the target reading material is analyzed through the internal parameter of the rear camera, so as to realize the conversion between the world coordinate system and the internal reference coordinate system, and obtain the coordinate matrix of the actual covered area in the internal reference coordinate system, that is, the covered area of the image.

Since OCR recognition technology is a widely used picture and character recognition technology, it is selected here as a recognition technology for automatically producing text format documents from page images. The OCR Recognition technology is (Optical Character Recognition) a process in which an electronic device (e.g., a scanner or a digital camera) inspects characters printed on paper, determines the shape of the characters by detecting dark and light patterns, and then translates the shape into computer characters by a Character Recognition method; the method is characterized in that characters in a paper document are converted into an image file with a black-white dot matrix in an optical mode aiming at print characters, and the characters in the image are converted into a text format through recognition software for further editing and processing by word processing software. How to debug or use auxiliary information to improve recognition accuracy is the most important issue of OCR, and the term of icr (intelligent Character recognition) is generated accordingly. The main indicators for measuring the performance of an OCR system are: the rejection rate, the false recognition rate, the recognition speed, the user interface friendliness, the product stability, the usability, the feasibility and the like. The working process is as follows: an OCR recognition system is aimed at simply converting the image to make the graph in the image continuously stored and the data in the table and the words in the image uniformly changed into computer words if there is a table, so reducing the storage quantity of image data, reusing and analyzing the recognized words and saving the manpower and time for keyboard input. From image to result output, the image input, image pre-processing, character feature extraction, comparison and identification are carried out, and finally the error characters are corrected through manual correction, and the result is output.

And S300, generating the associated graphics and texts matched with the shielded graphics and texts on the user terminal according to the preset matching relation between the graphics and text data and the associated data.

The matching relation is an AI auxiliary reading graphic display program in a memory executed by a processor of the user terminal, the AI auxiliary reading graphic display program belongs to a prerecorded database stored in the memory and a corresponding algorithm of the graphic data, and the specific algorithm content can apply the semantic matching algorithm in the prior art, such as a remote education system (see the prior patent CN 2016105919350-online real-time question answering system and method, CN 2016106003318-auxiliary recitation method and device, CN 2016108023655-expression processing method, device and terminal, CN 106489491-graphic drawing method and device in online question answering and the like), a remote education system (see the prior patent CN 2017143897-a poetry recitation evaluation method and system, a terminal and storage medium, CN 2018110255452-online examination method and device, a computing device and storage medium, a remote education system, a computer, a, CN 2020102867168-a dictation method and apparatus), and a remote education system using byte jumping to push out (see the prior patent CN 2020107144057-a question method, apparatus, question terminal and readable medium for remote teaching, CN 2020103125895-a work correction result display method and apparatus, CN 2020105830729-a work correction method and apparatus, etc.).

And S400, displaying the associated graphics through a display part.

The display in this step includes the position, size and order of the display, and specifically, the following use example is used for illustration:

use example 1

The question stem on the target reading material is recognized to be silent night thinking of Li Bai on the user terminal, but when the students answer the wrong or blank answering area, the display part can play the pre-recorded teaching video, and the position and the size of the playing window of the teaching video can be freely selected by the user.

The usage example 1 can display the associated graphics and text associated with the graphics and text data at any position of the display part in a general manner.

Use example 2

When the user terminal recognizes that the stem on the target reading material is "1 +1 =", but the student answers after the "=" number is "3", the display section displays one "beta" (the font color may be set to red) at a position directly above "3" on the target reading material, and then displays the correct answer "2" at a position immediately adjacent thereto.

The use example 2 can be summarized as follows: and displaying the relevant graphics and texts relevant to the graphics and text data in a contraposition mode on the display part.

Use example 3

The question stem on the target reading material is identified on the user terminal to write a composition according to the requirement, and the answering area is an article written by students. According to the content of the article, firstly displaying an adjustable video playing window at any position on a player, and playing the explanation of the teacher on the article; then, the font and grammar errors in the article are noted and corrected as shown using example 2.

The use example 3 can be summarized as follows: and simultaneously, the related pictures and texts displayed in the use example 1 and the use example 2 are displayed on a display part.

In addition, in order to match with the image-text display effect, a voice player can be arranged on the user terminal, and the teaching sound is played in cooperation with the image-text display.

Example 2:

in embodiment 1, the positioning of the landing point position of the user terminal is limited to that the user terminal first lands on the target reading material, for example, when the display area of the user terminal can cover the target reading material, the image-text display method can already meet the user requirement, and the optical positioning element arranged on the corresponding user terminal mainly has the function of detecting the micro-movement of the user terminal relative to the target reading material to optimize the display position of the associated image-text, and if the micro-movement distance is ignored, the optical positioning element can be omitted.

However, when the display part of the user terminal is not enough to cover the whole page of the target reading material, the user needs to translate the user terminal at least once to complete the auxiliary reading of the whole page of the target reading material. At this time, the method for secondarily positioning the drop point position of the user terminal is as follows:

s2210, translating the user terminal on the target reading material;

s2211, the optical positioning element records the moving track of the user terminal by picking up the optical signal of the moving path;

in this step, the optical positioning element transmits light including visible light and invisible light onto the target reading material, and the light sensing device therein receives a light signal returned from the paper surface of the target reading material to form an image in the light sensing device (micro imager). Thus, when the user terminal moves, the moving track is recorded as a group of consecutive images shot at high speed. Finally, a special image analysis chip (DSP, namely a digital microprocessor) of the optical positioning element is used for analyzing and processing a series of images shot on the moving track, and the moving direction and the moving distance of the user terminal are judged by analyzing the change of the positions of the characteristic points on the images, so that the secondary positioning of the user terminal is completed.

And S2212, calculating the terminal point coordinate of the moving track, and taking the coordinate as the position of the drop point of the user terminal.

It should be noted that the end point coordinate is actually a coordinate of the optical positioning element, and accordingly, the landing point position of the whole user terminal can be obtained through a geometric position relationship of the optical positioning element on the terminal body preset in the internal parameters of the user terminal.

In addition, as an alternative technical scheme, referring to an alternative technical scheme of a mouse positioning technology, a track ball, a grating positioning mode and the like can be adopted to replace an optical positioning element, and the secondary positioning can also be realized.

Example 3:

the difference between the embodiment and the embodiment 2 is that a high-speed camera is used to replace an optical positioning element, the setting position of the high-speed camera is unchanged, but the lens of the high-speed camera is higher than the back surface of the terminal body and forms a slope with the back surface of the terminal body, so that before the optical light enters the high-speed camera and a target reading, the framing mode of the high-speed camera can refer to the prior patent CN 2012100171114-digital electronic pen, while writing and leaving normal ink handwriting, the handwriting captures a small point on the digital paper scanned by the camera, the microprocessor converts the image received by the point into coordinates, calculates and records the walking track of the digital pen, thereby obtaining the written handwriting, and stores the written handwriting in the memory. The USB transceiver is connected with the computer and can transmit the handwriting data to the computer, so that the electronic file of the handwriting can be obtained. The content input by the digital electronic digital pen is converted into editable material after ICR recognition processing, and finally can be automatically imported into Word processing software such as WordPad, Microsoft Word, Microsoft PowerPoint and the like through a recognition tool.

In this case, the method for acquiring the location of the drop point only in the user terminal is changed by applying the method for displaying the text and text of the text and text display device of the embodiment, and the specific steps are as follows:

s221, the user terminal is placed on the target reading material, and a high-speed camera arranged on the user terminal is used for shooting a local paper surface image on the target reading material.

Under the teaching of this step, the function of the rear camera originally arranged on the user terminal in embodiment 2 is mainly to shoot the page image of the target reading material, which may not participate in the positioning of the landing point position of the user terminal. The local page image in this step is an image of the target reading material captured within the field of view of the high-speed camera, and is defined as a local page image because the occupied position of the local page image is almost negligible with respect to the whole target reading material.

Moreover, this step is an alternative step to S221 in embodiment 1, that is, when the user terminal approaches to the target reading material, the rear camera does not need to perform the location of the drop point by using a machine vision technology, a binocular vision technology, a monocular vision technology, or the like; of course, in order to improve the positioning accuracy, the two can also be used in a matching way, and then the selection of the position of the falling point is carried out after the coupling calculation is carried out.

S222, retrieving the image unit associated with the partial paper surface image in the page image in the step S210, and judging whether the image unit is unique in the page image, wherein,

if so, recording the position of the image unit in the page image as a first position, and obtaining a second position on the target reading material, which is associated with the first position, according to the position relationship between the page image and the target reading material, and taking the second position as the drop point position;

if not, the user terminal is translated on the target reading material, and S221 and step 222 are repeated.

In this step, the image unit refers to the content in the page image consistent with the local paper image, and this definition mode is because the image-text data in the page image can be converted by applying the OCR technology, and after the conversion, the image-text data is converted into individual characters or patterns, and this step requires that before the OCR conversion, the page image is firstly latticed and separated according to the field range of the high-speed camera, that is, the license plate grid picture, and in order to reduce the mismatch relationship between the combination of limited separation and the random picture of the drop point, an algorithm can be combined and omitted to correct the difficulty and accuracy of retrieving the local paper image in the picture after separation, so as to find the image unit more quickly and accurately.

The corresponding relationship between the picture shot by the high-speed camera and the position of the target reading material and the page image is consistent with the corresponding steps in the embodiment 1.

The above steps can be summarized as a single view matching method, that is, the retrieval and comparison are performed according to the single view of the high-speed camera, so as to find out the only image element in the target reading material, thereby determining the position of the user terminal on the target reading material. However, it is not excluded that the target reading material has many image elements that overlap, i.e., many characters that overlap and many blank regions, and if it is determined in S222 that the image elements do not match, the following steps may be performed:

and S223, translating the user terminal on the target reading material.

And S224, repeating the step 221 and recording the movement track of the user terminal.

And S225, splicing all partial paper images based on the movement track to obtain a paper image combination.

S226, in step S210, retrieving an image combination unit associated with the page image combination from the page image, and determining whether the image combination unit is unique in the page image, wherein,

if so, recording the position of the image combination unit in the page image as a third position, obtaining a fourth position on the target reading material, which is related to the third position, according to the position relationship between the page image and the target reading material, and generating the position of the drop point by using the fourth position;

if not, steps S223 to S226 are repeated.

Steps S223-S226 adopt a multi-image element combination mode as a search element, so as to obtain a unique element combination unit after combining a plurality of repeated elements, if "AAAA" is recorded on the target reading material, the location of the user terminal cannot be determined when 1-3 "a" S are read, but when 4 "a" S are read and the combination is pieced together according to the movement trajectory, a unique search result can be obtained, that is, the unique location of the user terminal on the target reading material, that is, the location of the drop point, can be determined.

Of course, in order to improve the positioning efficiency, the above-mentioned single-element search and multi-element combination search modes may be used simultaneously on the user terminal, and if the target reading material records "AABA", then when "B" is read, the next matching is not needed, and the position of the user terminal is obtained directly by adopting the above-mentioned methods of S221-S222.

Preferably, in order to further improve the positioning accuracy, on one hand, a prompt instruction may be preset in the AI-assisted reading image-text display program, and the prompt instruction may trigger a sound signal and/or an image signal to prevent the user from continuing to move the user terminal when determining the location of the user terminal; on the other hand, considering the agility of the user action, a compensation algorithm can be preset in the AI-assisted reading image-text display program, and the compensation algorithm can calculate the coordinate position when the user terminal stops moving according to the recorded track from the user terminal landing point position to the user terminal stopping moving, and takes the position as the landing point position of the user terminal.

Example 4:

referring to fig. 4, a hardware configuration diagram of the image-text display device in the present embodiment is shown. As described in embodiment 1, the teletext display arrangement according to this embodiment is optionally a stand-alone application, i.e. for AI assisted reading, and specifically differs from embodiment 1 in that:

difference 1: the processor for database matching is not retained in the user terminal, and is moved to the server, although the user terminal may retain any processor unit required by other functions, the functions and functions of the processor are not the same as those of the processor in embodiment 1.

Difference 2: the user terminal is additionally provided with a terminal receiver and a terminal transmitter which are connected with the server receiver and the server transmitter on the server in a matching way so as to upload image data acquired by the rear camera to the server and download the associated pictures and texts matched with the server to the user terminal. Specifically, for the hardware device, the receiving and transmitting functions may adopt wireless communication means such as bluetooth, WIFI, radio frequency, cellular network, and the like. In addition, in order to meet the requirements of customers, the connection mode can be degraded to a wired connection mode, namely, a data line is used for connecting the user terminal and the server.

Difference 3: the AI-assisted reading image-text display program is moved to the memory of the server, and only the terminal interaction program is reserved on the memory of the user terminal, so that the aim of adapting to the configuration requirement of the user terminal under the condition of unobstructed network is fulfilled, namely, the image-text display method in the embodiment 1 can be realized by adopting the user terminal with low configuration, and only the matching of the image-text data and the associated data is changed from the original user terminal to the server.

Difference 4: the server is added, the server can be a local server which is arranged near the user and connected through a local area network, and also can be a cloud server which is connected remotely, the local server and the cloud server can be arranged simultaneously for better popularization requirements and operation requirements, and an edge algorithm can be adopted on the local server at the moment, so that the customer experience of the user terminal is improved under the condition that the operation precision is guaranteed as much as possible.

Example 5:

referring to fig. 5 and fig. 6, the external view mode teletext display arrangement of the embodiment is shown, but the teletext display arrangement of the embodiment is also applicable to AI assisted reading, i.e. limited to stand-alone use. This example differs from example 1 in that:

for the user terminal, an optical positioning element and a rear camera are removed, and a processor, a memory and a display are reserved; while adding a terminal receiver and a terminal transmitter as described in embodiment 2 above.

A viewing support 200 is added, and the viewing support 200 and the user terminal 100 are used together in a kit form and are relatively independent physically. The viewing support 200 includes a support body 210 that can be attached to a carrier of a subject reading material, including but not limited to: tables and chairs, walls, human limbs, etc.; the camera 220 preferably is a wide-angle camera that can take pictures of a larger space on the carrier. Meanwhile, a framing receiver and a framing transmitter which are wirelessly connected with the terminal receiver and the terminal transmitter are arranged in the user terminal, when the user terminal is used, the terminal transmitter transmits a trigger signal to the framing receiver so as to switch on the camera 220 for framing, and as an alternative technical scheme, when other trigger modes are arranged on the user terminal, such as mechanical trigger, gesture recognition trigger, photoelectric switch trigger, microswitch trigger and the like, the framing receiver and/or the terminal transmitter can be omitted; the framing transmitter transmits the target reading material picture shot by the camera 220 to the terminal receiver in real time.

The change of the image-text display method brought by the image-text display device mainly focuses on the acquisition of the position of the drop point and the acquisition of the image-text data. Wherein the content of the first and second substances,

the method for acquiring the image-text data comprises the following steps: when the user terminal does not cover the target reading material, the camera directly shoots and views a target reading material picture below the camera, the picture is used as a page image, and image-text data is acquired by combining an OCR technology.

The method for acquiring the position of the drop point comprises the following steps: after the user terminal is covered on the surface of the target reading material, under the condition that the display part of the user terminal is kept transparent, the camera shoots the superposed page image of the user terminal and the target reading material, and the coverage range of the dark part covered by the display part is identified through gray comparison and is used as the drop point position of the user terminal.

Preferably, the configuration of the above embodiment is arbitrarily configured on the user terminal, and the positioning manner of the drop point position and the view finding manner of the image-text data are still applied, but before the corresponding associated data is displayed on the display part, the drop point position and the image-text data need to be statistically processed to obtain more accurate drop point position and image-text data.

Preferably, on the premise of applying the view finding support, the user terminal may also select a mode of wireless or wired connection with the local server and/or the cloud server, so as to reduce the configuration requirement of a single user terminal, one corresponding local server may be matched with a plurality of user terminals, and one cloud server may be matched with a plurality of local servers or user terminals.

Example 6:

referring to fig. 7, a diagram illustrating a teletext display arrangement according to another application scenario of the invention is shown, which is different from the above-mentioned manner of stand-alone AI assisted reading, and the teletext display arrangement according to the embodiment is applied to an instant data sharing scenario, and its specific structure is:

including student terminals for students and teacher terminals for teachers, wherein the student terminals and teacher terminals may be shaped or differentiated from each other in accordance with the user terminals in the above-described embodiments.

In terms of hardware, the student terminal in the present embodiment is different from the user terminal in the above-described embodiments in that a student terminal receiver and a student terminal transmitter are added. The teacher terminal does not need to position and identify the target reading materials, so that an optical positioning element and a rear camera are omitted, and a teacher end receiver and a teacher end transmitter are added.

The processors of the student terminal and the teacher terminal can call the student-end image-text display program and the teacher-end image-text display program stored in the memories of the student terminal and the teacher terminal, and execute the following image-text display method, which comprises the following steps:

and S100, covering a student terminal associated with the student on the target reading material, wherein the student terminal is provided with a student end display part adopting OLED self-luminous transparent display.

In the implementation process of the step, in order to ensure that the student terminal can fall on the title desired by the teacher, the implementation can be carried out according to the following steps:

s110, acquiring a page image of the target reading material, and receiving and displaying the page image on the teacher terminal display part by the teacher terminal.

And S120, selecting partial or all areas in the page image by the teacher end user, and recording the partial or all areas as set target areas.

And S130, obtaining the position of the student end display part on the target reading material to be covered according to the position of the set target area on the page image and the corresponding relation between the page image and the target reading material, and recording the position as the position to be fallen.

And S140, acquiring the relative position relationship between the target reading material and the current position of the student terminal, and displaying navigation guide information close to the position to be landed from the current position of the student terminal on a student terminal display part.

S150, the student moves the student terminal according to the navigation guide information until the student terminal moves to the set target area, and the student terminal sends out prompt information which can be a voice signal or a picture and text signal.

And S200, acquiring image-text data of an area covered by the student-end display part on the target reading material.

S300, selecting and/or matching teachers associated with the students, and establishing data sharing between the student terminals and the teacher terminals associated with the teachers, wherein the data sharing comprises the following steps:

s310, the image-text data is received by the teacher terminal and displayed on a teacher-end display part arranged on the teacher terminal;

s320, acquiring teacher instruction information input by a teacher on a teacher terminal, and receiving and displaying the teacher instruction information on a student terminal display part by a student terminal; when this step is executed, in order to ensure the alignment display, that is, when the student end display part performs annotation at a position right above the corresponding original text, the method can be implemented according to the following steps:

and defining the position of the target reading material covered by the student end display part as a covered area, and defining the display area of the image-text data on the teacher end display part as an original text display area.

S321, selecting and/or matching the position of the teacher instruction information in the original text display area, and recording the position as a teacher annotation setting position;

s322, obtaining the corresponding position of the teacher annotation setting position on the covered area according to the corresponding relation between the original text display area and the covered area, and recording the position as the teacher annotation related position;

and S323, selecting a position corresponding to the teacher annotation related position on the teacher end display part according to the position relation between the covered area and the teacher end display part, and displaying the teacher instruction information at the position.

S330, student instruction information input by students on the student terminals is obtained, and the student instruction information is received by the teacher terminal and displayed on the teacher terminal display part. In order to ensure that the display is aligned on the teacher-side display unit when this step is executed, the following steps are performed:

s331, after the teacher instruction information is displayed on the student end display part, the input of student instruction information is started;

s332, displaying student instruction information on the student end display part, and selecting and/or matching the position of the student instruction information on the student end display part to be recorded as a student annotation setting position;

s333, obtaining the position of the annotation position set by the student on the covered area according to the position relation between the student end display part and the covered area, and recording the position as the student annotation related position;

and S334, selecting a position corresponding to the student annotation associated position on the teacher end display part according to the corresponding relation between the shielded area and the original text display area, and displaying teacher instruction information at the position.

In the embodiment, in the determination mode of the selected relationship between the student and the teacher, the selection means that all the selectable teacher profiles are displayed on the display part of the student end, the student freely selects and determines the selected relationship between the student and the teacher, and the teacher can also select the corresponding student on the teacher terminal; matching means that the processor automatically distributes teachers to students and automatically establishes connection according to set conditions of the students, such as conditions of schools, classes, disciplines, historical mistakes and the like.

As an additional technical solution, in this embodiment, both the student terminal and the teacher terminal adopt the user terminal of the student terminal type as in embodiment 6, but the difference is that a student-side image-text display program and a teacher-side image-text display program are simultaneously set in the user terminal, and the user identity is identified and the corresponding image-text display program is started through technologies such as biometric identification and/or password identification and/or machine vision identification. Preferably, an AI auxiliary reading image-text display program which is parallel to the image-text display program of the student end and the image-text display program of the teacher end can be arranged in the user terminal, namely, the student can freely select the single-edition work or the online work.

As an alternative technical solution, the student terminal and the teacher terminal may adopt a terminal mode connected with the server, that is, the student terminal and/or the teacher terminal may be connected with the cloud server and/or the local server in a wired and/or wireless mode.

As an alternative, the student terminal and the teacher terminal may adopt the structure of the user terminal in embodiment 5.

The above description is only about the preferred embodiment of the present invention, but it should not be understood as limiting the claims, and the present invention may be modified in other structures, not limited to the above structures. In general, all changes which come within the scope of the invention are intended to be embraced therein.

Claims

1. The image-text display method applied to AI assisted reading is characterized by comprising the following steps:

and S400, displaying the associated graphics through a display part.

2. Teletext display method according to claim 1,

the image-text data acquisition method of the step S200 comprises the following steps:

s210, acquiring a page image of the target reading material;

3. The method for displaying text and text according to claim 2, wherein the method for obtaining the position of the falling point in step S220 is selected from the following methods:

if not, the following steps are continued:

s223, translating the user terminal on the target reading material;

If not, repeating steps S223 to S226; and/or

4. The image-text display method applied to the instant data sharing is characterized by comprising the following steps:

5. The method for displaying graphics and text according to claim 4, wherein the position of the target reading material covered by the first display part is defined as a covered area, and the display area of the graphics and text data on the second display part in the step S310 of data sharing is defined as an original text display area;

the step S320 of data sharing further includes:

6. Teletext display method according to claim 4, wherein the data sharing further comprises:

7. The method for displaying graphics and text according to claim 6, wherein the position of the target reading material covered by the first display part is defined as a covered area, and the display area of the graphics and text data on the second display part in the step S310 of data sharing is defined as an original text display area;

the step S330 of data sharing is defined as:

8. The teletext display method according to any one of claims 4-7, wherein step S100 further comprises:

9. Teletext display arrangement, comprising a terminal for performing a teletext display method according to any one of claims 1 to 8.

10. Computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out a teletext display method according to any one of claims 1 to 8.