CN117908669A - Auxiliary reading method and system - Google Patents
Auxiliary reading method and system Download PDFInfo
- Publication number
- CN117908669A CN117908669A CN202311823768.4A CN202311823768A CN117908669A CN 117908669 A CN117908669 A CN 117908669A CN 202311823768 A CN202311823768 A CN 202311823768A CN 117908669 A CN117908669 A CN 117908669A
- Authority
- CN
- China
- Prior art keywords
- target
- image
- finger
- user
- search
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000013473 artificial intelligence Methods 0.000 claims description 47
- 230000011218 segmentation Effects 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 4
- 208000003464 asthenopia Diseases 0.000 abstract description 8
- 238000013461 design Methods 0.000 description 25
- 230000000694 effects Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 238000003062 neural network model Methods 0.000 description 4
- 238000012015 optical character recognition Methods 0.000 description 3
- 230000001886 ciliary effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 210000003205 muscle Anatomy 0.000 description 2
- 206010063385 Intellectualisation Diseases 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 208000001491 myopia Diseases 0.000 description 1
- 230000004379 myopia Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Abstract
The application provides an auxiliary reading method and system, wherein the method comprises the following steps: acquiring a first image of a page in a book of a user, which is shot by shooting equipment; displaying a first image to a user through a display device, wherein the distance between the display device and the user is adjustable; if the first image is identified to contain the finger of the user, determining the position of the top end of the finger in the first image; according to the position of the top end of the finger, determining a first retrieval target closest to the top end of the finger in the first image, wherein the first retrieval target is a word or a picture; acquiring a first retrieval result corresponding to the first retrieval target; the first search result is displayed to the user while the first image is displayed to the user through the display device. Therefore, the reader can watch the paper book file in a long distance, the problem of asthenopia caused by long-time close reading of the reader is avoided, and if the image is identified to contain the finger of the user, the user can search according to the search target closest to the top end of the finger, the search result is displayed to the user, and the user can answer the question.
Description
The present application claims priority from the chinese patent office, application number 202311812523.1, entitled "a method and system for assisted reading" filed on day 26, 12, 2023, the entire contents of which are incorporated herein by reference.
Technical Field
The application relates to the technical field of image processing, in particular to an auxiliary reading method and an auxiliary reading system.
Background
Long close reading can place the eye ciliary muscles in tension all the time, resulting in visual fatigue. If the reader reads with eyes for a long time and high intensity, myopia of eyes is easily caused.
However, the traditional eye-protection reading device is provided with an ink screen reader and the like, and the reader can effectively reduce the harm of short-wavelength blue light emitted by a conventional display screen to human eyes, but still needs the reader to read in a short distance, and can not solve the problem that long-time short-distance reading can cause visual fatigue.
Therefore, a scheme for enabling readers to watch paper book files remotely is needed, so that the problem of asthenopia caused by long-time close reading of readers is avoided.
Disclosure of Invention
The application provides an auxiliary reading method and an auxiliary reading system, which can enable a reader to watch a paper book file in a long distance, so that the problem of asthenopia caused by long-time close reading of the reader is avoided.
In a first aspect, there is provided a method of assisting reading, the method comprising:
Acquiring a first image of a page in a book of a user, which is shot by shooting equipment;
displaying a first image to a user through a display device, wherein the distance between the display device and the user is adjustable;
If the first image is identified to contain the finger of the user, determining the position of the top end of the finger in the first image;
According to the position of the top end of the finger, determining a first retrieval target closest to the top end of the finger in the first image, wherein the first retrieval target is a word or a picture;
Acquiring a first retrieval result corresponding to a first retrieval target;
the first search result is displayed to the user while the first image is displayed to the user through the display device.
In one possible design, determining a first search target closest to the finger tip in the first image based on the position of the finger tip includes:
identifying the content of the page and the position of the content in the first image by adopting an artificial intelligence algorithm;
Determining target content closest to the top of the finger in the page according to the position of the top of the finger and the position of the content in the first image, wherein the target content is characters or pictures;
the first search target is determined based on the target content closest to the tip of the finger.
In one possible design, the target content is text, and determining the first search target according to the target content closest to the top of the finger includes:
dividing target content into a plurality of word segments by adopting an artificial intelligence algorithm;
Acquiring the position of each word in a plurality of word segments;
determining the distance between each word segment and the top of the finger according to the position of each word segment and the position of the top of the finger;
Determining a word segmentation with the smallest distance with the top end of the finger as a first retrieval target;
The target content is a picture, and the first retrieval target is determined according to the target content closest to the top of the finger, including:
The target content is determined as a first retrieval target.
In one possible design, obtaining a first search result corresponding to a first search target includes:
Acquiring a first search result corresponding to a first search target based on a first association relation stored in a cloud server, wherein the cloud server stores a plurality of first association relations, and each first association relation is used for associating the search target, target content to which the search target belongs and a search result corresponding to the search target;
And/or searching the first search target by adopting a search engine to obtain a first search result.
In one possible design, the target content is text, and the obtaining a first search result corresponding to the first search target based on a first association relationship stored in the cloud server includes:
Identifying target content of the first retrieval target by adopting an artificial intelligence algorithm;
The target content and the first retrieval target are used as first matching conditions, the first matching conditions are matched with each first association relation stored in the cloud server, whether the first association relations comprise the first target association relation matched with the first matching conditions or not is determined, and the first target association relation is used for associating the target content, the first retrieval target and the first retrieval result;
And if the plurality of first association relations comprise first target association relations, obtaining a first retrieval result in the first target association relations.
In one possible design, the method further comprises:
traversing target contents of pages in the first image, and in the process of traversing each target content:
Sequentially matching the current target content with each first association relation as a second matching condition;
Determining whether each first association relation comprises a second target association relation matched with a second matching condition, wherein the second target association relation is used for associating the current target content, the retrieval target of the current target content and the retrieval result corresponding to the retrieval target of the current target content;
If each first association relation comprises a second target association relation, displaying the first image to the user through the display equipment and displaying a corner mark control, wherein the corner mark control is used for controlling whether to display a second search result, and the second search result is a search result corresponding to a search target of the current target content.
In one possible design, the method further comprises:
Acquiring a second image of a page in a book of a user shot by the camera equipment, wherein the shooting time of the second image is later than that of the first image;
If the part which is overlapped with the corner mark control part in the finger area of the user is identified from the second image, determining a second retrieval result of the target content association corresponding to the corner mark control part according to a second target association relationship corresponding to the corner mark control part;
and displaying the second search result to the user through the display device.
In one possible design, the method further comprises:
and if the next frame image of the first image is identified to not contain fingers, stopping displaying the first search result when the next frame image is displayed through the display device.
In one possible design, the display device includes a display screen, the method further comprising:
responding to the sliding operation of the fingers of the user on the display screen, and converting the sliding track of the fingers of the user on the display screen into annotation text by adopting an artificial intelligence algorithm;
And storing the annotation text, the position of the annotation text in the first image, the account number of the user and a second association relation formed by the content of the preset position in the first image to the cloud server.
In a second aspect, an auxiliary reading system is provided, including an artificial intelligence processor, a display device and a camera device, wherein the display device includes a first wireless network module and a display screen, the camera device includes a second wireless network module, a camera module, a base, a bracket and a cantilever, the artificial intelligence processor is installed on the display device or the camera module, and the distance between the display device and a user is adjustable;
the camera module is fixed on the cantilever, and the cantilever is used for adjusting the angle of the camera module;
the cantilever is fixed at one end of the bracket;
the other end of the bracket is fixedly connected with a base, and the base is used for realizing stable placement of the camera equipment;
the camera module is used for sending all the shot images of pages in the book of the user to the display equipment through the second wireless network module according to the frame sequence;
The display device is used for receiving each image through the first wireless network module;
The display device is also used for displaying various images to a user through the display screen;
The artificial intelligence processor is used for determining the position of the top end of the finger if the image is identified to contain the finger of the user;
The artificial intelligence processor is also used for determining a first retrieval target closest to the top of the finger in the image according to the position of the top of the finger, wherein the first retrieval target is a word or a picture;
the artificial intelligence processor is also used for obtaining a first retrieval result corresponding to the first retrieval target;
The artificial intelligence processor is further configured to display the first search result while displaying the first image to the user through the display device.
In the embodiment of the application, the distance between the display equipment and the user can be adjusted according to the requirement of the user, so that the image of the book page shot by the camera equipment is displayed through the display equipment, and a reader can watch the paper book file in a long distance, thereby avoiding the problem of asthenopia caused by long-time short-distance reading of the reader. In addition, if the image is identified to contain the finger of the user, the user can search according to the search target closest to the top end of the finger, the search result is displayed to the user, the user can solve the problem at any time, the user does not need to search on the internet, and therefore the reading experience of the user is improved.
Drawings
In order to more clearly illustrate the technical solution of the present application, the drawings that are needed in the embodiments will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.
FIG. 1 is a schematic diagram of an auxiliary reading system according to an exemplary embodiment of the present application;
FIG. 2 is a schematic flow chart of an auxiliary reading method according to an exemplary embodiment of the present application;
FIG. 3 is a schematic diagram of content displayed on a display screen in an auxiliary reading system according to an exemplary embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
Fig. 1 is a schematic diagram of an auxiliary reading system according to an exemplary embodiment of the present application, and as shown in fig. 1, the system includes an artificial intelligence processor 1, a display device 2, and an image capturing device 3. The display device 2 includes a first wireless network module 21 and a display screen 22, the camera device 3 includes a camera module 31, a base 32, a bracket 33, a cantilever 34, and a second wireless network module 35, and the artificial intelligence processor 1 is installed on the display device 2 or the camera module 31, and a distance between the display device 2 and a user is adjustable. Wherein, base 32 is used for providing the support for whole image pickup apparatus 3, and the bottom of base 32 fixed connection support 33. The support 33 can be high, and 360 degrees swivelling joint cantilever 34's on the horizontal plane one end in the top of support 33, cantilever 34's the other end and camera module 31 fixed connection. The cantilever 34 can change the position of the camera module 31 by rotating, so that the camera module 31 can clearly photograph the page of the book to be read.
In the above system, the user places the page of the book to be read under the camera module 31, and the camera module 31 captures the page of the current book to obtain the first image. The camera module 31 transmits the first image to the first wireless network module 21 of the display device 2 through the second wireless network module 35, and the display device 2 displays the first image through the display screen 22 for the user to read through the display screen 22. Because the distance between the display device 2 and the user is adjustable, the user can adjust the distance between the display device 2 and the user according to the requirement of the user, thereby playing the effect of remotely watching the paper book file and avoiding the problem of asthenopia caused by long-time short-distance reading of readers. The present application is not limited in the type of wireless transmission technology employed by the first wireless network module 21 and the second wireless network module 35.
Further, the artificial intelligence processor 1 can identify the content of the book page in the first image, and also can identify the finger of the user, and according to the position of the finger, the position of the content in the book page and the position of the control displayed on the display screen 22, the interaction with the user can be realized, so as to promote the intellectualization of the auxiliary reading system. For example, when a user's finger is near a certain content of a book page, displaying the result of the network search on the content; for another example, when there is a portion where the display area of the user's finger on the display screen 22 coincides with the display area of the control on the display screen 22, content associated with the control is displayed, and so on. Therefore, the auxiliary reading system provided by the application can also realize real-time consulting, searching and the like of the reading content, thereby improving the reading experience and the interestingness of a user.
Illustratively, the auxiliary reading system further comprises a cloud server. The cloud server is used for storing the search targets and the corresponding search results generated when the user uses the auxiliary reading system, so that when other users use other auxiliary reading systems, the search targets and the corresponding search results uploaded to the cloud server by the auxiliary reading system can be checked through the cloud server, and the sharing of the search targets and the search results among different users is realized.
Based on the auxiliary reading system shown in fig. 1, as shown in fig. 2, the application provides an auxiliary reading method, which comprises the following steps:
s110, acquiring a first image of a page in a book of a user, which is shot by an image shooting device.
Specifically, the image pickup apparatus transmits a video stream of a photographed book page to the display apparatus, the video stream being composed of consecutive multi-frame images including the first image.
And S120, displaying the first image to a user through a display device.
Wherein the distance between the display device and the user is adjustable.
Because the distance between the display device and the user is adjustable, and the size of the display screen of the display device can be determined according to the user requirement, when the user is far away from the display screen to realize that the eye ciliary muscles of the user are in a relaxed state, the content of the book page displayed on the display screen can still be seen clearly.
And S130, if the first image is identified to contain the finger of the user, determining the position of the top end of the finger in the first image.
Wherein, the finger tip refers to one end of the finger tip.
For example, as shown in fig. 3, the position of the finger tip, that is, the coordinates of the pixel closest to the origin of the image coordinate system in the finger region.
Illustratively, an artificial intelligence algorithm is employed to identify whether the first image contains a user's finger, and if so, further identify the location of the finger tip in the image.
The type of artificial intelligence algorithm is not limited by the application, as long as the finger of the user can be recognized, and the characters and pictures in the page can be recognized by optical character recognition (Optical Character Recognition, OCR).
S140, determining a first retrieval target closest to the finger tip in the first image according to the position of the finger tip.
Wherein the first search target is text or picture.
In one possible design, determining the first search target closest to the finger tip in the first image based on the position of the finger tip is accomplished by:
identifying the content of the page and the position of the content in the first image by adopting an artificial intelligence algorithm;
Determining target content closest to the top of the finger in the page according to the position of the top of the finger and the position of the content in the first image, wherein the target content is characters or pictures;
the first search target is determined based on the target content closest to the tip of the finger.
The content of the page comprises text, pictures and the like, and the application is not limited to the form of the content. Thus, the target content may be a line of text, or may be a single picture or multiple pictures side by side.
Illustratively, the position of the content is the coordinates of the pixel where the vertical coordinate value is the largest in the pixel area occupied by the content on the image.
For example, as shown in fig. 3, a pixel with the largest vertical coordinate value in the pixel area occupied by the target content (i.e., the first target line character) is a first pixel point, and the coordinates (x 1, y 1) of the first pixel point are taken as the position of the first target line character. It can be seen that the first target line of text is the last line of text of the finger.
Illustratively, determining the target content in the page closest to the finger tip is accomplished by:
Respectively determining a target pixel point with the maximum vertical coordinate value in a pixel area occupied by each row of characters or each picture;
calculating the absolute value of the difference between the vertical coordinate of each target pixel point and the vertical coordinate of the pixel point at the top end of the finger;
And determining the content of the pixel region to which the target pixel point with the minimum absolute value of the difference value belongs as target content.
In the above example, the content of the page and the position of the content in the first image can be accurately and quickly identified by adopting an artificial intelligence algorithm. Thereby, the target content closest to the finger tip in the page is determined in combination with the position of the finger tip, so that the first retrieval target is accurately determined.
For example, if the target content closest to the tip of the finger includes a picture, the picture is determined as the first search target.
In one possible design, if the target content closest to the top of the finger is text, the method is implemented by determining the first search target according to the target content closest to the top of the finger:
dividing target content into a plurality of word segments by adopting an artificial intelligence algorithm;
acquiring the position of each word segmentation in a plurality of word segmentations identified by adopting an artificial intelligence algorithm;
determining the distance between each word segment and the top of the finger according to the position of each word segment and the position of the top of the finger;
and determining the word with the smallest distance from the top of the finger as a first retrieval target.
Illustratively, the location of each term is determined based on the pixel area occupied by the term on the image.
For example, the pixel coordinates of any one of the vertices of the pixel region occupied by the segmentation on the image are set as the position of the segmentation.
It should be understood that if the pixel coordinates of any one of the vertices of the pixel region occupied by the segmentation on the image are taken as the positions of the segmentation, the vertices at the same positions on the pixel region are used for different segmentation.
Illustratively, determining the distance of each word segment from the tip of the finger based on the location of each word segment and the location of the tip of the finger is accomplished by:
if the pixel coordinates of the vertices of the pixel area occupied by the segmentation on the image are taken as the segmentation positions, determining the distance d between each segmentation and the top of the finger through the following formula (1):
Wherein, the pixel coordinate of the vertex of the pixel area occupied by the segmentation on the image is (a 1,b1), and the pixel coordinate of the finger top is (a 2,b2).
The definition of the position of the word is not limited in the present application, and for example, the pixel coordinates of the center point of the pixel region occupied by the word may be used as the position of the word. Correspondingly, the calculation of the distance between each word segment and the top end of the finger is only needed by referring to the formula (1).
Illustratively, the method is implemented by obtaining a location of each of a plurality of tokens identified using an artificial intelligence algorithm by:
acquiring a neural network model corresponding to an artificial intelligent algorithm;
Training a neural network model by using a data set consisting of images divided into word segments and marked with word segment positions;
Inputting the first image into a trained neural network model, and obtaining the word segmentation output by the neural network model and the corresponding position.
In the above example, the artificial intelligence algorithm is adopted to accurately divide the target content into a plurality of words based on the semantics, and identify the position of each word. According to the position of each word and the position of the top of the finger, the word with the smallest distance with the top of the finger can be accurately determined, and the word is taken as a first search target.
S150, obtaining a first search result corresponding to the first search target.
In one possible design, when the auxiliary reading system includes a cloud server, the cloud server stores a plurality of first association relationships, and each first association relationship is used for associating a search target, target content to which the search target belongs, and a search result corresponding to the search target. Therefore, the method can be realized by acquiring the first search result corresponding to the first search target in the following manner:
And acquiring a first retrieval result corresponding to the first retrieval target based on the first association relation stored in the cloud server.
Based on the above example, different auxiliary reading systems can realize sharing of search results through the cloud server.
In one possible design, the specific steps of assisting the reading system to obtain the first search result in the case that the target content is text include:
Identifying target content of the first retrieval target by adopting an artificial intelligence algorithm;
The target content and the first retrieval target are used as first matching conditions, the first matching conditions are matched with each first association relation stored in the cloud server, whether the first association relations comprise the first target association relation matched with the first matching conditions or not is determined, and the first target association relation is used for associating the target content, the first retrieval target and the first retrieval result;
And if the plurality of first association relations comprise first target association relations, obtaining a first retrieval result in the first target association relations.
By way of example, an operation and maintenance person can research which search targets are generally searched by readers when reading books, compare the researched search targets with the first association relationship stored in the cloud server, acquire search results of search targets which are not stored in the cloud server through a search engine, form a new first association relationship and add the new first association relationship into the cloud server.
In the above example, the artificial intelligence algorithm can identify the target content to which the first search target belongs, while identifying the first search target. If a matched target first association relation can be searched according to the target content and the first search target, the front and rear text information of the search target in the first association relation is the same as the front and rear text information of the first search target, so that a first search result can be directly obtained according to the first target association relation. Because the front and rear text information of the target content plays a role in limiting the first retrieval target, the target content and the first retrieval target are simultaneously used as matching conditions, and the accuracy of the determined first target association relationship can be improved.
In order to cope with the situation that the first target association relationship is not included in the plurality of first association relationships, in a feasible design, the application further provides a way for obtaining the first search result:
and searching the first search target by adopting a search engine to obtain a first search result.
Specifically, the identified first search target is input to a search engine for searching, and a search result with a preset format output by the search engine is used as a first search result. The setting of the preset format can exclude redundant search results, for example, the explanation of the first search target output by hundred degrees corresponding to hundred degrees encyclopedia is used as the first search result, and related advertisement content can be excluded.
Illustratively, to ensure consistency of the search results obtained by different auxiliary reading systems, the different auxiliary reading systems use the same search engine for searching.
The order of the two ways of obtaining the first search result provided in the present application is not limited, for example, the first association relationships stored in the cloud server may be searched first, the first search result may be directly obtained if the first target association relationship is searched, and the first search target may be searched by using the search engine if the first target association relationship is not searched. And then, the search result, the search target and sentences to which the search target belongs form a first association relation and are stored in the cloud server.
For another example, the search engine may be first used to search the first search target, so as to obtain the first search result. And then searching in each first association relation stored in the cloud server by taking the first search target and sentences to which the first search target belongs as matching conditions, and if the matched first target association relation is searched, directly replacing the original first search result in the first target association relation with the first search result searched by the search engine in order to ensure the instantaneity of the first search result in the first target association relation. If the matched first target association relation is not searched, the first search result, the first search target and sentences to which the first search target belongs are directly formed into the first association relation and stored in the cloud server.
In the above example, the first association relationship stored in the cloud server only associates the information of the sentence to which the search target belongs, but does not associate the information such as the book name, the page number and the like, so that the information such as the book name, the page number and the like does not need to be scanned and stored, and the operation resource of the auxiliary reading system is saved.
S160, displaying the first image to the user through the display device while displaying the first search result.
The application does not limit the position of displaying the first search result on the display screen, as long as the pixel area occupied by the first search result and the pixel area occupied by the sentence to which the first search target belongs have no overlapping part, namely the first search result is displayed under the condition of not influencing the reading of the user.
For example, as shown in fig. 3, after the finger of the user points to the first search target, a dialog box is displayed in an area located on another page on the display screen, and a first search result corresponding to the first search target is displayed in the dialog box.
In one possible design, on the basis that the cloud server stores each first association relationship, when different users use different auxiliary reading systems, the effect of searching the page in the books of the photographed users can be shared, and the specific steps include:
traversing target contents of pages in the first image, and in the process of traversing each target content:
Sequentially matching the current target content with each first association relation as a second matching condition;
Determining whether each first association relation comprises a second target association relation matched with a second matching condition, wherein the second target association relation is used for associating the current target content, the retrieval target of the current target content and the retrieval result corresponding to the retrieval target of the current target content;
If each first association relation comprises a second target association relation, displaying the first image to the user through the display equipment and displaying a corner mark control, wherein the corner mark control is used for controlling whether to display a second search result, and the second search result is a search result corresponding to a search target of the current target content.
Illustratively, in traversing each target content, displaying the first image to the user via the display device while displaying the corner mark control is accomplished by:
Acquiring a first pixel area occupied by current target content;
Acquiring a second pixel area occupied by the last row of target content of the current target content and a third pixel area occupied by the next row of target content of the current target content;
Determining a fourth pixel region, wherein the fourth pixel region is positioned between the first pixel region and the second pixel region and has no overlapping part with the first pixel region and the second pixel region, or the fourth pixel region is positioned between the first pixel region and the third pixel region and has no overlapping part with the first pixel region and the third pixel region;
and displaying the first image to a user through the display device, and simultaneously displaying the corner mark control in a fourth pixel area of the first image.
It should be noted that, the display positions of the corner mark control are merely exemplary, and may be set according to requirements, which is not limited in the present application, for example, the corner mark control is fixedly displayed at the vertex angle of the image.
According to the example, after the page is acquired, the page is traversed, if the search records of other people on any content of the same page are found, the corner mark is displayed, and the user confirms whether to display the search results of other people by determining whether to place the finger at the position of the corresponding page of the corner mark. Therefore, the effect of sharing the search result across users is achieved, and the interestingness of the auxiliary reading system is improved.
In one possible design, the manner in which the user displays the search results through the corner mark control is achieved by:
Acquiring a second image of a page in a book of a user shot by the camera equipment, wherein the shooting time of the second image is later than that of the first image;
If the part which is overlapped with the corner mark control part in the finger area of the user is identified from the second image, determining a second retrieval result of the target content association corresponding to the corner mark control part according to a second target association relationship corresponding to the corner mark control part;
and displaying the second search result to the user through the display device.
In one possible design, the method further comprises:
and if the next frame image of the first image is identified to not contain fingers, stopping displaying the first search result when the next frame image is displayed through the display device.
When the image does not contain the finger of the user, the user is not required to watch the search result any more, and the display of the first search result is stopped directly.
In one possible design, the display device includes a display screen, the display screen being a touch display screen, the method further comprising:
responding to the sliding operation of the fingers of the user on the display screen, and converting the sliding track of the fingers of the user on the display screen into annotation text by adopting an artificial intelligence algorithm;
And storing the annotation text, the position of the annotation text in the first image, the account number of the user and a second association relation formed by the content of the preset position in the first image to the cloud server.
The preset position is set according to actual requirements. For example, the preset position is the position where the first two lines of characters are located on the page with the smaller page number in the first image.
In addition, the position of the endorsement text in the first image, the account number of the user and the content of the preset position in the first image are stored to the cloud server, so that when the user uses the account number of the user to read on the auxiliary reading system next time, if the system searches the matched second association relation according to the content of the preset position in the first image in the cloud server, the content of the endorsement text to be displayed and the position of the endorsement text in the first image are determined according to the searched second association relation, and the endorsement text is displayed at the position, so that the reading experience of the user is improved.
In the embodiment of the application, the distance between the display equipment and the user can be adjusted according to the requirement of the user, so that the image of the book page shot by the camera equipment is displayed through the display equipment, and a reader can watch the paper book file in a long distance, thereby avoiding the problem of asthenopia caused by long-time short-distance reading of the reader. In addition, if the image is identified to contain the finger of the user, the user can search according to the search target closest to the top end of the finger, the search result is displayed to the user, the user can solve the problem at any time, the user does not need to search on the internet, and therefore the reading experience of the user is improved.
The application also provides an auxiliary reading system, which comprises an artificial intelligent processor, a display device and a camera device, wherein the display device comprises a first wireless network module and a display screen, the camera device comprises a second wireless network module, a camera module, a base, a bracket and a cantilever, the artificial intelligent processor is arranged on the display device or the camera module, and the distance between the display device and a user is adjustable;
the camera module is fixed on the cantilever, and the cantilever is used for adjusting the angle of the camera module;
the cantilever is fixed at one end of the bracket;
the other end of the bracket is fixedly connected with a base, and the base is used for realizing stable placement of the camera equipment;
the camera module is used for sending all the shot images of pages in the book of the user to the display equipment through the second wireless network module according to the frame sequence;
The display device is used for receiving each image through the first wireless network module;
The display device is also used for displaying various images to a user through the display screen;
The artificial intelligence processor is used for determining the position of the top end of the finger if the image is identified to contain the finger of the user;
the artificial intelligence processor is also used for determining a first retrieval target closest to the top of the finger in the image according to the position of the top of the finger, wherein the first retrieval target is a word or a picture;
the artificial intelligence processor is also used for obtaining a first search result corresponding to the first search target;
The artificial intelligence processor is further configured to display the first search result while displaying the first image to the user through the display device.
In one possible design, the artificial intelligence processor may determine a first search target closest to the finger tip in the first image based on the position of the finger tip by:
identifying the content of the page and the position of the content in the first image by adopting an artificial intelligence algorithm;
Determining target content closest to the top of the finger in the page according to the position of the top of the finger and the position of the content in the first image, wherein the target content is characters or pictures;
the first search target is determined based on the target content closest to the tip of the finger.
In one possible design, the target content is text, and the artificial intelligence processor determines the first search target based on the target content closest to the top of the finger by:
dividing target content into a plurality of word segments by adopting an artificial intelligence algorithm;
Acquiring the position of each word in a plurality of word segments;
determining the distance between each word segment and the top of the finger according to the position of each word segment and the position of the top of the finger;
Determining a word segmentation with the smallest distance with the top end of the finger as a first retrieval target;
The target content is a picture, and the first retrieval target is determined according to the target content closest to the top of the finger, including:
The target content is determined as a first retrieval target.
In one possible design, the artificial intelligence processor obtains a first search result corresponding to a first search target by:
Acquiring a first search result corresponding to a first search target based on a first association relation stored in a cloud server, wherein the cloud server stores a plurality of first association relations, and each first association relation is used for associating the search target, target content to which the search target belongs and a search result corresponding to the search target;
And/or searching the first search target by adopting a search engine to obtain a first search result.
In one possible design, the target content is text, and the artificial intelligence processor obtains a first search result corresponding to the first search target based on a first association relationship stored in the cloud server by:
Identifying target content of the first retrieval target by adopting an artificial intelligence algorithm;
The target content and the first retrieval target are used as first matching conditions, the first matching conditions are matched with each first association relation stored in the cloud server, whether the first association relations comprise the first target association relation matched with the first matching conditions or not is determined, and the first target association relation is used for associating the target content, the first retrieval target and the first retrieval result;
And if the plurality of first association relations comprise first target association relations, obtaining a first retrieval result in the first target association relations.
In one possible design, the artificial intelligence processor is further configured to display the corner mark control while displaying the first image to the user by:
traversing target contents of pages in the first image, and in the process of traversing each target content:
Sequentially matching the current target content with each first association relation as a second matching condition;
Determining whether each first association relation comprises a second target association relation matched with a second matching condition, wherein the second target association relation is used for associating the current target content, the retrieval target of the current target content and the retrieval result corresponding to the retrieval target of the current target content;
If each first association relation comprises a second target association relation, displaying the first image to the user through the display equipment and displaying a corner mark control, wherein the corner mark control is used for controlling whether to display a second search result, and the second search result is a search result corresponding to a search target of the current target content.
In one possible design, the artificial intelligence processor is further configured to enable sharing of search results among users by:
Acquiring a second image of a page in a book of a user shot by the camera equipment, wherein the shooting time of the second image is later than that of the first image;
If the part which is overlapped with the corner mark control part in the finger area of the user is identified from the second image, determining a second retrieval result of the target content association corresponding to the corner mark control part according to a second target association relationship corresponding to the corner mark control part;
and displaying the second search result to the user through the display device.
In one possible design, the artificial intelligence processor is further configured to stop displaying the first search result when the next frame of image of the first image is displayed via the display device if it is recognized that the next frame of image of the first image does not include a finger.
In one possible design, the display device includes a display screen, and the display device is further configured to, in response to an operation of sliding a finger of a user on the display screen, convert a track of the sliding of the finger of the user on the display screen into annotation text by using an artificial intelligence algorithm;
The display device is further used for storing second association relations, which are formed by the annotation text, the position of the annotation text in the first image, the account number of the user and the content of the preset position in the first image, to the cloud server.
Other implementation manners and effects of the above system embodiments are referred to the description in the auxiliary reading method embodiments, and are not repeated herein.
The basic principles of the present application have been described above in connection with specific embodiments, but it should be noted that the advantages, benefits, effects, etc. mentioned in the present application are merely examples and not intended to be limiting, and these advantages, benefits, effects, etc. are not to be construed as necessarily possessed by the various embodiments of the application. Furthermore, the specific details disclosed herein are for purposes of illustration and understanding only, and are not intended to be limiting, as the application is not necessarily limited to practice with the above described specific details.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein. Moreover, at least some of the steps in the flowcharts of the figures may include a plurality of sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order of their execution not necessarily being sequential, but may be performed in turn or alternately with other steps or at least a portion of the other steps or stages.
The block diagrams of the devices, apparatuses, devices, systems referred to in the present application are only illustrative examples and are not intended to require or imply that the connections, arrangements, configurations must be made in the manner shown in the block diagrams. As will be appreciated by one of skill in the art, the devices, apparatuses, devices, systems may be connected, arranged, configured in any manner. Words such as "including," "comprising," "having," and the like are words of openness and mean "including but not limited to," and are used interchangeably therewith. The terms "or" and "as used herein refer to and are used interchangeably with the term" and/or "unless the context clearly indicates otherwise. The term "such as" as used herein refers to, and is used interchangeably with, the phrase "such as, but not limited to.
It is also noted that in the apparatus, devices and methods of the present application, the components or steps may be disassembled and/or assembled. Such decomposition and/or recombination should be considered as equivalent aspects of the present application.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the application to the form disclosed herein. Although a number of example aspects and embodiments have been discussed above, a person of ordinary skill in the art will recognize certain variations, modifications, alterations, additions, and subcombinations thereof.
Claims (10)
1. A method of aiding in reading, the method comprising:
Acquiring a first image of a page in a book of a user, which is shot by shooting equipment;
displaying the first image to the user through a display device, wherein the distance between the display device and the user is adjustable;
If the first image is identified to contain the finger of the user, determining the position of the top end of the finger in the first image;
determining a first search target closest to the top end of the finger in the first image according to the position of the top end of the finger, wherein the first search target is a word or a picture;
Acquiring a first retrieval result corresponding to the first retrieval target;
And displaying the first image to the user through a display device and displaying the first search result.
2. The method of claim 1, wherein determining a first search target closest to the finger tip in the first image based on the position of the finger tip comprises:
Identifying the content of the page and the position of the content in the first image by adopting an artificial intelligence algorithm;
Determining target content closest to the top end of the finger in the page according to the position of the top end of the finger and the position of the content in the first image, wherein the target content is characters or pictures;
and determining a first retrieval target according to the target content closest to the top end of the finger.
3. The method of claim 2, wherein the target content is text, and wherein the determining the first search target based on the target content closest to the tip of the finger comprises:
dividing the target content into a plurality of word segments by adopting an artificial intelligence algorithm;
Acquiring the position of each word segment in a plurality of word segments;
determining the distance between each word segment and the top of the finger according to the position of each word segment and the position of the top of the finger;
Determining a word segmentation with the smallest distance with the top end of the finger as a first retrieval target;
The target content is a picture, and the determining a first search target according to the target content closest to the top of the finger comprises:
The target content is determined as a first retrieval target.
4. The method according to claim 2, wherein the obtaining the first search result corresponding to the first search target includes:
Acquiring a first search result corresponding to a first search target based on a first association relation stored in a cloud server, wherein the cloud server stores a plurality of first association relations, and each first association relation is used for associating the search target, target content to which the search target belongs and a search result corresponding to the search target;
and/or searching the first search target by adopting a search engine to obtain a first search result.
5. The method of claim 4, wherein the target content is text, and the obtaining a first search result corresponding to a first search target based on a first association stored in a cloud server includes:
Identifying target content to which the first retrieval target belongs by adopting an artificial intelligence algorithm;
the target content and the first retrieval target are used as first matching conditions and are matched with each first association relation stored in a cloud server, whether a plurality of first association relations comprise first target association relations matched with the first matching conditions is determined, and the first target association relations are used for associating the target content, the first retrieval target and a first retrieval result;
And if the plurality of first association relations comprise the first target association relation, obtaining the first retrieval result in the first target association relation.
6. The method according to claim 2, wherein the method further comprises:
Traversing the target content of the page in the first image, and in the process of traversing each target content:
Sequentially matching the current target content with each first association relation by taking the current target content as a second matching condition;
Determining whether each first association relation comprises a second target association relation matched with the second matching condition, wherein the second target association relation is used for associating the current target content, the retrieval target of the current target content and the retrieval result corresponding to the retrieval target of the current target content;
If each first association relation comprises the second target association relation, displaying the first image to the user through the display equipment and displaying a corner mark control, wherein the corner mark control is used for controlling whether to display a second search result, and the second search result is a search result corresponding to a search target of the current target content.
7. The method of claim 6, wherein the method further comprises:
Acquiring a second image of a page in a book of a user shot by the image pickup equipment, wherein the shooting time of the second image is later than that of the first image;
If the part which is overlapped with the corner mark control part in the finger area of the user is recognized from the second image, determining a second retrieval result of target content association corresponding to the corner mark control part according to a second target association relationship corresponding to the corner mark control part;
And displaying the second search result to the user through the display device.
8. The method according to claim 1 or 2, characterized in that the method further comprises:
And if the next frame image of the first image is identified to not contain fingers, stopping displaying the first search result when the next frame image is displayed through the display device.
9. The method of claim 1 or 2, wherein the display device comprises a display screen, the method further comprising:
Responding to the operation of sliding the finger of the user on the display screen, and converting the sliding track of the finger of the user on the display screen into annotation text by adopting an artificial intelligence algorithm;
And storing a second association relation formed by the annotation text, the position of the annotation text in the first image, the account number of the user and the content of the preset position in the first image to a cloud server.
10. The auxiliary reading system is characterized by comprising an artificial intelligent processor, display equipment and a camera shooting equipment, wherein the display equipment comprises a first wireless network module and a display screen, the camera shooting equipment comprises a second wireless network module, a camera shooting module, a base, a bracket and a cantilever, the artificial intelligent processor is arranged on the display equipment or the camera shooting module, and the distance between the display equipment and a user is adjustable;
The camera module is fixed on the cantilever, and the cantilever is used for adjusting the angle of the camera module;
the cantilever is fixed at one end of the bracket;
The other end of the bracket is fixedly connected with the base, and the base is used for realizing stable placement of the camera equipment;
The camera module is used for sending all the shot images of pages in the book of the user to the display equipment through the second wireless network module according to the frame sequence;
the display device is used for receiving each image through the first wireless network module;
the display device is further used for displaying each image to the user through the display screen;
The artificial intelligence processor is used for determining the position of the top end of the finger if the image is identified to contain the finger of the user;
The artificial intelligence processor is further used for determining a first retrieval target closest to the top end of the finger in the image according to the position of the top end of the finger, and the first retrieval target is a word or a picture;
the artificial intelligence processor is also used for acquiring a first retrieval result corresponding to the first retrieval target;
The artificial intelligence processor is further configured to display the first search result while displaying the first image to the user through a display device.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311812523 | 2023-12-26 | ||
| CN2023118125231 | 2023-12-26 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117908669A true CN117908669A (en) | 2024-04-19 |
Family
ID=90690283
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311823768.4A Pending CN117908669A (en) | 2023-12-26 | 2023-12-27 | Auxiliary reading method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117908669A (en) |
-
2023
- 2023-12-27 CN CN202311823768.4A patent/CN117908669A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109271945B (en) | Method and system for realizing job correction on line | |
| US10741167B2 (en) | Document mode processing for portable reading machine enabling document navigation | |
| US9626000B2 (en) | Image resizing for optical character recognition in portable reading machine | |
| US20230049533A1 (en) | Image gaze correction method, apparatus, electronic device, computer-readable storage medium, and computer program product | |
| US8320708B2 (en) | Tilt adjustment for optical character recognition in portable reading machine | |
| US8531494B2 (en) | Reducing processing latency in optical character recognition for portable reading machine | |
| US7505056B2 (en) | Mode processing in portable reading machine | |
| US7840033B2 (en) | Text stitching from multiple images | |
| US8150107B2 (en) | Gesture processing with low resolution images with high resolution processing for optical character recognition for a reading machine | |
| US8626512B2 (en) | Cooperative processing for portable reading machine | |
| US7325735B2 (en) | Directed reading mode for portable reading machine | |
| CN111444908A (en) | Image recognition method, device, terminal and storage medium | |
| US8186581B2 (en) | Device and method to assist user in conducting a transaction with a machine | |
| CN104575120B (en) | Display system for aided teaching | |
| WO2022156622A1 (en) | Sight correction method and apparatus for face image, device, computer-readable storage medium, and computer program product | |
| US20100201793A1 (en) | Portable reading device with mode processing | |
| US20060008122A1 (en) | Image evaluation for reading mode in a reading machine | |
| CN110163211B (en) | Image recognition method, device and storage medium | |
| WO2022089170A1 (en) | Caption area identification method and apparatus, and device and storage medium | |
| CN110012116A (en) | It is a kind of intelligent with system and device | |
| US20170103124A1 (en) | Remote search engine using camera | |
| CN114821620A (en) | Text content extraction and identification method based on longitudinal combination of line text boxes | |
| CN117908669A (en) | Auxiliary reading method and system | |
| CN116434253A (en) | Image processing method, device, equipment, storage medium and product | |
| TWM596951U (en) | Oracle Learning Book and System Using Augmented Reality Technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |