WO2021120420A1

WO2021120420A1 - Reading assistance method and apparatus, and electronic device

Info

Publication number: WO2021120420A1
Application number: PCT/CN2020/079181
Authority: WO
Inventors: 钟波; 肖适; 王鑫; 余金清
Original assignee: 成都极米科技股份有限公司
Priority date: 2019-12-16
Filing date: 2020-03-13
Publication date: 2021-06-24
Also published as: CN110990107A

Abstract

Provided in the present application are a reading assistance method and apparatus, and an electronic device. The method comprises: collecting first image data in a target region, the first image data comprising an indication action image of a target object, and reading content being displayed in the target region; recognizing the indication action image so as to determine a character string to be recognized that corresponds to the indication action image; performing designated processing on said character string so as to determine a target character string corresponding to the character string to be recognized; and displaying the target character string in the target region.

Description

Reading assistance method, device and electronic equipment

Technical field

This application relates to the field of image processing technology, and in particular, to a reading assistance method, device, and electronic equipment.

Background technique

The current reading mode is generally: 1) physical books carry reading content; 2) electronic devices display the content that needs to be read. In the above two reading methods, when encountering unfamiliar content, you can check related content through electronic devices such as mobile phones. As far as reading is concerned, the reading efficiency of the above two methods is relatively low.

Summary of the invention

In view of this, the purpose of the embodiments of the present application is to provide a reading assistance method, device, and electronic equipment. It can assist the user to process the reading, thereby improving the reading effect.

In the first aspect, an embodiment provides a reading assistance method, including:

Acquiring first image data in a target area, where the first image data includes an instruction action image of a target object, and reading content is displayed in the target area;

Recognizing the instruction action image to determine the character string to be recognized corresponding to the instruction action image;

Performing designation processing on the character string to be recognized to determine a target character string corresponding to the character string to be recognized;

The target character string is displayed in the target area.

In an optional implementation manner, the step of displaying the target character string in the target area includes:

Collecting second image data of the target area;

By identifying the second image data to determine whether the target area includes a blank area;

If the target area includes a blank area, the target character string is projected to the blank area of the target area for display.

The reading assistance method provided by the embodiment of the present application can also display the target character string in a blank area, which can prevent the target character string from blocking the content that the user may need to read, reduce the display effect of the reading, and affect the customer experience.

In an optional embodiment, the method further includes:

Collecting the third image data of the target area according to a preset period;

When the third image data is collected, identifying whether the third image data has updated content relative to the image data collected at the previous moment of the current moment;

If there is updated content, store the updated content.

The reading assistance method provided by the embodiment of the present application can also store the updated content when there is updated content, which can facilitate the user to subsequently query the content generated during the reading process.

In an optional implementation manner, if there is updated content, the step of storing the updated content includes:

If it is detected that the projected electronic reading is displayed in the target area, the updated content is stored in association with the electronic reading.

In the reading assistance method provided by the embodiments of the present application, the updated content is stored in association with the electronic reading, so that it is convenient for the user to query the updated content when viewing the electronic reading.

If it is detected that a physical reading object is placed in the target area, collecting fourth image data of the target area;

Determining the display surface corresponding to the physical reading object according to the fourth image data;

The target character string is projected to the display surface for display.

In the reading assistance method provided by the embodiments of the present application, since the physical reading object may not be a standard plane, if the target character string is displayed on the plane, it may cause the character string misalignment, etc., so that the display surface is determined first, and then based on the display surface. The target character string is displayed, so that the display effect can be more in line with the visual effect required by the human eye.

In an optional implementation manner, the step of performing designation processing on the character string to be recognized to determine a target character string corresponding to the character string to be recognized includes:

Translating the character string to be recognized to obtain the target character string of the character string to be recognized in the target language, or/and,

The interpretation document corresponding to the character string to be recognized is retrieved, and the interpretation document is used as the target character string corresponding to the character string to be recognized.

The reading assistance method provided by the embodiment of the present application can also translate or interpret the character string to be recognized, which can reduce the user's query operations during the reading process and improve the reading experience.

In an optional embodiment, the method further includes:

Projecting an electronic reading into the target area, and the character string to be identified is a character string in the electronic reading.

The reading assistance method provided by the embodiment of the present application can also directly project the electronic reading material that needs to be read, so that it is convenient for the user to read more content.

In a second aspect, an embodiment provides a reading aid device, including:

An acquisition module, configured to acquire first image data in a target area, the first image data including an instruction action image of the target object;

A recognition module, configured to recognize the instruction action image to determine the character string to be recognized corresponding to the instruction action image;

A processing module, configured to perform designated processing on the character string to be recognized to determine a target character string corresponding to the character string to be recognized;

The first projection module is configured to display the target character string in the target area.

In a third aspect, an embodiment provides an electronic device including: a processor and a memory, the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the machine-readable instructions are When the processor is executed, the steps of the method described in any of the foregoing embodiments are executed.

In a fourth aspect, an embodiment provides a computer-readable storage medium with a computer program stored on the computer-readable storage medium, and the computer program executes the steps of the method described in any of the foregoing embodiments when the computer program is run by a processor.

The reading assistance method, device, electronic equipment, and computer-readable storage medium provided by the embodiments of the present application can realize the specified processing of the character string to be recognized through instruction actions, which can facilitate the user to obtain the corresponding character string to be recognized when reading The target character string can improve the understanding of the character string to be recognized when reading.

In order to make the above-mentioned objectives, features and advantages of the present application more obvious and understandable, the following embodiments are specially cited in conjunction with the accompanying drawings, and detailed descriptions are made as follows.

Description of the drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application, and therefore do not It should be regarded as a limitation of the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative work.

FIG. 1 is a schematic block diagram of an electronic device provided by an embodiment of the application.

FIG. 2 is a flowchart of a reading assistance method provided by an embodiment of the application.

FIG. 3 is a detailed flowchart of step 204 of the reading assistance method provided by an embodiment of the application.

FIG. 4 is a detailed flowchart of step 204 of the reading assistance method provided by an embodiment of the application.

FIG. 5 is a partial flowchart of a reading assistance method provided by an embodiment of this application.

FIG. 6 is a schematic diagram of functional modules of a reading aid device provided by an embodiment of the application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

It should be noted that similar reference numerals and letters indicate similar items in the following figures. Therefore, once a certain item is defined in one figure, it does not need to be further defined and explained in subsequent figures. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

Example one

To facilitate the understanding of this embodiment, first, an electronic device that executes a reading assistance method disclosed in the embodiment of this application is introduced in detail.

As shown in Figure 1, it is a block diagram of an electronic device. The electronic device 100 may include a memory 111, a storage controller 112, a processor 113, a peripheral interface 114, an input and output unit 115, a collection unit 116, a projector 117, and a radio frequency unit 118. Those of ordinary skill in the art can understand that the structure shown in FIG. 1 is only for illustration, and does not limit the structure of the electronic device 100. For example, the electronic device 100 may also include more or fewer components than those shown in FIG. 1 or have a different configuration from that shown in FIG. 1.

The aforementioned components of the memory 111, the storage controller 112, the processor 113, the peripheral interface 114, the input output unit 115, and the collection unit 116 are directly or indirectly electrically connected to each other to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The aforementioned processor 113 is used to execute executable modules stored in the memory.

The memory 111 may be, but is not limited to, random access memory (Random Access Memory, RAM for short), Read Only Memory (ROM for short), Programmable Read-Only Memory (PROM for short) ), Erasable Programmable Read-Only Memory (EPROM), Electrical Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 111 is used to store a program, and the processor 113 executes the program after receiving an execution instruction. The method executed by the electronic device 100 of the process definition disclosed in any embodiment of the present application can be applied to processing In the processor 113, or implemented by the processor 113.

The aforementioned processor 113 may be an integrated circuit chip with signal processing capability. The aforementioned processor 113 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; it may also be a digital signal processor (DSP for short). ), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.

The aforementioned peripheral interface 114 couples various input/output devices to the processor 113 and the memory 111. In some embodiments, the peripheral interface 114, the processor 113, and the storage controller 112 may be implemented in a single chip. In some other instances, they can be implemented by independent chips.

The aforementioned input and output unit 115 is used to provide input data to the user. The input and output unit 115 may be, but is not limited to, a mouse, a keyboard, and the like.

The aforementioned acquisition unit 116 is used to capture images (for example, photos, videos, etc.), and to store the captured images for use by other components. Optionally, the acquisition unit 116 may be an RGB-D (Red Green Blue-Deep) camera. The acquisition unit 116 can be used to capture depth images.

Optionally, the electronic device 100 in this embodiment may further include a projector 117, and the projector 117 includes a light source, a projection optical system, and other projection elements, which are used to realize image projection.

A radio frequency (RF) unit 118 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and communicate with communication networks or other devices. The radio frequency unit 118 may include various existing circuit elements for performing these functions, for example, an antenna, a radio frequency transceiver, a digital signal processor, an encryption/decryption chip, a subscriber identity module (SIM) card, a memory, and so on. The radio frequency unit 118 can communicate with various networks such as the Internet, an intranet, and a wireless network, or communicate with other devices through a wireless network. The aforementioned wireless network may include a cellular telephone network, a wireless local area network, or a metropolitan area network. The above-mentioned wireless network can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment, EDGE, and broadband code Wideband code division multiple access (W-CDMA), code division multiple access (CDMA), time division multiple access (TDMA), wireless fidelity technology (Wireless, Fidelity) , WiFi) (such as the American Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and/or IEEE 802.11n), Internet telephony (Voice over internet protocol, VoIP), Worldwide Interoperability for Microwave Access, Wi-Max), other protocols used for mail, instant messaging and short messages, and any other appropriate communication protocols, even those that have not yet been developed. In this embodiment, the above-mentioned radio frequency unit 118 can realize the communication between the electronic device 100 and external devices.

The electronic device 100 in this embodiment can be used to execute each step in each method provided in the embodiment of the present application. The implementation process of the reading assistance method will be described in detail below through several embodiments.

Example two

Please refer to FIG. 2, which is a flowchart of a reading assistance method provided by an embodiment of the present application. The specific process shown in FIG. 2 will be described in detail below.

Step 201: Collect first image data in the target area.

Wherein, the first image data includes an instruction action image of the target object. The above-mentioned target object can be any object that can point to the content in the target area. Illustratively, the target object may be a pen, a baton, a user's finger, and so on. The indicating action image can be that the target object touches a character string; it can also be that the target object is located below a character string.

In this embodiment, the reading content is displayed in the target area. Exemplarily, the reading content may be the content of the electronic reading displayed by projection, or the content printed in the physical reading placed in the target area.

Optionally, the aforementioned target area may be a desktop. Physical reading objects can be placed on the desktop. Exemplarily, the entity reading material may be a novel book, a foreign document, learning materials, and the like.

Optionally, electronic reading materials may be displayed in the above-mentioned target area. The content represented by the electronic book may be learning content, novel fragments, and so on. Optionally, the target area carrying the electronic reading can be any interface that can be used to display the projection screen, such as a solid-color wall surface, a solid-color white paper, or the like.

In one embodiment, if a projection is displayed in the target area, the method in this embodiment further includes: projecting the electronic reading into the target area.

Exemplarily, the first image data may include a character string in the electronic book.

Step 202: Recognizing the indicating action image to determine the character string to be recognized corresponding to the indicating action image.

In an alternative embodiment, step 202 may include: recognizing the location of the target object to determine the target location pointed to by the target object; performing text recognition on the content around the target location to extract the character string to be recognized.

Exemplarily, the following describes the detection of the position of the target object by taking the target object as the user's finger as an example.

Optionally, the indicating action image can be detected by edge detection to determine the edge of the user's finger. Exemplarily, the determined specified orientation of the edge of the user's finger may be used as the target position. For example, the detected upper edge can be used as the target position.

Optionally, it is also possible to classify the content in the indicating action image by using a classification model implemented by a neural network, so as to filter out the user's finger in the indicating action image and the area where the user's finger is located. Exemplarily, the determined designated position of the area where the user's finger is located may be used as the target position. For example, the upper left edge of the area where the user's finger is located can be used as the target position.

Optionally, performing text recognition on the content around the target location to extract the character string to be recognized may be implemented as: recognizing the content around the target location using a neural network model to extract the character string to be recognized.

Optionally, performing text recognition on the content around the target location to extract the character string to be recognized can be implemented as: using an OCR (Optical Character Recognition, Chinese name: optical character recognition) model to perform text recognition on the content around the target location Recognition to extract the string to be recognized.

Optionally, text recognition can be performed on the area around the target location that is not covered by the target object. Optionally, only the line of character strings closest to the target location can be identified. Optionally, it is also possible to perform text recognition on only the word or sentence closest to the target location.

In one embodiment, if an electronic reading can be displayed in the target area, the above-mentioned character string to be identified is the character string in the electronic reading.

In one embodiment, if a physical reading object is placed in the target area, the above-mentioned character string to be identified is a character string extracted from the physical reading object.

Step 203: Perform designation processing on the character string to be recognized to determine a target character string corresponding to the character string to be recognized.

In an embodiment, step 203 may include: translating the character string to be recognized to obtain a target character string of the character string to be recognized in a target language.

Exemplarily, the character string to be recognized may be a character in the first language, and the target character string may be a character in the second language. The above-mentioned designation process may be to translate the character string to be recognized. Optionally, the character string to be recognized may be characters in languages such as English, French, and Italian. The target string can be Chinese characters. For example, the aforementioned character string to be identified can be "patent", and the target character string can be "patent". It can be known that the language types corresponding to the above-mentioned character string to be recognized and the target character string are merely exemplary, and the embodiment of the present application does not limit the language types corresponding to the recognized character string and the target character string.

Optionally, there may be a translation application in the electronic device, and a language database may also be stored. Exemplarily, in a non-network environment, translating the character string to be recognized to obtain the target character string of the character string to be recognized in the target language may include: translating the character string to be recognized through the translation application. The recognized character string is translated offline to obtain the target character string of the character string to be recognized in the target language. Exemplarily, if the electronic device is in a networked state, the above-mentioned translating the character string to be recognized to obtain the target character string of the character string to be recognized in the target language may include: translating the character string to the target language through the translation application. The character string to be recognized is translated online to obtain the target character string of the character string to be recognized in the target language.

In another embodiment, step 203 may include: searching for the interpretation document corresponding to the character string to be recognized, and using the interpretation document as the target character string corresponding to the character string to be recognized.

Exemplarily, the character string to be recognized may be an abbreviation of a proper noun, and the target character string may be the full name corresponding to the abbreviation. For example, the character string to be identified may be "IP", and the corresponding target character string may be "Internet Protocol (IP)". Illustratively, if one abbreviation corresponds to multiple full names, all full names can be displayed. For example, the string to be identified can be "CNN", and the corresponding target string can be "Cable News Network (CNN)" or "Convolutional Neural Networks (CNN)" Wait.

Exemplarily, the character string to be recognized may be an idiom, and the target character string may be the meaning, allusion, or source corresponding to the idiom. For example, the character string to be recognized can be "sounds like a bell", and the corresponding target character string can be "speaking or singing loudly, like striking a bell". Optionally, the target string may also include the source of the idiom. For example, the target string in the above example can also include "Ming Feng Menglong's "Eastern Zhou Dynasty" Chapter 72: Recalling Xu's eyes is like lightning, and the sound is like a bell."

Exemplarily, the character string to be recognized may be a professional word, and the target character string may be the corresponding interpretation of the idiom. For example, the character string to be identified can be the "leverage principle" in the field of physics, and the target string can be an explanation of the lever principle. Condition". To balance the lever, the two moments (the product of the force and the arm) acting on the lever must be equal in size."

Optionally, a search engine may be installed in the electronic device. Exemplarily, if the electronic device is in a networked state, the aforementioned retrieval of the interpretation document corresponding to the character string to be identified, and using the interpretation document as the target character string corresponding to the character string to be identified may include: using a search engine to retrieve For the interpretation document corresponding to the character string to be recognized, the interpretation document is used as the target character string corresponding to the character string to be recognized.

Optionally, a local database may also be stored in the electronic device. Exemplarily, searching for the interpretation document corresponding to the character string to be recognized, and using the interpretation document as the target character string corresponding to the character string to be recognized may include: querying a local database corresponding to the character string to be recognized Interpretation document, the interpretive document is used as the target character string corresponding to the character string to be identified.

Step 204: Display the target character string in the target area.

Optionally, the target character string may be displayed in an area that does not obscure the electronic reading displayed in the target area, or the text part of the physical reading. Optionally, the target character string can also be displayed in the read area, for example, the target character string can be displayed above the position of the character string to be recognized.

Optionally, the target character string can be projected to the target area for display by means of projection. Optionally, the target area may be an electronic display, and the target character string may be displayed on the electronic display.

Optionally, as shown in FIG. 3, step 204 may include the following steps.

Step 2041: Collect second image data of the target area.

Step 2042: Determine whether the target area includes a blank area by identifying the second image data.

Exemplarily, the aforementioned blank area may indicate an area in the target area where no characters or images are mapped.

Optionally, the blank area of the target area can be determined by recognizing the color of each pixel in the second image data. Illustratively, the color of the target area may be a designated color. The designated color can be white, green, black, etc. For example, if the color value is determined to be the specified color in the pixel point corresponding to the specified color, and all pixels in the area formed by the specified number of pixels are determined to be the specified color, it can be determined that the specified number of pixels are formed The area is the designated color area, that is, it can be judged as a blank area.

Step 2043: If the target area includes a blank area, project the target character string to the blank area of the target area for display.

Step 2044: If the target area does not include a blank area, project the target character string to the position of the character string in the previous designated line of the character string to be recognized in the target area.

Optionally, the projection optical system of the projector of the electronic device can be adjusted to adjust the projection angle of the projected target character string, so as to realize the projecting of the target character string to the blank area of the target area or the character to be recognized The position of the specified line string before the string is displayed.

Through the above-mentioned display mode, it is possible to prevent the displayed target character string from obscuring the area that the user needs to read.

Optionally, the target character string can be matched with the surface carrying the display target character string. For example, when the surface bearing the display target character string is a flat surface, the target character string can be displayed on the flat surface according to the display standard of the flat surface. For another example, when the surface bearing the display target character string is a curved surface, the target character string can be mapped and displayed on the curved surface according to the shape of the curved surface.

Optionally, if a physical reading object is placed in the target area, when the physical reading object is open, it may cause the surface of the physical reading object that can be used to project the screen to be non-planar. On this basis, the projection can be The image of is corrected according to the non-plane presented by the entity reading object. As shown in FIG. 4, step 204 may include the following steps.

Step 2045: If it is detected that a physical reading object is placed in the target area, collect fourth image data of the target area.

Optionally, the fourth image data may be a depth image. Each pixel in the fourth image data represents the distance between the collection device and the object in the fourth image data.

Optionally, a depth image of the target area can be collected to determine whether a physical reading object is placed in the target area. For example, the collecting unit may collect a target depth image when any plane in the target area containing the character string to be read is parallel, and judge whether the pixel value of each pixel corresponding to the target area in the depth image is equal. If the values are not all equal, it means that there is a physical reading material in the target area; if the pixel value of each pixel is equal, it means that there is no object in the target area, and the target area may have an electronic reading material projected.

Step 2046: Determine the display surface corresponding to the physical reading object according to the fourth image data.

Optionally, the location of the physical reading object and the plane formed by the surface of the physical reading object are determined by the pixels of each pixel.

Step 2047: Project the target character string to the display surface for display.

Exemplarily, the above-mentioned display surface can be subdivided into grids, the pre-distortion matrix of the projector can be solved, and geometric correction can be performed in real time.

Exemplarily, in the corresponding display surface, for each triangle mesh, use its own three vertices and at least two adjacent nodes to obtain the corresponding pre-distortion matrix M using the least square method, and finally use the The screen-like method pre-distorted the corresponding area in the buffer before projection.

Optionally, as shown in FIG. 5, the reading assistance method in this embodiment may further include the following steps.

Step 205: Collect third image data of the target area according to a preset period.

Step 206: When the third image data is collected, identify whether the third image data has updated content relative to the image data collected at the previous moment of the current moment.

In one embodiment, the third image data can be compared with the image data collected at the previous moment of the current moment by pixel to determine whether there is a difference between the third image data and the image data collected at the previous moment of the current moment. If there is a difference, it can be determined that there is updated content in the third image data.

In another embodiment, the first image feature of the third image data can be extracted; the second image feature of the image data collected at the previous moment of the current time can be extracted; and then the first image feature and the second image feature can be calculated. If the Euclidean distance is greater than the preset value, it can be determined that there is updated content in the third image data.

Exemplarily, if the third image data is exactly the same as the image data collected at the moment before the current moment, the calculated Euclidean distance is zero. However, since the images collected at different times may have light errors, even if two images are exactly the same, the images collected at different times may be different. Therefore, the aforementioned preset value may be a positive number greater than zero. The size of the preset value can be set according to requirements, and the embodiment of the present application is not limited to the size of the preset value.

Exemplarily, the updated content may be notes made in the target area. Exemplarily, the aforementioned update content may not include the target object mentioned in step 201.

Step 207: If there is updated content, store the updated content.

Optionally, step 207 may include: if it is detected that the projected electronic reading is displayed in the target area, then storing the updated content in association with the electronic reading.

Optionally, the generation time of the updated content can also be stored in association with the updated content.

Optionally, step 207 can also be implemented as: if there is updated content and the specified action is detected in the third image data, the updated content is stored.

Optionally, the designated action may be directed to update content.

Optionally, the designated action may be directed to the time to update the content for a designated period of time. Exemplarily, the third image data may include multiple pictures, and when the specified number of images continuously collected all include the target object pointing to the updated content, it can be determined that the time pointing to the updated content lasts for the specified period of time. Exemplarily, the third image data may be a video, and when the specified duration of the video includes the target object pointing to the updated content, it can be determined that the time pointing to the updated content lasts for the specified duration.

By digitizing the paper-based update content and storing the electronic update content, it is convenient for users to inquire about the content generated during the reading process.

The detailed process of the method in this embodiment is described below through two actual application scenarios.

In an example, the method provided in the embodiment of the present application can be used for teaching; the target area can be a blackboard or whiteboard for teaching. Teaching materials are currently projected in the target area. When the teaching stick points to any content, the annotation corresponding to the content can be displayed. For example, an English text is displayed in the current target area, and when the teaching stick points to an English word, the Chinese meaning corresponding to the English word can be displayed on the blank area of the blackboard or whiteboard. Further, if the teacher writes some teaching notes on the blackboard, the teaching notes can be stored in association with the teaching materials currently displayed on the projection.

In another embodiment, the method provided by the embodiment of this application can be used for personal reading; a book is placed in the target area, and when the user’s finger points to some words and sentences in the book, it can be retrieved from various search websites Query the corresponding meaning of the word and sentence, and display the found content in the blank area of the book. Further, if the user writes some notes in a blank area of the book, the notes can be saved to the storage space pointed to by the designated account. Exemplarily, the designated account may be an account of a cloud storage space.

Through the above method in this embodiment, the specified processing of the character string to be recognized can be realized through the instruction action, which can facilitate the user to obtain the target character string corresponding to the character string to be recognized when reading, and can improve the assisting understanding of the character string to be recognized when reading. String. Further, the new information generated during the reading or teaching process can also be stored, which can be the user's ability to view useful information generated during the reading or teaching at any time in the future.

Example three

Based on the same application concept, the embodiment of the application also provides a reading assistance device corresponding to the reading assistance method. Since the device in the embodiment of the application solves the problem in principle similar to the foregoing embodiment of the reading assistance method, in this embodiment The implementation of the device can refer to the description in the embodiment of the above method, and the repetitive parts will not be repeated.

Please refer to FIG. 6, which is a schematic diagram of functional modules of a reading aid device provided by an embodiment of the present application. Each module in the reading aid device in this embodiment is used to execute each step in the foregoing method embodiment. The reading aid device includes: an acquisition module 301, an identification module 302, a processing module 303, and a first projection module 304; among them,

The collection module 301 is configured to collect first image data in a target area, where the first image data includes an instruction action image of a target object, and the reading content is displayed in the target area;

The recognition module 302 is configured to recognize the instruction action image to determine the character string to be recognized corresponding to the instruction action image;

The processing module 303 is configured to perform designated processing on the character string to be recognized to determine a target character string corresponding to the character string to be recognized;

The first projection module 304 is configured to display the target character string in the target area.

In a possible implementation manner, the first projection module 304 is configured to:

Collecting second image data of the target area;

In a possible implementation manner, the reading aid device may further include a storage module 305 for:

If there is updated content, store the updated content.

In a possible implementation manner, the storage module 305 is also used for:

The target character string is projected to the display surface for display.

In a possible implementation manner, the processing module 303 is configured to:

In a possible implementation manner, the reading aid device may further include:

The second projection module 306 is configured to project an electronic reading into the target area, and the character string to be recognized is a character string in the electronic reading.

In addition, the embodiments of the present application also provide a computer-readable storage medium having a computer program stored on the computer-readable storage medium, and the computer program executes the steps of the reading assistance method described in the above method embodiment when the computer program is run by a processor. .

The computer program product of the reading assistance method provided by the embodiment of the present application includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the reading assistance method described in the above method embodiment For details, please refer to the above method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed device and method may also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions, and functions of the devices, methods, and computer program products according to multiple embodiments of the present application. operating. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of the code, and the module, program segment, or part of the code contains one or more functions for realizing the specified logical function. Executable instructions. It should also be noted that in some alternative implementations, the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, or they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions Or it can be realized by a combination of dedicated hardware and computer instructions.

In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part.

If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks or optical disks and other media that can store program codes. . It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply one of these entities or operations. There is any such actual relationship or order between. Moreover, the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes those that are not explicitly listed Other elements of, or also include elements inherent to this process, method, article or equipment. If there are no more restrictions, the element defined by the sentence "including..." does not exclude the existence of other same elements in the process, method, article or equipment that includes the element.

The above descriptions are only preferred embodiments of the application, and are not intended to limit the application. For those skilled in the art, the application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included in the protection scope of this application. It should be noted that similar reference numerals and letters indicate similar items in the following figures. Therefore, once a certain item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Should be covered within the scope of protection of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

A reading assistance method, characterized in that it comprises:

Acquiring first image data in a target area, where the first image data includes an instruction action image of a target object, and reading content is displayed in the target area;

Recognizing the instruction action image to determine the character string to be recognized corresponding to the instruction action image;

Performing designation processing on the character string to be recognized to determine a target character string corresponding to the character string to be recognized;

The target character string is displayed in the target area.
The method according to claim 1, wherein the step of displaying the target character string in the target area comprises:

Collecting second image data of the target area;

By identifying the second image data to determine whether the target area includes a blank area;

If the target area includes a blank area, the target character string is projected to the blank area of the target area for display.
The method according to claim 1, wherein the method further comprises:

Collecting the third image data of the target area according to a preset period;

When the third image data is collected, identifying whether the third image data has updated content relative to the image data collected at the previous moment of the current moment;

If there is updated content, store the updated content.
The method according to claim 3, characterized in that, if there is updated content, the step of storing the updated content comprises:

If it is detected that the projected electronic reading is displayed in the target area, the updated content is stored in association with the electronic reading.
The method according to claim 1, wherein the step of displaying the target character string in the target area comprises:

If it is detected that a physical reading object is placed in the target area, collecting fourth image data of the target area;

Determining the display surface corresponding to the physical reading object according to the fourth image data;

The target character string is projected to the display surface for display.
The method according to claim 1, wherein the step of performing designation processing on the character string to be recognized to determine a target character string corresponding to the character string to be recognized comprises:

Translating the character string to be recognized to obtain the target character string of the character string to be recognized in the target language, or/and,

The interpretation document corresponding to the character string to be recognized is retrieved, and the interpretation document is used as the target character string corresponding to the character string to be recognized.
The method according to claim 1, wherein the method further comprises:

Projecting an electronic reading into the target area, and the character string to be identified is a character string in the electronic reading.
A reading aid device, characterized in that it comprises:

An acquisition module, configured to acquire first image data in a target area, where the first image data includes an instruction action image of a target object, and reading content is displayed in the target area;

A recognition module, configured to recognize the instruction action image to determine the character string to be recognized corresponding to the instruction action image;

A processing module, configured to perform designated processing on the character string to be recognized to determine a target character string corresponding to the character string to be recognized;

The first projection module is configured to display the target character string in the target area.
An electronic device, comprising: a processor and a memory, the memory storing machine-readable instructions executable by the processor, and when the electronic device is running, the machine-readable instructions are used by the processor When executed, the steps of the method according to any one of claims 1 to 7 are executed.
A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the method according to any one of claims 1 to 7 when the computer program is run by a processor.