WO2021218432A1

WO2021218432A1 - Method and apparatus for interpreting picture book, electronic device and smart robot

Info

Publication number: WO2021218432A1
Application number: PCT/CN2021/080269
Authority: WO
Inventors: 周琦; 张冲; 吴鹤松
Original assignee: 华为技术有限公司
Priority date: 2020-04-30
Filing date: 2021-03-11
Publication date: 2021-11-04
Also published as: CN111613100A

Abstract

The present application provides a method and an apparatus for interpreting a picture book, an electronic device, and a smart robot, relating to the field of picture book interpretation technology. Said method comprises: determining the current picture book being read by a reader; and acquiring historical information of the current picture book, and raising a first round of questions with the reader. In the present application, after a picture book being read by a reader is determined, the number of previous interpretations by the reader on the current picture book and a positive answer rate of each round of questions raised during each interpretation are acquired from a resource library, and then a round of questions, matching an understanding degree of the reader, are raised with the reader according to the number of interpretations and the average value of a plurality of positive answer rates of each round of questions, thereby preventing the first round of question raised with the reader during an interpretation of a picture book from going beyond the reader's comprehension ability or being too simple.

Description

A method, device, electronic equipment and intelligent robot for interpreting picture books

This application requires the priority of a Chinese patent application filed with the State Intellectual Property Office of China, the application number is 202010365911.X, and the application title is "A method, device, electronic equipment and intelligent robot for interpreting picture books" on April 30, 2020. The entire content is incorporated into this application by reference.

Technical field

The present invention relates to the technical field of picture book interpretation, and in particular to a method, device, electronic equipment and intelligent robot for picture book interpretation.

Background technique

Listening to stories is an activity that all children love when they grow up. Children also hope that their parents will accompany themselves by their side and do parent-child reading together. Nowadays, most parents are unable to tell stories to their children due to busy work and leaving their children to the care of the elderly. Therefore, there are many devices on the market for reading picture books for children, such as point reading machines and picture book reading robots. However, these devices can only play audio simply, and cannot interact with children in question and answer, guide children to think and inspire the imagination of readers.

Summary of the invention

In order for the terminal device in the prior art to be unable to interact with children in question and answer and guide children to think, the embodiments of the present application provide a method for interpreting picture books, device electronic equipment and intelligent robots.

In order to achieve the foregoing objectives, the embodiments of the present application adopt the following technical solutions:

In the first aspect, the present application provides a method for interpreting picture books, which is executed by a terminal device, and the method includes: determining the current picture book being read by the reader; acquiring historical information of the current picture book, the historical information including the historical information The number of readers’ interpretations of the current picture book and the positive answer rate of each round of questions asked by the readers in each interpretation in history; the reader is asked the first round of questions, the The first round of questions is determined based on the historical information. Among them, each round of questions refers to: divide the multiple questions associated with the picture book into N sets according to certain rules, and then sort them in a certain order, and then put forward the next set after the questions in the previous set are asked The problem, which constitutes a round-by-round problem.

In the above invention, after the picture book being read by the reader is determined, the number of interpretations of the current picture book by the reader before that and the positive answer rate of each round of questions raised during each interpretation are obtained from the resource library. , And then according to the number of interpretations and the average value of multiple positive answer rates for each round of questions, the reader is asked the round question that meets the level of understanding, so as to avoid the first time that the reader is asked in the process of interpreting the picture book. A round question is beyond the scope of the reader’s comprehension or is too simple.

In another possible implementation, the method further includes: receiving the reader's answer to each question in the first round of questions; when the positive answer rate of the answer to each question is greater than a set threshold When, ask the reader a second round of questions, the questions in the second round of questions are different from the questions in the first round of questions

In the above invention, after the first round of questions is asked, it is determined whether to ask the reader the more difficult first-level round of questions based on the positive answer rate of the reader's reply, so as to guide the reader to the story step by step To improve the reader’s reading comprehension ability.

In another possible implementation, the method further includes: the sequence of the first round of questions and the second round of questions is set according to the degree of difficulty.

In another possible implementation, before the acquiring the historical information of the current picture book, it further includes: acquiring the identity information of the reader, and the acquiring the historical information of the current picture book includes: acquiring the historical information of the current picture book. The historical information of the current picture book corresponding to the identity information.

In the above-mentioned invention, by identifying the identity information of the reader and referring to the historical information related to the reader, it is possible to more accurately ask the reader a round question matching the degree of understanding of the reader.

In another possible implementation, a data package of at least one picture book is stored in the resource library, the data package including the name of the picture book, the page number of the picture book, the content corresponding to the page number, and at least one round corresponding to the content Questions and answers to each question in the at least one round of questions, the at least one picture book includes the current picture book being read by the reader, and the at least one round question includes the first round question and the first round question Second round question.

In the above-mentioned invention, the picture book designer enters the text prepared by each picture book into the memory in advance, so that the terminal device can still interpret the picture book for the reader even when the terminal device is not connected to the Internet.

In another possible implementation, after the determining the current picture book that the reader is reading, the method further includes: determining whether the current picture book is stored in a resource library; when the resource library does not have the current picture book , To determine whether the content of the current picture book is the same as the content of the first picture book stored in the resource library; when the content of the current picture book is the same as the content of the first picture book, suggest to the reader with The round question corresponding to the first picture book.

In the above invention, due to the huge number of picture books on the market, some picture books have the same content but different forms due to different publishers and different layouts. Therefore, the picture book designer does not need to record all the picture books with the same content into the memory. , Only need to make a picture book information according to the content of the picture book, and then enter it into the memory, which not only reduces the workload of the picture book designer, but also reduces the storage space of the memory.

In another possible implementation, the method further includes: determining the page number of the current picture book that the reader is reading; and asking the reader a question about the turn of the content corresponding to the page number.

In the above invention, due to the relatively large content of picture books, it is impossible for readers to read a picture book in one reading at a time. Therefore, the problem of interpretation of picture books can be divided into N parts according to the page number of the picture book. After one page of content, ask the reader questions to better assist the reader in interpreting the picture book.

In another possible implementation, the determining the current picture book being read by the reader includes: acquiring the voice information of the current picture book being read by the reader through a microphone; converting the voice information into picture book text, so The picture book text is used to determine the current picture book being read by the reader from the resource library. In another possible implementation, the determining the current picture book being read by the reader includes: acquiring the image of the current picture book being read by the reader through a camera; identifying features in the image to obtain the picture book feature value, The picture book feature value is used to determine the current picture book being read by the reader from the resource library. In another possible implementation, before it is determined that the positive answer rate of the reader's answer to the first round of questions is greater than a set threshold, the method further includes: obtaining voice information of the reader's answer to the question through a microphone; The voice information is converted into answer text, and the answer text is used to determine from the resource database whether the reader’s answer to the first question in the first round of questions is correct, and the first round The problem includes the first problem.

In another possible implementation, before determining that the positive answer rate of the reader's answer to the first round of questions is greater than a set threshold, the method includes: acquiring the reader's actions or gestures through a camera; and identifying the reader The feature in the image of the action or gesture is obtained, and the feature value of the answer is used to determine from the resource database whether the reader’s answer to the second question in the first round of questions is correct, so The first round question includes the second question.

In another possible implementation, the method further includes: adding, in the historical information, the number of interpretations of the reader's interpretation of the current picture book and storing the reader's replies during the interpretation process. The positive answer rate of the answers to the questions in each round.

In another possible implementation, before the first round of questions is asked to the reader, the method includes: receiving an interpretation instruction, where the interpretation instruction is used to instruct to ask the reader the first round of questions .

In the above-mentioned invention, after the terminal device determines the picture book that the reader is currently reading, sometimes the reader does not want to interpret the part of the content, or interpret it after a period of time, so the terminal device is controlled by instructions to interpret it. Or when to interpret to meet the needs of readers.

In the second aspect, an embodiment of the present application provides a picture book interpretation device, including: a transceiver, a processor, and a memory; the transceiver is used to receive and send data; the memory stores one or more programs, the one The or multiple programs include instructions, and when the instructions are executed by the processor, the electronic device executes each possible implementation solution in the first aspect.

In a third aspect, an embodiment of the present application provides an electronic device, including: a camera and/or a microphone, a memory, and a processor that executes each possible implementation of the first aspect.

In a fourth aspect, an embodiment of the present application provides an intelligent robot, including: a camera and/or a microphone, for receiving voice information or image information of a reader reading a picture book, and the reader’s answer to a question corresponding to the picture book Voice information or image information; memory for storing the second information of at least one picture book, as well as the number of interpretations of the current picture book by the reader in the history of each picture book and each time the reader in the history The positive answer rate of each round of questions raised during interpretation; a processor for processing the voice information or image information obtained by the camera and/or the microphone, and then determining the reader from the memory The current picture book being read, and based on the number of interpretations of the current picture book in the memory and the average value of the positive answer rate obtained at each interpretation of each round of questions, determine the corresponding round of questions to the reader The speaker is used to play the voice to the reader; the communication unit is used to receive the second information of each picture book, and the threshold of the positive answer rate from the current round to the next round.

In the fifth aspect, the embodiments of the present application provide a readable storage medium for storing instructions. When the instructions are executed, each possible implementation of the first aspect is realized.

In the sixth aspect, the embodiments of the present application provide a computer program device containing instructions, which when running on a terminal, enables each possible implementation of the first aspect to be implemented.

Description of the drawings

The following briefly introduces the drawings needed in the description of the embodiments or the prior art.

FIG. 1 is an architecture diagram of an application system for interpreting picture books provided by an embodiment of the application;

FIG. 2 is a schematic structural diagram of a terminal device provided by an embodiment of this application;

FIG. 3 is a schematic diagram of different gestures representing different numbers provided by an embodiment of the application;

FIG. 4 is a schematic diagram of a template for making a picture book provided by an embodiment of the application;

FIG. 5 is a schematic diagram of a template for making a picture book provided by an embodiment of the application;

Fig. 6 is a schematic diagram of a template for making a picture book provided by an embodiment of the application;

FIG. 7 is a flowchart of an interpretation method provided by an embodiment of this application;

FIG. 8 is a schematic diagram of an image displayed on a touch screen according to an embodiment of the application;

FIG. 9 is a schematic structural diagram of a terminal device provided by an embodiment of the application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

FIG. 1 is an architecture diagram of an application system for interpreting picture books provided by an embodiment of the application. As shown in Figure 1, the system architecture includes picture books, terminal equipment and readers.

The picture book can be a paper book, or a tablet, kindle, and other devices that can be used by readers to read stories.

In the embodiments of the present application, terminal devices include, but are not limited to, smart devices such as tablet computers and smart phones. For example, it can also include intelligent robots independently developed for various specific business scenarios, and so on. In the terminal device, each story in the picture book, as well as a series of questions corresponding to each story and an answer corresponding to each question are stored. Among them, a series of questions corresponding to each story is divided into several rounds according to the degree of difficulty, and the terminal device asks readers questions of different rounds in the order of easy first and then difficult. When the terminal device assists the reader in the process of interpreting the picture book, after determining the story that the reader is reading, find out the question and answer corresponding to the story from the database; then, after the reader finishes reading the story, interpret the picture book according to the readers in history The number of times and the positive answer rate in each round, to ask the readers the corresponding difficult round questions to avoid the first round of questions asked to the reader in the process of interpreting a picture book that exceeds the reader’s comprehension ability or is too simple . In addition, after a round of questions are asked, according to the correct rate of the answers answered by the reader, it is determined whether to ask questions that are more difficult for the first round, so as to gradually guide the reader’s understanding of the story and improve the reader Reading comprehension skills.

Fig. 2 is a schematic structural diagram of a terminal device provided by an embodiment of the application. As shown in FIG. 2, the terminal device includes an input unit 1, an output unit 2, a processing unit 3, a storage unit 4, and an input unit 5.

The terminal device needs to have the ability to perceive the current user's identity, such as identifying the current user's identity through face recognition, voiceprint recognition, user account passwords, etc., based on user input or through smarter methods (such as identifying user gender based on face) , Age) to obtain the current user's personal information. All personal data during the use of the user will be stored and recorded under the user account.

The input unit 1 includes a microphone 11 and a camera 12. Among them, the microphone 11 is used to collect voice information, and the camera 12 is used to collect image information.

In the embodiment of the present application, for the function of the microphone 11, the terminal device can obtain the voice information of the reader reading the story in the picture book through the microphone 11, so as to determine the story read by the reader. The terminal device may also obtain the voice information of the reader's answer to the question raised by the terminal device through the microphone 11, so as to obtain the reader's answer to the question.

For the function of the camera 12, the terminal device can obtain the page of the picture book that the reader is reading through the camera 12, and identify the story read by the reader according to the content of the page. The terminal device can also acquire the reader's motion or gesture through the camera 12, so as to recognize the reader's answer to the question based on the reader's motion or gesture.

The output unit 2 includes a speaker 21. In the embodiment of the present application, the terminal device can play through the speaker 21 recordings of other people reading the story that the reader is reading, broadcast the question corresponding to the story that the reader is reading, the correct answer corresponding to the question, the words to encourage the reader, etc. .

The processing unit 3 includes an automatic speech recognition (ASR) unit 31. Among them, ASR technology is a technology that converts human speech into text. In the embodiment of the present application, the ASR unit 31 cooperates with the microphone 11 to convert the voice information of the story that the reader is reading obtained by the microphone 11 and the voice information of the answer to the question raised by the terminal device to obtain the corresponding text and text Then, the processing unit 3 searches the database for the corresponding story or answer according to the obtained text.

The processing unit 3 also includes a vision processing unit 32. In the embodiment of the present application, the vision processing unit 32 cooperates with the camera 12 to process the image information obtained by the camera 12, and then extract the required features in the image. Among them, the visual processing unit 32 includes a picture book image recognition unit 321, a picture book click recognition unit 322 and a gesture recognition unit 323.

When the picture book content designer enters the picture book in advance, he extracts the cover of the entered picture book and the feature value of each inner image, and then generates the unique identity document (ID) of the picture book through the feature value, and combines the picture book ID and feature The value is associated and finally stored in the storage unit 4.

The picture book image recognition unit 321 is used to identify the current picture book cover, specific page number and other characteristic values after the camera 12 acquires the image of the page where the reader is reading the picture book, and then compares it to the storage unit 4 to find the corresponding picture book ID and The page ID is used to determine the picture book that the reader is reading and the current page of the picture book that the reader is reading. In a possible embodiment, the picture book image recognition unit 321 uses a scale-invariant feature transform (SIFT) algorithm to detect or describe local features in the image.

The picture book click recognition unit 322 is used for detecting the position of the reader's finger click area by taking a video frame image including the reader's finger through the camera 12. In a possible embodiment, the picture book click recognition unit 322 first preprocesses the collected image (that is, performs processing such as noise reduction on the hand shape area in the image, excluding areas with obvious skin color differences, etc.); and then extracts the edge of the image , To extract the edge of the convex area (that is, extract the image of the finger according to the shape and contour of the finger area); finally, according to the collected image and the extracted finger image, determine the area where the reader clicks on the picture book.

The gesture recognition unit 323 is used to recognize the actions or gestures of the reader in the image, and then determine the answer indicated by the reader according to the corresponding gesture. Among them, gesture recognition methods include, but are not limited to, recognizing gestures based on geometric features, through gesture edges (such as contours) and gesture area features (such as palm color, area, etc.).

Exemplarily, as shown in FIG. 3, the designer of the picture book content can define "1, 2, 3, 4, 5, 6, 7, 8, 9, 10" according to the number of fingers and the specific gestures of the fingers. In the process of picture book interpretation, readers can use gestures to answer questions. For example, the terminal device asks the reader: How many small animals are there on this screen? The reader can use any gesture as shown in Figure 3 as an answer. The gesture recognition unit 323 determines the answer of the reader's reply by recognizing the gesture displayed by the reader.

The processing unit 3 also includes a reading result calculation unit 33 and an interpretation round calculation unit 34. In the embodiment of the present application, the reading result calculation unit 33 is used to calculate the correct rate of the reader after answering all the questions in a round, and then store the calculated result in the storage unit 4.

The interpretation round calculation unit 34 is used to determine according to the correct rate of the answers answered by the reader in the current round, the number of times the reader has interpreted the picture book and similar picture books in the storage unit 4, and the calculation rule selected when the picture book is entered At this time, the reader will interpret the corresponding round of the picture book. All personal related data (such as reading times, reading rounds, Q&A results, etc.) generated by the reader during the reading process will be associated with the reader’s account, so as to realize the data isolation of different readers and conduct personalized reading rounds calculate.

The storage unit 4 includes a database 41 and a resource library 42. Among them, the database 41 is used to store data processed by the processing unit 3 on voice information and image information. The resource library 42 is used to store the data entered by the designer of the picture book content.

Exemplarily, the database 41 divides the database 41 into a picture book image feature database 411, a first picture book reading record database 412, and a second picture book reading record database 413 according to the type of data stored.

The picture book image feature database 411 is used to extract the image of the cover and each page of each picture book in the resource library 42 and identify the feature value, and then associate the image feature value with the data of the picture book and the corresponding page. In a possible embodiment, the picture book developer extracts the cover of the picture book and the image of each page entered in the resource library 42, recognizes the feature value, stores it in the picture book image feature database 411, and then combines each feature value with the resource library The corresponding cover and page content stored in 42 are associated with data such as questions and answers for each round. The specific association relationship is as follows:

Table 1 The association table between the feature values of the cover of the picture book and the image of each page and the data stored in the resource library

After the processing unit 3 obtains the cover of the picture book or a certain page of the image that the reader is reading through the camera 11, it is processed by the visual processing unit 32 to obtain the feature value of the image, and then sent to the picture book image feature database 411, and the picture book The feature values stored in the image feature database 411 are compared. If the feature value stored in the picture book image feature database 411 has a feature value that matches the sent image feature value, the processing unit 3 can correspond to the feature value stored in the picture book image feature database 411 that matches the sent image feature value The association relationship of the corresponding picture book or the content of the corresponding page, the questions, answers and other data of each round are obtained from the resource library 42.

The first picture book reading record database 412 is used to record the information of each picture book being interpreted, that is, when the picture book is interpreted, the correct rate of each round calculated by the reading result calculation unit 33 is used to generate a piece of interpretation information. The information includes the correct rate of each round of a picture book in an interpretation process. In a possible embodiment, the format of the interpretation information stored in the first picture book reading record database 412 is as follows:

Table 2 The relationship between the picture book in the interpretation information and the correct rate of answers in each round

Each time the terminal device interprets the picture book for the reader, it will generate a piece of interpretation information to record the reader’s understanding of the picture book during the interpretation process. When the picture book is subsequently interpreted again, the interpretation round calculation unit 34 According to the interpretation information in the history of the picture book, determine the round of directly entering the picture book to be interpreted again.

The second picture book reading record database 413 is used to record the interpretation information of each type of each picture book. Compared with the interpretation information stored in the first picture book reading record database 412, the interpretation information stored in the second picture book reading record database 413 is increased Type classification, the interpretation information is divided into number learning interpretation information, letter learning interpretation information, etc., as follows:

Table 3 The relationship between the picture book in the interpretation information and the correct rate of answers in each round

The resource library 42 is used to store at least one picture book. Among them, the stored picture book content includes the basic information of the picture book (name, picture book type, subtype, number of pages, cover picture, specific page number picture, etc.), picture book cover and images of each page, image feature values, text and text of each page Images, questions and answers for each round corresponding to each page, round pass calculation criteria and other information. When the reader interprets the picture book, the processing unit 3 obtains information related to the picture book for interpretation from the resource library 42 after determining the picture book read by the reader and the page currently being read.

The input unit 5 is used to download the content of the picture book and then input it into the resource library 42. Wherein, the input unit 5 may be a physical interface such as a USB interface, a Type-C interface, etc., or a wireless communication module such as a WiFi module, a Bluetooth module, etc., which is not limited in this application.

Before the terminal device assists the reader in interpreting the picture book, the picture book developer needs to make the picture book content in the development tool in advance, and then store it in the resource library 42 through the input unit 5.

In the following, in conjunction with the terminal device shown in Figure 2, it will be described how a picture book developer makes a picture book. Among them, the picture book is described with the children’s picture book "Dad, Don't Fear" as an example, and this application is not limited here.

Picture book developers make picture book content through dedicated APP, cloud, etc. (this application uses APP as an example). After opening the APP to make, the APP displays the template shown in Figure 4, which includes the name of the picture book, author, and interpretation rounds The total number, the name of the picture book series, the picture book classification, the picture book sub-category, and the cover picture are options. The picture book developer can fill in the various options in the template according to the picture book to be entered.

Exemplarily, as shown in Figure 4, each option in the template can be divided into required items and non-required items. Required items need to be filled in by the picture book developer, such as picture book name, author, cover image and other options. Required items do not need to be filled in, such as picture book classification, picture book sub-category and other options.

After the picture book developer has filled in the various options in the template shown in Figure 4, click on the "Start Entering Picture Book Interpretation Content" option to enter the template shown in Figure 5. The template includes at least one round of interpretation (this application takes three rounds of interpretation as an example), at least one picture book page (this application takes a picture book with 15 pages as an example), reading content, question and answer interaction and other options.

Exemplarily, as shown in Figure 5, in a situation, the options on each page include three options: the first interpretation, the second interpretation, and the third interpretation, and each interpretation option includes There are two options for reading content and Q&A interaction. When the picture book developer produces the contents of the first round of interpretation of the first page of the picture book "Dad, Don't Fear", after selecting the option "Page 1" from the options on the 15 pages, three interpretation options will appear ; Then, after selecting the "first round of interpretation" option among the three interpretation options, two options for reading content and Q&A interaction will appear. The picture book developer uploads the image on the first page of the picture book "Dad, don’t be afraid" and the text content recorded on the first page to the "Read aloud" option, and then upload the questions and answers that need to be asked to the "Question and answer interaction" option .

In another case, the options on each page include two options: reading content and Q&A interaction. The two options for reading content and Q&A interaction include the first interpretation, the second interpretation, and the third interpretation. Options. When the picture book developer makes the first round of interpretation of the first page of the picture book "Dad, Don't Be Fear", after selecting the "1st page" option from the 15 page options, the reading content and Q&A will appear There are two interactive options; after selecting the "read aloud" option in the two options, three interpretation options will appear; then after the "first round of interpretation" is selected in the three interpretation options, the picture book developer will set the picture book The image of the first page of "Dad, don't be afraid" and the text content recorded on the first page are uploaded to the "Read aloud" option; after selecting the "Q&A interaction" option in the two options, three interpretations will appear Option; then after selecting the "first round of interpretation" option in the three interpretation options, the picture book developer uploads the image on the first page of the picture book "Dad, Don’t Be Fear" and the questions and answers that need to be asked on the first page to " Q&A interaction" option.

In the same way, the way that the picture book developers make the other rounds of interpretation of the contents of the picture book on the other pages of the picture book "Dad, Don't Fear" is the same as the above-mentioned method of making the first round of interpretation of the contents of the picture book on the first page of the picture book "Dad, Don't Fear" same.

After the picture book developer has filled in the options of each page and each round of interpretation in the template shown in Figure 5, click on the "Complete Interpretation" option to enter the template shown in Figure 6. The template includes options such as entering the next round of interpretation rules and the average number of forward answers.

Exemplarily, as shown in Figure 6, the "Enter the next round of interpretation rules" options include "Method 1: The first round of interpretation of the picture book Q&A interaction, the number of readers' positive answers exceeds" and "Method 2: The same series have been read There are more than 2 picture books, and in the first round of interpretation and answer interaction, the average number of readers' positive answers exceeds the two "choose one" options. If the picture book you make is a series picture book, select "Method 1"; if the picture book you make is a non-series picture book, select "Method 2". Then select the number of positive answer rate in the "options such as the average number of positive answers", and the terminal device will enter the second round of interpretation only if the positive answer rate of the reader's reply is greater than the set positive answer rate.

After the picture book developer fills in the various options in the template shown in Figure 6, click on the "Submit" option, which indicates that the picture book "Dad, don't be afraid" is finished. The picture book developer can upload the picture book to the cloud server, and then the reader downloads the picture book through the input unit 5 as needed.

When the picture book is downloaded to the resource library 42 through the entry unit 5, the picture book image feature value database 411 extracts the image of the cover and each page in the picture book, identifies the feature value, generates the picture book cover and the ID of each page, and adds each Each feature value is associated with each ID, and the specific relationship is shown in Table 1, and then stored in the picture book image feature value database 411.

The technical solution of the embodiment of the present application will be described in more detail below in conjunction with the terminal device shown in FIG. 2 (the terminal device is an intelligent robot as an example below).

FIG. 7 is a work flow chart of a terminal device provided by an embodiment of the application to assist a reader in interpreting a picture book.

Step S701: Determine the current picture book that the reader is reading.

When the reader uses the intelligent robot to accompany himself to interpret the story, the reader initiates a picture book interpretation instruction to the intelligent robot through voice, keystrokes, etc., to let the intelligent robot work. After receiving the instructions, the intelligent robot judges the current user's identity information (through voiceprints, passwords, fingerprints, face information, etc.), and then turns on the microphone 11 and/or the camera 12 to work to obtain the voice and voice of the reader who is reading the picture book. / Or the image of the current page of the picture book that the reader is reading.

In a possible embodiment, the microphone 11 of the intelligent robot collects the voice information of the reader, and then the ASR unit 31 in the processing unit 3 converts the collected voice into corresponding text, and then extracts multiple texts from the converted text. The keywords are compared with the corresponding keywords in each picture book stored in the resource library 42 in the storage unit 4. If the keywords converted from the collected voice information match the corresponding keywords of a picture book in the resource library 42, it indicates that the reader is reading the picture book, so as to determine the picture book that the reader is reading and the picture book that the reader is reading. The page number corresponding to the content; if the collected voice information is converted to keywords and there is no corresponding keyword in any picture book in the resource library, it means that the reader is not "reading the story" or the database 42 does not store the reader The picture book being read, so that the intelligent robot cannot interpret the story for the reader.

Among them, each picture book stored in the resource library 42 is made in the manner shown in Figures 4-6 and the corresponding description content, which will not be repeated here in this application.

In a possible embodiment, the camera 12 of the smart robot collects images or videos, and then the vision processing unit 32 in the processing unit 3 processes the collected images (if it is a video, the vision processing unit 32 processes each frame of the video separately) The recognition is performed to obtain multiple feature values in the current image, and then the feature values of the corresponding images in each picture book stored in the picture book image feature database 411 in the storage unit 4 are compared. If the feature value of the captured image or video matches the corresponding feature value of a picture book in the resource library, it indicates that the reader is reading the picture book, so as to determine the picture book the reader is reading and the content of the picture book. Corresponding page number; if the feature value of the collected image or video does not match the corresponding feature value of any picture book in the resource library, it indicates that the reader is not "reading the story" or the database 42 does not store the reader is reading So that the intelligent robot cannot read the story.

Of course, the intelligent robot in the embodiment of the present application can also turn on the microphone 11 and the camera 12 to work at the same time, and obtain voice information, image or video information at the same time, so as to more accurately determine the picture book the reader is reading and the content of the picture book. The corresponding page number.

Step S703: Obtain historical information of the current picture book.

In step S705, the reader is asked the first round of questions.

After determining the page number corresponding to the picture book the reader is reading and the content of the picture book, the intelligent robot starts the picture book interpretation work.

In a possible embodiment, the interpretation round calculation unit 34 in the processing unit 3 obtains data about the historical reading situation of the picture book from the first picture book reading record database 412 of the storage unit 4. When the data obtained by the interpretation round calculation unit 34 does not record the information related to the interpretation of the picture book, it indicates that the picture book is interpreted for the first time, and the processing unit 3 obtains the first round of interpretation question of the picture book from the resource library 42. Then the reader is asked the question of the first round of interpretation through the speaker 21; when the interpretation round calculation unit 34 obtains a piece of data in the following table,

Table 4 The relationship between the picture book in the interpretation information and the correct rate of answers in each round

It indicates that the picture book has been interpreted once in history, and the positive answer rate of the reader in the first round of interpretation is 60% (assuming that the positive answer rate from the first round to the second round of interpretation is 60%), no The positive answer rate of readers in the second round of interpretation, and there is no positive answer rate of readers in the third round of interpretation. The processing unit 3 obtains the average positive answer rate of the first round of questions at 60%, and then retrieves it from the resource library 42 Obtain the second-round interpretation question of the picture book, and then ask the reader the second-round interpretation question through the speaker 21; when the interpretation-round calculation unit 34 obtains two pieces of data in the following table,

Table 5 The relationship between the picture book in the interpretation information and the correct rate of answers in each round

It indicates that the picture book has been interpreted twice in history. In the first interpretation process, the positive answer rate of the reader in the first round of interpretation is 60% (assuming the positive answer from the first round to the second round of interpretation) The answer rate is 60%), the positive answer rate of readers in the second round of interpretation is 10%, and there is no positive answer rate of readers in the third round of interpretation. During the second interpretation, read in the first round of interpretation. The positive answer rate of readers is 60%. The positive answer rate of readers in the second round of interpretation is 60%. There is no positive answer rate of readers in the third round of interpretation. Processing unit 3 gets the positive answer to the first round of questions. The rate is 60%, and the positive answer rate for the second round of questions is 35%. Then, obtain the second round of interpretation questions for the picture book from the resource library 42, and then ask the readers the second round of interpretation through the speaker 21; others; The situation can be deduced by analogy.

In a possible embodiment, the interpretation round calculation unit 34 in the processing unit 3 obtains data about the historical reading situation of the picture book from the second picture book reading record database 413 of the storage unit 4. If the current reader is studying "English picture books", when the data obtained by the interpretation round calculation unit 34 does not record that the picture book is of the "English picture book" type, it indicates that the picture book is of the "English picture book" type. When it is interpreted for the first time, the processing unit 3 obtains the first-round interpretation question of the "English picture book" type of the picture book from the resource library 42, and then asks the reader the first-round interpretation question through the speaker 21; When the calculation unit 34 obtains four pieces of data in the following table,

Table 6 The relationship between the picture book in the interpretation information and the correct rate of answers in each round

It indicates that the “English picture book” type of the picture book has been interpreted twice in history. In the first interpretation process, the positive answer rate of the reader in the first round of interpretation is 50% (assuming that the first round of The positive answer rate of the second round of interpretation is 60%), there is no positive answer rate of the reader in the second round of interpretation, and there is no positive answer rate of the reader in the third round of interpretation. The positive answer rate of readers in the first round of interpretation is 70% and the positive answer rate of readers in the second round of interpretation is 60%. Without the positive answer rate of readers in the third round of interpretation, processing unit 3 gets the first round of questions The positive answer rate for the second round of questions is 60%, and the positive answer rate for the second round of questions is 60%. Then the third round of interpretation questions of the "English picture book" type of the picture book is obtained from the resource library 42, and then the speaker 21 The reader asks the questions of the third round of interpretation; other situations are analogous to this.

Among them, the reader can let the intelligent robot enter the specified type of the picture book through voice instructions, instructions input on the screen, etc.

Step S707: Receive the reader's answer to each question in the first round of questions.

In step S709, when the positive answer rate of the answer to each question is greater than the set threshold, the reader is asked the second round of questions.

After the intelligent robot broadcasts the question through the speaker 21, the microphone 11 and/or the camera 12 are turned on for a set time period to obtain the reader's response to the question.

In a possible embodiment, the microphone 11 of the intelligent robot collects the voice information of the reader, and then the ASR unit 31 in the processing unit 3 converts the collected voice into corresponding text, and then extracts multiple texts from the converted text. The keywords are compared with the answers to the corresponding questions stored in the resource library 42 in the storage unit 4. If the key words converted from the collected voice information match the key words of the answer to the corresponding question in the resource library 42, it indicates that the answer answered by the reader is correct, and the processing unit 3 can broadcast "the answer is correct" to the reader through the speaker 21 , "You are great, you answered correctly" and other voices, so that the reader knows that the answer you answered is correct; if the keywords converted from the collected voice information are not the same as the keywords of the answer to the corresponding question in the resource library 42 Match, it indicates that the reader’s answer is wrong. The processing unit 3 can broadcast to the reader voices such as "Answer Wrong", "Think about it again, is there a better answer" and other voices to the reader through the speaker 21, so that the reader knows what he has answered. The answer is wrong, then broadcast the correct answer.

In addition, when the reader’s answer is wrong, the microphone 11 can also be turned on again to allow the reader to answer again. When the reader fails to answer the correct answer within the set number of times, the processing unit 3 then passes through the speaker 21 Broadcast the correct answer.

In a possible embodiment, if a gesture is used to answer the reading, the camera 12 of the smart robot collects images or videos, and then the vision processing unit 32 in the processing unit 3 performs the processing on the collected images (if it is a video, the vision processing unit 32 responds to the video Each frame of the image is processed separately) for identification to obtain multiple feature values in the current image, and when the reader’s hand index and finger combination are determined based on the multiple feature values, the specific number of the reader’s expression is determined, and then stored with The answers to the corresponding questions stored in the resource library 42 in unit 4 are compared. If the specific number determined is the same as the answer to the corresponding question in the resource library 42, it indicates that the answer answered by the reader is correct, and the processing unit 3 can broadcast to the reader through the speaker 21 "the answer is correct", "you are great, the answer is correct" "And so on, let the reader know that the answer he answered is correct; if the specific number determined is not the same as the answer to the corresponding question in the resource library 42, it indicates that the answer answered by the reader is wrong, and the processing unit 3 can use the speaker 21 Announce the "Answer Wrong", "Think about it again, is there a better answer" and other voices to the reader, let the reader know that the answer he answered is wrong, and then announce the correct answer.

Of course, the intelligent robot in the embodiment of the present application can also turn on the microphone 11 and the camera 12 to work at the same time, and obtain voice information, image or video information at the same time, so as to more accurately determine the answer of the reader.

In a possible embodiment, as shown in FIG. 8, if the smart robot has a touch screen, the processing unit 3 may present the problem on the touch screen in the form of a picture. For example, the intelligent robot asks the reader "Who do you think will become a bunny in this story?", after the reader knows the problem, he can locate the "Daddy Bear", "Little Bear" and other objects on the screen. Then, the processing unit 3 compares the position clicked by the reader with the answer of the corresponding question stored in the resource library 42 to determine whether the position clicked by the reader is correct. If the clicked position is correct, it means that the reader’s answer is correct, and then broadcast through the speaker 21 or touch the screen to tell the reader that the answer is correct; if the clicked position is wrong, it means that the reader’s answer is wrong, and then pass The speaker 21 broadcasts or the display mode on the touch screen tells the reader that the answer is wrong.

After the intelligent robot asks the readers all the questions in the first round, the reading result calculation unit 33 in the processing unit 3 calculates the number of questions answered correctly by the readers in this round, and then calculates the correct rate. If the correct rate does not exceed the set When the threshold is set, the intelligent robot does not enter the second round of interpretation; if the correct rate exceeds the set threshold, the intelligent robot enters the second round of interpretation.

In addition, if the intelligent robot finishes this interpretation, the processing unit 3 will store the picture book, type, round of interpretation, and forward answer rate of each round of interpretation in the format of Table 2 and Table 3. In the first picture book reading record database 412 and the second picture book reading record database 413, subsequent readers can refer to the result of this interpretation to determine the round that they directly enter when re-interpreting.

Furthermore, due to the large number of picture books on the market, it may happen that some stories in different picture books are the same, the same picture books are published by different publishers, and the same picture books are presented in different languages. The processing unit 3 may not need to include the picture book cover when generating the interpretation information. In this way, it is not necessary to determine the interpretation round based on the reading situation of each picture book rigidly. Instead, the interpretation round of the current picture book can be determined based on whether there has been a reading record similar to the picture book.

Furthermore, take Miyoshi Tatsuya's Superman series of picture books (a total of three books) as an example. This series of picture books includes three picture books of "Additional Superman and Arithmetic Starman", "Fantasy Superman" and "Prince of Justice". When the three picture books made in the manner shown in Figure 4-6 and the corresponding description content are stored in the resource library 42 (wherein the "Enter the next round of interpretation rules" option is selected "Method 2"), if the reader According to the aforementioned rules, I have read the first round of "Additional Superman and Arithmetic Starman" and the first round of "The Righteous Man" in the series of picture books. Seven of the 10 questions in one round were answered positively, and eight were answered positively in the first round of "The Righteous Man". At this time, the first picture book reading record database 412 contains one piece of interpretation information for each of the two books, and the second picture book reading record database 413 records a set of interpretation information. The average positive answer rate of reading picture books: (0.7+0.8)/2=75%.

When readers start to read "Fantasy Superman" for the first time, according to the records in the basic information of the picture book, "Fantasy Superman" belongs to the Superman series of picture books. According to the reading round calculation rules, query the interpretation information in the reading record database 413 of the second picture book According to the interpretation information, there are more than 2 picture books in the Superman series of picture books. Since in the second round of interpretation question-and-answer interaction of the series of picture books, the average number of readers’ positive answers is 75%>60%, so the intelligent robot directly starts from the first Interpretation began in the second round.

The picture book interpretation method provided in the embodiments of this application classifies the problems of interpreting picture books according to the degree of difficulty. If the reader reads the picture book for the first time, the reader will be asked the simplest type of questions, if based on historical records The information indicates that the reader has done many interpretations, then ask the reader the corresponding round question, so as to intelligently ask the reader reasonable questions, and effectively help the reader understand the picture book. When one type of question is asked, according to the correct rate of the answers from the reader, determine whether to ask the more difficult type one question, so as to gradually guide the reader to understand the story and improve the reader's reading comprehension ability .

FIG. 9 is a schematic structural diagram of a terminal device according to an embodiment of the present invention. A terminal device 900 shown in FIG. 9, the electronic device 900 may be the aforementioned intelligent robot, and includes a sensor 901, a display 902, a processor 903, a memory 904, a communication interface 905, and a bus 906. The processor 903, the memory 904, and the communication interface 905 in the electronic device can establish a communication connection through the bus 906.

The sensor 901 is used to obtain the reader's voice information and image or video information, and to send audio and video information. The sensor 901 may include a camera, a microphone, a speaker, and so on.

The display 902 is used to display processed data, such as videos and images.

The processor 903 may be a central processing unit (CPU).

The memory 904 may include a volatile memory (volatile memory), such as a random-access memory (random-access memory, RAM); the memory may also include a non-volatile memory (non-volatile memory), such as a read-only memory (read-only memory). Only memory, ROM), flash memory, hard disk drive (HDD) or solid state drive (SSD); the memory 904 may also include a combination of the foregoing types of memories.

The interpretation methods provided in the foregoing embodiments are all executed by the processor 903. Data such as pictures, voices, and picture book content will be stored in the memory 904. In addition, the memory 904 will also be used to store program instructions executed by the processor 903 for implementing the terminal information protection method described in the foregoing embodiment, and so on.

A person of ordinary skill in the art may realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered as going beyond the scope of the embodiments of the present application.

In addition, various aspects or features of the embodiments of the present application can be implemented as methods, devices, or products using standard programming and/or engineering techniques. The term "article of manufacture" used in this application encompasses a computer program accessible from any computer-readable device, carrier, or medium. For example, the computer-readable medium may include, but is not limited to: magnetic storage devices (for example, hard disks, floppy disks, or tapes, etc.), optical disks (for example, compact discs (CD), digital versatile discs (DVD)) Etc.), smart cards and flash memory devices (for example, erasable programmable read-only memory (EPROM), cards, sticks or key drives, etc.). In addition, various storage media described herein may represent one or more devices and/or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and/or carrying instructions and/or data.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instruction may be transmitted from a website, computer, server, or data center through a cable (Such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center integrated with one or more available media. The usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, and a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)).

It should be understood that in the various embodiments of the embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of the processes should be determined by their functions and internal logic, and should not be dealt with. The implementation process of the embodiments of the present application constitutes any limitation.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, device, and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application are essentially or the part that contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium. , Including several instructions to make a computer device (which may be a personal computer, a server, or an access network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks or optical disks and other media that can store program codes. .

The above are only specific implementations of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited to this. Anyone familiar with the technical field in the technical scope disclosed in the embodiments of the present application can easily Any change or replacement should be included in the protection scope of the embodiments of the present application.

Claims

A method for interpreting picture books, executed by a terminal device, characterized in that the method includes:

Determine the current picture book the reader is reading;

Acquire the historical information of the current picture book, the historical information includes the number of interpretations of the current picture book by the reader in the history and the correctness of each round of questions raised by the reader each time the reader in the history interprets the picture book. Answer rate

A first round of questions is asked to the reader, and the first round of questions is determined according to the historical information.
The method according to claim 1, wherein the method further comprises:

Receiving the reader's answer to each question in the first round of questions;

When the positive answer rate of the answers to the various questions is greater than the set threshold, the reader is asked a second round of questions, and the questions in the second round of questions are different from those in the first round of questions The problem.
The method according to claim 2, wherein the method further comprises:

The order of the first round of questions and the second round of questions is set according to the degree of difficulty.
The method according to any one of claims 1 to 3, wherein before said obtaining the historical information of the current picture book, it further comprises: obtaining the identity information of the reader,

The obtaining the historical information of the current picture book includes: obtaining the historical information of the current picture book corresponding to the identity information.
The method according to any one of claims 1 to 4, wherein after the determining the current picture book that the reader is reading, the method further comprises:

Determine whether the current picture book is stored in the resource library;

When the resource library does not have the current picture book, determining whether the content of the current picture book is the same as the content of the first picture book stored in the resource library;

When the content of the current picture book is the same as the content of the first picture book, a round question corresponding to the first picture book is asked to the reader.
The method according to any one of claims 1-5, wherein the method further comprises:

Determine the page number of the current picture book that the reader is reading;

Ask the reader a question about the turn of the content corresponding to the page number.
The method according to any one of claims 1-6, wherein the determining the current picture book currently being read by the reader comprises:

Acquire the voice information of the current picture book that the reader is reading through a microphone;

The voice information is converted into picture book text, and the picture book text is used to determine the current picture book currently being read by the reader from the resource library.
The method according to any one of claims 1-6, wherein the determining the current picture book currently being read by the reader comprises:

Acquiring, through a camera, the image of the current picture book that the reader is reading;

Identify the features in the image to obtain the picture book feature value, and the picture book feature value is used to determine the current picture book being read by the reader from the resource library.
The method according to claim 1, wherein the method further comprises:

In the historical information, the number of interpretations of the reader's interpretation of the current picture book is added once, and the positive answer rate of the answers of the readers' answers to each round of questions during this interpretation process is stored.
The method according to any one of claims 1-9, wherein before the first round of questions is asked to the reader, the method comprises:

An interpretation instruction is received, and the interpretation instruction is used to instruct to ask the reader the first round of questions.
A picture book interpretation device, which is characterized by comprising: a transceiver, a processor and a memory;

The transceiver is used to receive and send data;

The memory stores one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the electronic device executes any one of claims 1-10. The method described.
An electronic device, characterized by comprising: a camera and/or a microphone, a memory, and a processor for executing the method according to claims 1-10.
An intelligent robot, characterized in that it includes:

The camera and/or the microphone are used to receive the voice information or image information of the reader reading the picture book, and the voice information or the image information of the reader's answer to the question corresponding to the picture book;

The memory is used to store data packets of at least one picture book, as well as the number of interpretations of the current picture book by the reader in the history of each picture book and the rounds proposed by the reader in each interpretation in the history The positive answer rate of the question;

The processor is configured to process the voice information or image information acquired by the camera and/or the microphone, and then determine from the memory the current picture book that the reader is reading, and according to the information in the memory State the number of interpretations of the current picture book and the average value of the positive answer rate obtained at each interpretation of each round of questions, and determine to ask the reader the corresponding round of questions;

Loudspeaker, used to play voice to readers;

The communication unit is used to receive the data packet of each picture book and the threshold of the forward answer rate from the current round to the next round.
A readable storage medium for storing instructions. When the instructions are executed, the method according to any one of claims 1-10 is realized.
A computer program device containing instructions, when it runs on a terminal, causes the terminal to execute the method according to any one of claims 1-10.