CN109191940B

CN109191940B - Interaction method based on intelligent equipment and intelligent equipment

Info

Publication number: CN109191940B
Application number: CN201811015615.6A
Authority: CN
Inventors: 徐杨
Original assignee: Guangdong Genius Technology Co Ltd
Current assignee: Guangdong Genius Technology Co Ltd
Priority date: 2018-08-31
Filing date: 2018-08-31
Publication date: 2021-09-24
Anticipated expiration: 2038-08-31
Also published as: CN109191940A

Abstract

The invention provides an interaction method based on intelligent equipment and the intelligent equipment, when a user encounters difficulty, the user can point to questioning content to be asked and ask the intelligent equipment for answering content, and when the intelligent equipment identifies that voice information of the user contains questioning keywords, the intelligent equipment acquires questioning area images through a camera module; then analyzing the question content from the question area image acquired by the camera module, and acquiring the answer content corresponding to the question content; and generating projection information according to the answer content, and projecting the projection information to a preset projection area according to a projection mode selected by a user. The intelligent device integrates the functions of acquiring questions, solving questions, displaying solved contents and the like, can acquire questioning contents of students when the students have difficulty in learning, and displays the solved contents in a projection mode, so that the intelligent device provides learning assistance for the students and reduces the burden of parents.

Description

Interaction method based on intelligent equipment and intelligent equipment

Technical Field

The invention relates to the field of intelligent learning products, in particular to an interaction method based on intelligent equipment and the intelligent equipment.

Background

Under the current environment, the study pressure of students in China is generally high, and the students need to complete the piled operation every day. Students often encounter various problems when completing homework, such as the fact that new words cannot be read and the fact that arithmetic cannot be done, in which case tutoring needs to be performed in time. However, many parents are under the work pressure and cannot guide the children to complete the post-school work at all, so that the learning performance of many children is reduced.

With the development of science and technology, electronic equipment such as mobile phones, tablet computers, learning tablets and the like appear in the market, and help students solve problems encountered in homework. One way is to search the answer to the question after taking a picture against a book with the electronic device; another way is to speak a question to the electronic device and search for an answer to the question.

Above mode of helping the student to solve the homework problem has following defect: the method comprises the following steps that (I) a student breaks away from a homework scene, and inquiry and check are carried out on additional intelligent equipment; the students can not get rid of intelligent equipment such as mobile phones and tablet computers, and the students can delay learning by borrowing the mobile phones or the tablet computers from homework; and (III) when the problem is spoken to the electronic equipment, if the student expresses the unclear problem, the student cannot be helped to solve the homework problem.

The invention provides an interaction method based on intelligent equipment and the intelligent equipment, which are used for providing timely help when a student is difficult to learn and enabling the student to finish autonomous learning when the student is difficult to learn.

Disclosure of Invention

The invention aims to provide an interaction method based on intelligent equipment and the intelligent equipment, which can provide timely help when students encounter difficulty in learning.

The technical scheme provided by the invention is as follows:

the invention provides an interaction method based on intelligent equipment, which comprises the following steps:

when the intelligent equipment identifies that the voice information of the user contains a question keyword, the intelligent equipment acquires a question area image through a camera module;

the intelligent equipment analyzes the question content from the question area image acquired by the camera module and acquires the answer content corresponding to the question content;

and the intelligent equipment generates projection information according to the answer content and projects the projection information to a preset projection area according to a projection mode selected by a user.

The scheme combines the lamps which are necessities used by students during learning, integrates the functions of acquiring questions, solving the questions, displaying the content of the solution and the like on the lamps, and can provide help for the students in time when the students encounter difficulty in learning. In the scheme, after the intelligent device acquires the answer content, the answer content can be projected in a projection mode, an external display screen is not needed, or a display screen is not needed to be additionally arranged on the lamp, and the application universality of the intelligent device is improved.

Preferably, when the intelligent device monitors that the voice information of the user contains a question keyword, the intelligent device acquires a question area image through the camera module, and the method further comprises the following steps: the intelligent equipment monitors voice information of a user and detects whether the voice information contains a question keyword or not;

if the intelligent equipment detects that the voice information does not contain the question keywords, the voice information of the user is monitored continuously; and if the intelligent equipment detects that the voice information contains the question keywords, starting a camera module.

In the scheme, when a student encounters difficulty, the student can ask a question of 'how to do the question' or 'how to do the fifth question', and after the intelligent device analyzes that the voice contains a question keyword, the intelligent device can open the camera module to acquire a question area image. According to the scheme, the camera module is not started in real time, but is started under the condition that specific conditions are met, so that the energy consumption of the camera module is reduced.

Preferably, when the intelligent device recognizes that the voice information of the user contains a question keyword, the intelligent device obtains a question area image through the camera module, and the method specifically includes the following steps:

when the intelligent equipment identifies that the voice information of the user contains a question keyword, the intelligent equipment opens a camera module, and obtains an interface image of the book by taking the detected edge of the book as a boundary;

and the intelligent equipment detects a target pointing area from the interface image of the book and takes an image corresponding to the target pointing area as the question area image.

When the questioning area image is obtained, the interface image of the book is obtained firstly, then the target pointing area is obtained in the interface image and serves as the questioning area image, the detection range can be reduced to the inside of the interface of the book, and the detection interference is reduced.

Preferably, the projection information is projected to a preset projection area in a projection mode selected by a user, and the step specifically includes:

when the user selects the projection mode to be LED projection, the intelligent device projects the projection information to a preset projection area in an LED projection mode through a projection module;

or when the user selects the projection mode to be light projection, the intelligent equipment projects the projection information to a preset projection area in a light projection mode through a projection module;

or when the projection mode selected by the user is 3D holographic projection, the intelligent equipment projects the projection information to a preset projection area in a 3D holographic projection mode through a projection module;

or when the projection mode selected by the user is AR projection, the intelligent device projects the projection information to a preset projection area in the AR projection mode through the projection module.

The scheme provides multiple projection modes, different projection requirements can be met, for example, when the projected answer content comprises a three-dimensional image or a chemical experiment image, the 3D holographic projection mode can be used for projection, and when the projected answer content comprises multiple colors, the LED projection mode can be used, so that the projected colors are distinguished more obviously.

Preferably, the method further comprises the following steps:

the intelligent equipment records the question content to form a question record, and searches whether a question record similar to the question content exists in a historical question record;

and if the question records similar to the question contents exist in the historical question records, projecting the question contents and the answer contents corresponding to the similar question records to the preset projection area.

In this scheme, the problem that the student asked all can be got rid of by the smart machine record, forms the record of asking questions, can throw out similar problem and answer content when proposing similar problem next time, lets the student review the problem that has not understood before, deepens the study to same knowledge point, consolidates the knowledge point.

The present invention also provides an intelligent device, comprising:

the camera module is used for acquiring a question area image when the intelligent equipment identifies that the voice information of the user contains a question keyword;

the analysis module is electrically connected with the camera module and used for analyzing the question content from the question area image acquired by the camera module;

the answer obtaining module is electrically connected with the analysis module and is used for obtaining answer contents corresponding to the questioning contents;

the projection information generation module is electrically connected with the answer acquisition module and used for generating projection information according to the answer content;

and the projection module is electrically connected with the projection information generation module and used for projecting the projection information to a preset projection area according to a projection mode selected by a user.

The intelligent device that provides in this scheme has integrateed camera module, analysis module, answer acquisition module, projection information generation module and projection module on, has realized functions such as acquireing the problem, answering the problem, showing the content of answering, when student's study meets the difficulty, can in time provide help for the student. In this scheme, after the smart machine acquires the answer content, throw out the answer content with the mode of projection, need not external display screen or add display screen on lamps and lanterns, increase the universality of using.

Preferably, the smart device further includes:

the voice detection module is used for monitoring voice information of a user and detecting whether the voice information contains a question keyword;

the voice detection module is further configured to continue to monitor the voice information of the user if the intelligent device detects that the voice information does not contain the question keyword;

and the control module is electrically connected with the voice detection module and is used for starting the camera module when the voice information contains the question keywords.

Preferably, the intelligent terminal includes:

the control module is electrically connected with the voice detection module and used for starting the camera module when the voice detection module identifies that the voice information of the user contains question keywords;

the camera module is also electrically connected with the control module and is used for acquiring an interface image of the book by taking the detected edge of the book as a boundary;

and the question area identification module is electrically connected with the camera module and is used for detecting a target pointing area from the interface image of the book and taking an image corresponding to the target pointing area as the question area image.

Preferably, the projection module is configured to, when the user selects the projection mode to be LED projection, project the projection information to a preset projection area through the projection module by the smart device in the LED projection mode;

or when the user selects the projection mode to be light projection, the intelligent device projects the projection information to a preset projection area in a light projection mode through a projection module;

or when the projection mode selected by the user is 3D holographic projection, the intelligent device projects the projection information to a preset projection area in a 3D holographic projection mode through a projection module;

Preferably, the smart device further includes:

the recording module is used for recording the question content to form a question record;

the searching module is electrically connected with the recording module and is used for searching whether question records similar to the question contents exist in historical question records or not;

the projection module is electrically connected with the search module and is further used for projecting question content and answer content corresponding to similar question records to the preset projection area if the question records similar to the question content exist in the historical question records.

Preferably, or; the intelligent equipment comprises an intelligent desk lamp, the camera module and the projection module are arranged at the head of the intelligent desk lamp, and the camera shooting direction of the camera module and the projection direction of the projection module face one side of a base of the intelligent desk lamp; or; the intelligent equipment comprises an intelligent wall lamp, the camera module and the projection module are arranged on the intelligent wall lamp, and the camera shooting direction of the camera module and the projection direction of the projection module face one side of the preset projection area; or; the intelligent equipment comprises an intelligent ceiling lamp, the camera module and the projection module are arranged on the intelligent ceiling lamp, and the camera direction of the camera module and the projection direction of the projection module face one side of the preset projection area.

The interaction method based on the intelligent device and the intelligent device provided by the invention can bring at least one of the following beneficial effects:

in order to provide timely help when a student encounters difficulty in learning, the intelligent device integrates the functions of acquiring questions, solving questions, displaying the content of the solution and the like, and when the student encounters difficulty in learning, the intelligent device only needs to say that the student does not understand the question: the intelligent equipment can acquire images of the questioning area, then acquire questioning contents through an image analysis technology, acquire answering contents through a knowledge base, and project the answering contents for students to learn and refer. The visible intelligent equipment plays a role in tutoring, provides learning help for students and reduces the burden of parents.

Drawings

The above features, technical features, advantages and implementations of a smart device-based interaction method and a smart device will be further described in the following detailed description of preferred embodiments in a clearly understandable manner with reference to the accompanying drawings.

FIG. 1 is a flow chart of one embodiment of an intelligent device based interaction method of the present invention;

FIG. 2 is a flow chart of another embodiment of an intelligent device based interaction method of the present invention;

FIG. 3 is a flow chart of another embodiment of an intelligent device based interaction method of the present invention;

FIG. 4 is a schematic block diagram of one embodiment of a smart device of the present invention;

FIG. 5 is a schematic diagram of another embodiment of a smart device of the present invention;

fig. 6 is a schematic structural diagram of an intelligent device.

The reference numbers illustrate:

the system comprises a camera module 1, a camera module 2, an analysis module 3, an answer acquisition module 4, a projection information generation module 5, a projection module 6, a voice detection module 7, a control module 8, a recording module 9, a search module and a question area identification module 10.

Detailed Description

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following description will be made with reference to the accompanying drawings. It is obvious that the drawings in the following description are only some examples of the invention, and that for a person skilled in the art, other drawings and embodiments can be derived from them without inventive effort.

For the sake of simplicity, the drawings only schematically show the parts relevant to the present invention, and they do not represent the actual structure as a product. In addition, in order to make the drawings concise and understandable, components having the same structure or function in some of the drawings are only schematically illustrated or only labeled. In this document, "one" means not only "only one" but also a case of "more than one".

The invention provides an embodiment of an interaction method based on intelligent equipment, as shown in fig. 1, comprising:

s1, when the intelligent equipment identifies that the voice information of the user contains a question keyword, the intelligent equipment acquires a question area image through the camera module;

the camera module can be a camera and can be installed on intelligent equipment, the intelligent equipment comprises an intelligent desk lamp, an intelligent pendant lamp, an intelligent wall lamp and the like and can be installed near a desk, and the intelligent equipment can acquire image information of books on the desk through the camera module installed on the body.

When a student learns, if the student encounters a question which is unclear or can not be done, the student can use fingers to do the question which is not going to be done, and then similar questions such as 'how to do the question' or 'how to do the first question' are spoken. The intelligent device can acquire voice information of a user in real time, convert the voice information into question text information, judge whether question keywords (such as 'how to do', 'how to say' and the like) exist, analyze images acquired by the camera when the intelligent device identifies that the voice information of the user contains the question keywords, identify the outline of a hand and the direction of the finger through an image identification technology, find out the area where the question the user wants to inquire on the finger direction area, and take the images of the area as question area images.

S2, the intelligent equipment analyzes the question content from the question area image acquired by the camera module and acquires the answer content corresponding to the question content;

specifically, an image processing chip (for example, an FH8510 image processing chip or an FMS6502 image processing chip) may be integrated in the smart device, and the image of the question area may be identified, and the question content in the image of the question area may be analyzed, for example, the image of the question area may be analyzed, so as to obtain the question content: "how much money is paid for the fourth topic, xiaoming who bought 5 jin of apples, 3 yuan per jin of apples, and all xiaoming? ". The intelligent device can search the answering content (including question analysis, detailed answering steps and the like) corresponding to the questioning content in a pre-stored knowledge base according to the questioning content.

S3, the intelligent device generates projection information according to the answer content and projects the projection information to a preset projection area according to a projection mode selected by a user.

Specifically, the intelligent device may be provided with a projection module, such as a micro projector. The intelligent device can add corresponding content in the answer content to each version block of the projection template according to a preset projection template (comprising an answer version block, an analysis version block and a knowledge point version block), so as to generate projection information. The user can preset the projection mode (including LED projection, light projection, 3D holographic projection and AR projection) of the projection module, so that the projection module projects the projection information to the preset projection area according to the selected projection mode.

During projection, a desktop area can be detected from image information collected by the camera module through an image recognition technology, and the area is used as a preset projection area. In order to generate a better projection effect during projection, the projection plate can be added in a matched manner, when students study, the projection plate can be placed on a desk, the position of the projection plate can be automatically identified by a camera through an image identification technology, and meanwhile, the projection direction of the projection module is adjusted to project an image onto the projection plate.

The intelligent equipment can also be provided with a sight line detection device, the sight line detection device collects head images of a user, detects the sight line direction of the user according to the head images, and then determines a preset projection area in the sight line direction; or; the method comprises the steps that a sight line detection device on the intelligent equipment collects eye images of a user, detects the sight line direction of the user according to the eye images, and then determines a preset projection area in the sight line direction;

specifically, in order to facilitate the user to view the projected three-dimensional image, the sight line direction of the user may be detected by a sight line detection device. In the present embodiment, two examples of detecting the user gaze direction are provided:

1. the head image of a human body can be obtained through the sight line detection device, and each characteristic point (such as nose, eyes, cheekbone, eyebrow, mouth and the like) on the human face and the coordinates of the characteristic points in a world three-dimensional coordinate system are detected through a human face recognition technology, the linear direction formed by the two eyes is taken as the X coordinate axis direction, the linear direction formed by the middle point between the two eyes and the nose tip is taken as the Y coordinate axis direction, the eyes of a user are acquiescent to be in head-up front, and the direction which is vertical to the X coordinate axis and the Y coordinate axis is determined as the straight line where the sight line direction is located. Then the direction of the line of sight is confirmed with the face facing.

2. The visual line detection device comprises a shooting unit, a control unit and a control unit, wherein the shooting unit is used for shooting an eye image of a user and obtaining a zoom value of the eye image; a cornea determination unit that discriminates a cornea image from an eye image; a base point specification unit that specifies an eyeball center of a user based on the cornea image, and specifies an intersection point between a subject and a perpendicular line from the eyeball center to the subject as a base point; a distance measurement unit that specifies a zoom value indicating a predetermined size of an image of the cornea, and specifies a distance from the cornea to the subject based on the zoom value; a line of sight moving amount specifying unit that specifies a moving amount of the cornea based on a moving amount of the cornea image, and specifies a line of sight moving amount on the object based on the moving amount of the cornea and a distance from the cornea to the object; and an attention point specifying unit that specifies an attention point based on the reference point and the gaze movement amount, thereby determining the gaze direction of the user. The sight line detection device on the intelligent equipment can detect the sight line direction of the user, so that the determined preset projection area is in the watching range of the user, and the user can know the answering content very intuitively.

The scheme combines the lamps which are necessities used by students during learning, integrates the functions of acquiring questions, solving the questions, displaying the content of the solution and the like on the lamps, and can provide help for the students in time when the students encounter difficulty in learning. In this scheme, after the smart machine acquires the answer content, throw out the answer content with the mode of projection, need not external display screen or add display screen on lamps and lanterns, increase the universality of using.

As shown in fig. 2, the present invention further provides another embodiment of an interaction method based on an intelligent device, including the steps of:

s01, the intelligent device monitors the voice information of the user and detects whether the voice information contains a question keyword;

when a student encounters a problem in the process of learning or doing a question, the student can point to an unknown question and then say: similar questions such as "how this question is made" or "how the first question is made". The voice detection module in the intelligent device comprises a microphone and can acquire voice information of a user, and after the voice information of the user is acquired, the voice detection module converts the voice information of the user into text information; then, it is determined whether or not the text information contains question keywords (keywords such as "how to do", and the like).

S02, if the intelligent device detects that the voice information does not contain the question keywords, continuing to monitor the voice information of the user;

s03, if the intelligent equipment detects that the voice information contains the question keywords, starting a camera module;

s11, when the intelligent equipment identifies that the voice information of the user contains a question keyword, the intelligent equipment acquires an interface image of the book by taking the edge of the detected book as a boundary through the camera module;

specifically, in order to better recognize the book on the desktop, the book recognition model may be trained, and the embodiment takes training the Unet segmentation model as an example. And constructing the Unet segmentation network based on the tensoflow deep learning framework. In order to reduce the size of the split network and improve the efficiency of the split network, the existing lightweight mobilenet _ v2 network structure is modified and used for a unet network coding part, and a unet decoding part uses a simple convolutional neural network.

Before training, various single desktop images and various single book images can be collected, each single desktop image and each single book image are combined in a diversified mode (for example, the single book images are placed at different positions on the single book images), combined images of the single book images and the single desktop images are obtained, then the combined images are divided into a training set and a verification set according to the ratio of 4:1, the training set is trained through a Unet segmentation network, and when IOU (index of the segmentation network) values of the Unet segmentation network on the training set and the testing set are larger than 0.95, a Unet segmentation model is obtained.

After the camera acquires the desktop image, binary segmentation is carried out on the desktop image through the Unet segmentation model to obtain a binary segmentation map of the desktop image, the edge of the book is extracted from the binary segmentation map of the desktop image, the position of the book is determined, and the interface image of the book is acquired from the desktop image.

According to the scheme, the camera module is not started in real time, but is started under the condition that specific conditions are met, so that the energy consumption of the camera module is reduced.

S12, the intelligent device detects a target pointing area from the interface image of the book, and takes an image corresponding to the target pointing area as the question area image.

After the intelligent device obtains the interface image of the book, the fingers of the user can be identified from the interface image through an image identification technology, the area pointed by the fingertips is defined as a target pointing area, and then the image of the target pointing area is used as a question area image.

in this embodiment, after the intelligent device analyzes the question content from the question area image, the intelligent device may send the question content to the server. The server can pre-store a knowledge base, and the server can search corresponding answering contents in the knowledge base according to the questioning contents and return the answering contents to the intelligent equipment. In the technical scheme, the intelligent device does not need to store related data of a knowledge base, so that the requirement on the intelligent device storage is reduced, and the manufacturing cost of the intelligent device is reduced.

The intelligent device can be provided with a projection module, such as a micro projector. The intelligent device can add corresponding content in the answer content to each version block of the projection template according to a preset projection template (comprising an answer version block, an analysis version block and a knowledge point version block), so as to generate projection information. The user can preset the projection mode (including LED projection, light projection, 3D holographic projection and AR projection) of the projection module, and project the projection information to the preset projection area.

Preferably, in this embodiment, the intelligent device may analyze, according to the questioning content, to obtain a subject type to which the questioning content belongs; and according to the subject type of the questioning content, selecting a corresponding projection mode to project the answering content to the preset projection area.

Specifically, the intelligent terminal can carry out semantic understanding on the questioning content and analyze the subject type of the questioning content; the subject matching can also be carried out through subject keywords in the questioning content, for example, if a chemical formula appears in the question, the questioning content can be judged to be in a chemical category, and if a large amount of English appears in the question, the questioning content can be judged to be in an English category; or matching a corresponding book according to the question number of the questioning content, the page number of the page where the questioning content is located, or the position of the page where the questioning content is located, and then determining the subject category of the questioning content according to the subject category of the book.

In order to make the projection mode more conform to the characteristic of the subject to which the questioning content belongs, the corresponding projection mode is automatically selected for projection during projection.

For example, when the subject to which the questioning content belongs is english, since english is the answer of the character type, projection by using a plane projection method can achieve a better viewing effect; when the subject to which the questioning content belongs is the geometry, the geometry is a solid figure, and if the projection is performed in a plane projection mode, the students can difficultly understand the answers, and the purpose of solving the problems for the students cannot be achieved; and projection is carried out in a three-dimensional projection mode, students can understand the projected three-dimensional geometric figures from different angles, and the encountered problems are solved more easily.

For another example, when the subject to which the questioning content belongs is the geography, since many researches on mountainous regions, oceans, climate, etc. are performed geographically, if the answering content is projected in a planar projection manner, it may not be possible for students to have an intuitive experience. If the student is when understanding topography such as karst landform, danxia landform, coast landform, glacier landform, if with the relevant picture of plane projection mode projection, the student probably can not understand the characteristic of all kinds of landforms completely to be difficult to distinguish different landforms, and throw the content of answering with the three-dimensional projection mode, the student can follow different angles and observe various landforms, distinguish the characteristic of different landforms more easily.

As shown in fig. 3, the present invention provides an embodiment of an interaction method based on an intelligent device, including:

s11, when the intelligent equipment identifies that the voice information of the user contains a question keyword, the intelligent equipment opens a camera module, and obtains an interface image of the book by taking the detected edge of the book as a boundary;

s3, the intelligent device generates projection information according to the answer content and projects the projection information to a preset projection area according to a projection mode selected by a user, and optionally, when the projection mode selected by the user is an LED projection mode, the intelligent device projects the projection information to the preset projection area through a projection module in the LED projection mode; or when the user selects the projection mode to be light projection, the intelligent equipment projects the projection information to a preset projection area in a light projection mode through a projection module; or when the projection mode selected by the user is 3D holographic projection, the intelligent equipment projects the projection information to a preset projection area in a 3D holographic projection mode through a projection module; or when the projection mode selected by the user is AR projection, the intelligent device projects the projection information to a preset projection area in the AR projection mode through the projection module.

In the embodiment, a plurality of projection modes are provided for users to select, and the projection requirements of different projection contents can be met.

S41, the intelligent device records the question content to form a question record, and searches whether a question record similar to the question content exists in a history question record;

and S42, if a question record similar to the question content exists in the historical question record, projecting the question content and the answer content corresponding to the similar question record to the preset projection area.

Specifically, the intelligent device records questions asked by the user to form a question record, and when the user subsequently asks similar questions, the similar questions and answer contents can be pushed to the user. For example, when the question content smelled by the user is ____ "in the next sentence of" best up in whooping mountains "and ____" in the previous sentence of "the next floor higher than the last floor" that the user has asked before, the smart device projects the solution information to the user, and then projects the question content and the solution content corresponding to the similar question record to the preset projection area, thereby enhancing the user's awareness of the similar knowledge points.

Optionally, in this embodiment, the intelligent device analyzes the questioning content to obtain a subject to which the questioning content belongs, a specific chapter of the subject to which the questioning content belongs, and a specific knowledge point of the chapter to which the questioning content belongs, and records and forms a historical questioning record; the intelligent device generates learning evaluation reports of different disciplines, and/or learning evaluation reports of different sections in each discipline, and/or learning evaluation reports of different knowledge points in each section of each discipline according to historical questioning records in a preset time period.

After a user asks a question, the intelligent device records related question content and analyzes the question content to obtain a subject to which the question content belongs, a specific chapter of the subject to which the question content belongs and a specific knowledge point of the chapter to which the question content belongs. When a user wants to know the learning condition in the recent period of time, the intelligent device can generate a corresponding learning evaluation report according to the historical question record in the period of time, so that the user can be helped to know the current learning condition. For example, when a user needs a learning evaluation report of the subject of mathematics in the current period, the intelligent device can extract the history of the questions asked in the current period according to the history question records, classify the questions according to the affiliated sections and the affiliated knowledge points of the questions, and evaluate the learning condition of each section and each knowledge point of the user in the mathematical subject.

In addition, when a geometric figure or a plurality of stereoscopic images such as experimental equipment are required to be projected, the embodiment considers that the students often have operations such as drawing, connecting auxiliary lines, adding other figures and the like when doing geometric questions, and often needs to combine a plurality of equipment when doing experimental questions, meanwhile, in order to enable the students to view the projected geometric figure more completely and comprehensively, the experiment can be performed with more visual experience, the scheme increases editing operation when performing projection in a three-dimensional projection mode, and a user can perform corresponding editing on the projected three-dimensional image, so that the user can know learning contents better.

For example, when the time for touching any point on the three-dimensional image reaches a first preset time, or when any at least three points on the three-dimensional image are touched, the intelligent device analyzes the editing operation as an image dragging operation, and drags the three-dimensional image according to the image dragging operation of a user;

and when the time for touching any point on the three-dimensional image reaches second preset time, the intelligent equipment analyzes the editing operation as an image rotation operation, and rotates the three-dimensional image according to the image rotation operation of the user.

When any two points on the three-dimensional image are touched and the distance between the two touched points is reduced, the intelligent equipment analyzes the editing operation as an image reduction operation; when any two points on the three-dimensional image are touched and the distance between the two touched points is increased, the intelligent equipment analyzes the editing operation into an image amplification operation;

when the time for touching any point on the three-dimensional image reaches fourth preset time and the moving distance of the touched point is greater than a preset distance, the intelligent equipment analyzes the editing operation as a drawing operation;

when a user grabs a certain graph in the three-dimensional image by hands, the intelligent equipment analyzes that the editing operation is an image deleting operation; when the voice detection module 6 of the intelligent device monitors that the voice information of the user contains the graph adding information, a graph corresponding to the graph adding information is added to the three-dimensional image, for example, the user says "add cylinder", and when the intelligent device monitors the graph, the graph of the "cylinder" to be added can be analyzed through semantic understanding.

As shown in fig. 4, the present invention further provides an embodiment of an intelligent device, where the intelligent device includes a camera module 1, an analysis module 2, an answer obtaining module 3, a projection information generating module 4, and a projection module 5:

the camera module 1 is used for acquiring a question area image when the intelligent equipment identifies that the voice information of the user contains a question keyword;

the analysis module 2 is electrically connected with the camera module 1 and is used for analyzing the question content from the question area image acquired by the camera module 1;

the answer obtaining module 3 is electrically connected with the analysis module 2 and is used for obtaining answer contents corresponding to the questioning contents;

the projection information generating module 4 is electrically connected with the answer obtaining module 3 and is used for generating projection information according to the answer content;

and the projection module 5 is electrically connected with the projection information generation module 4 and is used for projecting the projection information to a preset projection area according to a projection mode selected by a user.

Specifically, the camera module 1 in the invention can be a camera and can be installed on an intelligent device, and the intelligent device comprises an intelligent desk lamp, an intelligent ceiling lamp, an intelligent wall lamp and the like and can be installed near a desk, so that the intelligent device can acquire image information of books on the desk through the camera module 1 installed on the body.

When a student learns, if the student encounters a subject which is unclear or can not be done, the student can point to the subject which is not going to be done by fingers, and then similar questions such as 'how to do the subject' or 'how to do the first subject' are spoken. The intelligent device can acquire voice information of a user in real time, convert the voice information into question text information, judge whether question keywords (such as 'how to do', 'how to say' and the like) exist in the text information, analyze images acquired by the camera when the intelligent device identifies that the voice information of the user contains the question keywords, identify the outline of a hand and the direction of the finger through an image identification technology, find out the area where the question the user wants to inquire on the finger direction area, and take the images of the area as the question area images.

An image processing chip (for example, an FH8510 image processing chip or an FMS6502 image processing chip) can be integrated in the intelligent device, and the image of the question area can be identified, and the question content in the image-text area image can be analyzed, for example, the image of the question area can be analyzed, so that the question content can be obtained: "how much money is paid for the fourth topic, xiaoming who bought 5 jin of apples, 3 yuan per jin of apples, and all xiaoming? ". The intelligent device can search the answering content (including question analysis, detailed answering steps and the like) corresponding to the questioning content in a pre-stored knowledge base according to the questioning content.

The intelligent device can add corresponding content in the answer content to each version block of the projection template according to a preset projection template (comprising an answer version block, an analysis version block and a knowledge point version block), so as to generate projection information. Can be equipped with projection module 5 on the smart machine, like miniature projector etc. the user can preset projection module 5's projection mode (including LED throws, light throws, 3D holographically projects, AR throws), after smart machine will answer content and generate projection information, can throw projection information to predetermineeing the projection area through projection module 5.

During projection, a desktop area can be detected from the image information collected by the camera module 1 through an image recognition technology, and the area is used as a preset projection area. In order to generate a better projection effect during projection, the projection plate can be added in a matched manner, when students study, the projection plate can be placed on a desk, the position of the projection plate can be automatically identified by a camera through an image identification technology, and meanwhile, the projection direction of the projection module is adjusted to project an image onto the projection plate.

In order to facilitate the user to view the projected image, a sight line detection device can be arranged on the intelligent device to detect the sight line direction of the user, and the preset projection area is determined in the sight line direction of the user. Specifically, the sight line detection device collects a head image of a user, detects a sight line direction of the user according to the head image, and then determines a preset projection area in the sight line direction; the sight line detection device on the intelligent equipment collects eye images of a user, detects the sight line direction of the user according to the eye images, and then determines a preset projection area in the sight line direction. In the present embodiment, two examples of detecting the user gaze direction are provided:

2. The visual line detection device comprises a shooting unit, a control unit and a control unit, wherein the shooting unit is used for shooting an eye image of a user and obtaining a zoom value of the eye image; a cornea determination unit that discriminates a cornea image from an eye image; a base point specification unit that specifies an eyeball center of a user based on the cornea image, and specifies an intersection point between a subject and a perpendicular line from the eyeball center to the subject as a base point; a distance measurement unit that specifies a zoom value indicating a predetermined size of an image of the cornea, and specifies a distance from the cornea to the subject based on the zoom value; a line of sight moving amount specifying unit that specifies a moving amount of the cornea based on a moving amount of the cornea image, and specifies a line of sight moving amount on the object based on the moving amount of the cornea and a distance from the cornea to the object; and an attention point specifying unit that specifies an attention point based on the reference point and the gaze movement amount, thereby determining the gaze direction of the user.

As shown in fig. 5, the present invention also provides an embodiment of an intelligent device, where the intelligent device includes:

the voice detection module 6 is used for monitoring voice information of a user and detecting whether the voice information contains a question keyword;

the voice detection module 6 is further configured to continue to monitor the voice information of the user if the intelligent device detects that the voice information does not contain the question keyword;

and the control module 7 is electrically connected with the voice detection module 6 and is used for starting the camera module 1 when the voice information is identified to contain the question keywords.

The camera module 1 is also electrically connected with the control module 7 and is used for acquiring an interface image of the book by taking the detected edge of the book as a boundary;

and the questioning area identification module 10 is electrically connected with the camera module 1, and is used for detecting a target pointing area from the interface image of the book and taking an image corresponding to the target pointing area as the questioning area image.

The analysis module 2 is electrically connected with the camera module 1 and the question area identification module 10 and is used for analyzing question content from the question area image acquired by the camera module 1;

the projection module 5 is configured to, when the user selects the projection mode to be LED projection, project the projection information to a preset projection area through the projection module 5 in an LED projection mode by the intelligent device;

or when the user selects the projection mode to be light projection, the intelligent device projects the projection information to a preset projection area in a light projection mode through the projection module 5;

or when the user selects the projection mode to be 3D holographic projection, the intelligent device projects the projection information to a preset projection area through the projection module 5 in a 3D holographic projection mode;

or when the user selects the projection mode to be AR projection, the intelligent device projects the projection information to a preset projection area in an AR projection mode through the projection module 5.

The recording module 8 is electrically connected with the analysis module 2 and is used for recording the question content to form a question record;

the searching module 9 is electrically connected with the recording module 8 and is used for searching whether a question record similar to the question content exists in the historical question record;

the projection module 5 is electrically connected to the search module 9, and is further configured to project, if a question record similar to the question content exists in the historical question record, the question content and the answer content corresponding to the similar question record to the preset projection area.

Specifically, when a student encounters a problem in the process of learning or doing a question, the student can point to a question which is not understood, and then: similar questions such as "how this question is made" or "how the first question is made". The voice detection module 6 in the intelligent device comprises a microphone, and can acquire the voice information of the user and convert the voice information of the user into text information; then, it is determined whether or not the text information contains question keywords (keywords such as "how to do", and the like).

When the intelligent device monitors that the voice information of the user contains a question keyword, the image of the question area is obtained through the camera module 1. Firstly, the camera module 1 of the intelligent device acquires an interface image of the book with the detected edge of the book as a boundary, detects a target pointing area from the interface image of the book, and takes an image corresponding to the target pointing area as the question area image. In order to better recognize the book on the desktop, the book recognition model can be trained, and the embodiment takes training the Unet segmentation model as an example. And constructing the Unet segmentation network based on the tensoflow deep learning framework. In order to reduce the size of the split network and improve the efficiency of the split network, the existing lightweight mobilenet _ v2 network structure is modified and used for a unet network coding part, and a unet decoding part uses a simple convolutional neural network.

Before training, various single desktop images and various single book images can be collected, each single desktop image and each single book image are combined in a diversified mode (for example, the single book images are placed at different positions on the single book images), combined images of the single book images and the single desktop images are obtained, then the combined images are divided into a training set and a verification set according to the ratio of 4:1, the training set is trained through a Unet segmentation network, and when IOU (index of error) values of the Unet network on the training set and the testing set are larger than 0.95, a Unet segmentation model is obtained.

After the camera module 1 acquires the desktop image, binary segmentation is performed on the desktop image through the Unet segmentation model to obtain a binary segmentation map of the desktop image, and the edge of the book is extracted from the binary segmentation map of the desktop image, so that the position of the book is determined, and the interface image of the book is acquired from the desktop image.

In this embodiment, after the intelligent device analyzes the question content from the question area image, the intelligent device may send the question content to the server, the server may store the knowledge base in advance, and the server may search the corresponding answer content in the knowledge base according to the question content and return the answer content to the intelligent device.

The intelligent device can be provided with a projection module 5, such as a micro projector. The intelligent device can add corresponding content in the answer content to each version block of the projection template according to a preset projection template (comprising an answer version block, an analysis version block and a knowledge point version block), so as to generate projection information. The user can preset the projection mode (including LED projection, light projection, 3D holographic projection, AR projection) of the projection module 5, and project the projection information to the preset projection area. In the embodiment, a plurality of projection modes are provided for users to select, and the projection requirements of different projection contents can be met.

The intelligent device records the questions asked by the user to form question records, and when the user subsequently asks similar questions, the similar questions and the answering content can be pushed to the user. For example, when the question content smelled by the user is ____ "in the next sentence of" best up in whooping mountains "and ____" in the previous sentence of "the next floor higher than the last floor" that the user has asked before, the smart device projects the solution information to the user, and then projects the question content and the solution content corresponding to the similar question record to the preset projection area, thereby enhancing the user's awareness of the similar knowledge points.

In addition, when the projection module 5 needs to project a geometric figure or three-dimensional images such as various experimental devices, the embodiment considers that the students often have operations such as drawing, connecting auxiliary lines, adding other figures and the like when doing geometric questions, and often needs to combine various devices when doing experimental questions, and meanwhile, in order to enable the students to view the projected geometric figure more completely and comprehensively, the experiment can be performed with more visual experience.

The intelligent equipment comprises an intelligent desk lamp, the camera module and the projection module are arranged at the head of the intelligent desk lamp, and the camera shooting direction of the camera module and the projection direction of the projection module face to the base of the intelligent desk lamp; or; the intelligent equipment comprises an intelligent wall lamp, the camera module and the projection module are arranged on the intelligent wall lamp, and the camera shooting direction of the camera module and the projection direction of the projection module face the preset projection area; or; the intelligent equipment comprises an intelligent ceiling lamp, the camera module and the projection module are arranged on the intelligent ceiling lamp, and the camera direction of the camera module and the projection direction of the projection module face the preset projection area.

As shown in fig. 6, the invention provides a schematic diagram of an intelligent desk lamp, wherein the intelligent desk lamp body comprises a desk lamp base, a connecting rod and a desk lamp head part; the camera is arranged at the desk lamp head part, and the camera shooting direction of the camera faces one side of the desk lamp base; the microphone and the loudspeaker are respectively arranged on the desk lamp base. When the students have difficulty in learning, the students can complete autonomous learning through the auxiliary learning function of the intelligent equipment in the embodiment.

It should be noted that the above embodiments can be freely combined as necessary. The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.

Claims

1. An interaction method based on intelligent equipment is characterized by comprising the following steps:

the intelligent equipment generates projection information according to the answer content and projects the projection information to a preset projection area according to a projection mode selected by a user;

when the intelligent equipment monitors that the voice information of the user contains a question keyword, the intelligent equipment acquires a question area image through the camera module, and the method also comprises the following steps:

the intelligent equipment monitors voice information of a user and detects whether the voice information contains a question keyword or not;

if the intelligent equipment detects that the voice information does not contain the question keywords, the voice information of the user is monitored continuously;

if the intelligent equipment detects that the voice information contains the question keywords, starting a camera module;

when the intelligent equipment identifies that the voice information of the user contains a question keyword, analyzing an image acquired by the camera module, identifying the outline of a hand and the direction of a finger by an image identification technology, finding out the area where the question the user wants to inquire is located on the finger direction area, and taking the image of the area as a question area image;

the intelligent equipment can also be provided with a sight line detection device, the sight line detection device collects head images of a user, detects the sight line direction of the user according to the head images, and then determines a preset projection area in the sight line direction;

or; the sight line detection device on the intelligent equipment collects eye images of a user, detects the sight line direction of the user according to the eye images, and then determines a preset projection area in the sight line direction.

2. The interaction method based on the intelligent device according to claim 1, wherein when the intelligent device recognizes that the voice information of the user contains a question keyword, the intelligent device obtains a question area image through the camera module, and the method specifically comprises the following steps:

3. The interaction method based on the intelligent device as claimed in claim 1 or 2, wherein the projection information is projected to a preset projection area in a projection manner selected by a user, and the step specifically comprises:

when the user selects the projection mode to be light projection, the intelligent equipment projects the projection information to a preset projection area in a light projection mode through a projection module;

when the projection mode selected by the user is 3D holographic projection, the intelligent equipment projects the projection information to a preset projection area in a 3D holographic projection mode through a projection module;

when the user selects the projection mode to be AR projection, the intelligent device projects the projection information to a preset projection area in the AR projection mode through a projection module.

4. The intelligent device based interaction method according to claim 1 or 2, characterized by further comprising the steps of:

5. A smart device, comprising:

the projection module is electrically connected with the projection information generation module and used for projecting the projection information to a preset projection area according to a projection mode selected by a user;

when the intelligent equipment identifies that the voice information of the user contains question keywords, the analysis module analyzes the image acquired by the camera module, identifies the outline of the hand and the pointing direction of the finger through an image identification technology, finds out the area where the question the user wants to inquire is located on the pointing area of the finger, and takes the image of the area as a question area image;

6. The smart device of claim 5, wherein:

the intelligent equipment comprises an intelligent desk lamp, the camera module and the projection module are arranged at the head of the intelligent desk lamp, and the camera shooting direction of the camera module and the projection direction of the projection module face one side of a base of the intelligent desk lamp;

or; the intelligent equipment comprises an intelligent wall lamp, the camera module and the projection module are arranged on the intelligent wall lamp, and the camera shooting direction of the camera module and the projection direction of the projection module face one side of the preset projection area;

or; the intelligent equipment comprises an intelligent ceiling lamp, the camera module and the projection module are arranged on the intelligent ceiling lamp, and the camera direction of the camera module and the projection direction of the projection module face one side of the preset projection area.

7. The smart device of claim 5, further comprising:

the control module is electrically connected with the voice detection module and used for starting the camera module when the voice information is identified to contain the question keywords;

8. A smart device as claimed in any one of claims 5 to 7, wherein:

the projection module is used for projecting the projection information to a preset projection area in an LED projection mode through the projection module by the intelligent equipment when the projection mode selected by the user is the LED projection;

9. The smart device of any one of claims 5-7, wherein the smart device further comprises: