CN114202741A - User learning monitoring method and device, computer equipment and storage medium - Google Patents
User learning monitoring method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN114202741A CN114202741A CN202111522585.XA CN202111522585A CN114202741A CN 114202741 A CN114202741 A CN 114202741A CN 202111522585 A CN202111522585 A CN 202111522585A CN 114202741 A CN114202741 A CN 114202741A
- Authority
- CN
- China
- Prior art keywords
- portrait
- user
- distance
- eyeball
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000012544 monitoring process Methods 0.000 title claims abstract description 34
- 210000005252 bulbus oculi Anatomy 0.000 claims abstract description 126
- 238000012806 monitoring device Methods 0.000 claims abstract description 6
- 238000004364 calculation method Methods 0.000 claims description 29
- 210000001747 pupil Anatomy 0.000 claims description 23
- 238000006073 displacement reaction Methods 0.000 claims description 7
- 210000001508 eye Anatomy 0.000 claims description 7
- 238000013528 artificial neural network Methods 0.000 claims description 4
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 210000001331 nose Anatomy 0.000 description 77
- 230000008569 process Effects 0.000 description 9
- 238000012549 training Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 6
- 238000011156 evaluation Methods 0.000 description 6
- 230000014509 gene expression Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 230000008921 facial expression Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012854 evaluation process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000003908 quality control method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/20—Education
- G06Q50/205—Education administration or guidance
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Human Resources & Organizations (AREA)
- General Physics & Mathematics (AREA)
- Strategic Management (AREA)
- Educational Administration (AREA)
- Health & Medical Sciences (AREA)
- Tourism & Hospitality (AREA)
- Economics (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Entrepreneurship & Innovation (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Molecular Biology (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- Development Economics (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- General Engineering & Computer Science (AREA)
- Educational Technology (AREA)
- Artificial Intelligence (AREA)
- Marketing (AREA)
- General Business, Economics & Management (AREA)
- Life Sciences & Earth Sciences (AREA)
- Quality & Reliability (AREA)
- Operations Research (AREA)
- Game Theory and Decision Science (AREA)
- Primary Health Care (AREA)
- Image Analysis (AREA)
Abstract
The application belongs to the technical field of artificial intelligence, is applied to the field of intelligent education, and relates to a monitoring method, a monitoring device, computer equipment and a storage medium for user learning, wherein the monitoring method comprises the steps of receiving a portrait data stream, positioning positions of a portrait eyeball and a portrait nose in the portrait data stream in real time, and obtaining an eyeball coordinate and a nose coordinate; calculating the distance from the eyeball of the portrait to the user side based on the eyeball coordinate to obtain a target distance, and calculating the distance from the nose of the portrait to the user side based on the nose coordinate to obtain a standard distance; calculating eyeball offset according to the target distance and the standard distance to obtain target offset; and scoring the learning condition of the user based on the portrait data stream and the target deviation degree to obtain a target score. The portrait data stream may be stored in a block chain. The method and the device realize effective monitoring of the learning condition of the user.
Description
Technical Field
The present application relates to the field of artificial intelligence, and in particular, to a method and an apparatus for monitoring user learning, a computer device, and a storage medium.
Background
With continuous innovation of computer technology, when an enterprise learns and trains employees, the enterprise often selects to learn and train the employees through online courses, and manpower and material resources are greatly saved. However, this method also has many problems, such as the situation that the employee can not be effectively learned in time.
Currently, there are multiple monitoring methods for the learning condition of the user, for example, the learning condition of the user is monitored by combining facial expressions and body movements of the user, but in these methods, monitoring of the facial expressions of the user needs to rely on a large number of expression training samples and models with excellent expression, and in the evaluation process, it is difficult to achieve effective monitoring due to the fact that the monitoring result of the user is not accurate enough due to the fact that the model learning effect is not good.
Disclosure of Invention
The embodiment of the application aims to provide a method and a device for monitoring user learning, computer equipment and a storage medium, so as to realize effective monitoring on the learning condition of a user.
In order to solve the above technical problem, an embodiment of the present application provides a monitoring method for user learning, which adopts the following technical solutions:
a monitoring method for user learning comprises the following steps:
receiving a portrait data stream, and positioning positions of a portrait eyeball and a portrait nose in real time from the portrait data stream to obtain an eyeball coordinate and a nose coordinate;
calculating the distance from the eyeball of the portrait to the user side based on the eyeball coordinate to obtain a target distance, and calculating the distance from the nose of the portrait to the user side based on the nose coordinate to obtain a standard distance;
calculating eyeball offset according to the target distance and the standard distance to obtain target offset;
and scoring the learning condition of the user based on the portrait data stream and the target deviation degree to obtain a target score.
Further, the user side positions the positions of the human image eyeballs and the human image nose from the human image data stream in real time, and the step of obtaining the eyeball coordinates and the nose coordinates comprises the following steps:
inputting the portrait data stream into a pre-trained multi-task cascade convolution neural network to obtain an output eyeball position and a nose position;
and determining the eyeball coordinates and the nose coordinates corresponding to the eyeball positions and the nose positions based on a preset virtual coordinate system.
Further, the step of locating the positions of the human image eyeballs and the human image nose in real time from the human image data stream to obtain the eyeball coordinates and the nose coordinates comprises:
carrying out portrait intercepting operation on a portrait video frame of the portrait data stream to obtain a face image;
converting the face image into a binary image, and identifying a plurality of circular images in the binary image based on Hough transform;
taking the two circular images with the same gray value in a preset gray threshold interval as the portrait eyeball, and taking the circle centers of the two circular images as the corresponding eyeball coordinates;
and determining the nose coordinate of the face image based on a preset face proportion and the eyeball coordinate.
Further, the step of calculating the eyeball displacement degree through the standard distance and the target distance to obtain the target displacement degree includes:
calculating the target offset degree according to the following formula:
wherein Ang represents the target offset degree, L1Represents the standard distance, L2Representing the target distance.
Further, the step of calculating the distance from the human figure eyeballs to the user side based on the eyeball coordinates to obtain the target distance includes:
calculating the interpupillary distance in the portrait data stream according to the eyeball coordinates to serve as the interpupillary distance of the portrait;
and acquiring an actual pupil distance and the front-end camera focal length, and calculating the distance from the eyeballs to the user side based on the actual pupil distance, the front-end camera focal length and the portrait pupil distance to obtain the target distance.
Further, the step of acquiring the actual interpupillary distance and the front-end camera focal length includes:
acquiring a user login name, calling prestored user personal information according to the user login name, and searching the mobile phone model of the user and the actual pupil distance of the user from the user personal information;
and searching the corresponding front-end camera focal length according to the user mobile phone model.
Further, the step of scoring the condition learned by the user based on the portrait data stream and the target offset degree to obtain a target score includes:
splitting the portrait data stream into a plurality of portrait periods according to a preset percentage, wherein each portrait period corresponds to a preset period value;
counting the occupation ratio time of each target offset angle in each portrait period, and determining a preset time interval to which the occupation ratio time belongs, wherein each time interval corresponds to a preset angle base number, and each target offset angle corresponds to a preset offset base number;
calculating scores corresponding to the portrait periods based on the period scores, the angle base, and the shift base, and taking the sum of the scores of all the portrait periods as the target score.
In order to solve the above technical problem, an embodiment of the present application further provides a monitoring device for user learning, which adopts the following technical solutions:
a monitoring device for user learning, comprising:
the receiving module is used for receiving the portrait data stream, positioning the positions of the eyeballs and the nose of the portrait in real time from the portrait data stream and obtaining the coordinates of the eyeballs and the nose;
the distance calculation module is used for calculating the distance from the eyeballs of the portrait to the user side based on the eyeball coordinates to obtain a target distance, calculating the distance from the nose of the portrait to the user side based on the nose coordinates to obtain a standard distance;
the offset calculation module is used for calculating eyeball offset according to the target distance and the standard distance to obtain target offset;
and the scoring module is used for scoring the learning condition of the user based on the portrait data flow and the target deviation degree to obtain a target score.
In order to solve the above technical problem, an embodiment of the present application further provides a computer device, which adopts the following technical solutions:
a computer device comprising a memory having computer readable instructions stored therein and a processor that implements the steps of the user learned monitoring method described above when executing the computer readable instructions.
In order to solve the above technical problem, an embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solutions:
a computer readable storage medium having computer readable instructions stored thereon which, when executed by a processor, implement the steps of the user learning monitoring method described above.
Compared with the prior art, the embodiment of the application mainly has the following beneficial effects:
according to the method and the device, models which need a large number of training samples and have high requirements on the accuracy of the models, such as the user limb models and the user expression models, do not need to be trained. The positions of eyeballs and noses are located in the portrait data stream to obtain eyeball coordinates and nose coordinates, and then target offset is calculated according to the eyeball coordinates and the nose coordinates, so that the learning condition of the user can be scored, the learning condition of the user can be monitored, and the effectiveness evaluation of the learning condition of the user can be realized through a rapid calculation mode.
Drawings
In order to more clearly illustrate the solution of the present application, the drawings needed for describing the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and that other drawings can be obtained by those skilled in the art without inventive effort.
FIG. 1 is an exemplary system architecture diagram in which the present application may be applied;
FIG. 2 is a flow diagram of one embodiment of a monitoring method for user learning according to the present application;
FIG. 3 is a schematic block diagram of one embodiment of a user learned monitoring device according to the present application;
FIG. 4 is a schematic block diagram of one embodiment of a computer device according to the present application.
Reference numerals: 200. a computer device; 201. a memory; 202. a processor; 203. a network interface; 300. a monitoring device for user learning; 301. a receiving module; 302. a distance calculation module; 303. an offset calculation module; 304. and a scoring module.
Detailed Description
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "including" and "having," and any variations thereof, in the description and claims of this application and the description of the above figures are intended to cover non-exclusive inclusions. The terms "first," "second," and the like in the description and claims of this application or in the above-described drawings are used for distinguishing between different objects and not for describing a particular order.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
As shown in fig. 1, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The terminal devices 101, 102, 103 may have various communication client applications installed thereon, such as a web browser application, a shopping application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, mpeg compression standard Audio Layer 3), MP4 players (Moving Picture Experts Group Audio Layer IV, mpeg compression standard Audio Layer 4), laptop portable computers, desktop computers, and the like.
The server 105 may be a server providing various services, such as a background server providing support for pages displayed on the terminal devices 101, 102, 103.
It should be noted that the monitoring method for user learning provided in the embodiment of the present application is generally executed by a server/terminal device, and accordingly, the monitoring apparatus for user learning is generally disposed in the server/terminal device.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
When the user side recognizes that the user clicks a video playing button of a learning page, the user is determined to be in a learning training scene, a camera opening request is popped out from a front-end page, so that the user can apply permission to the user, and a front-facing camera is opened to capture the portrait data stream. And if the approval key of the camera opening request is triggered, determining that the user approves to open the camera, and then opening the front camera by the user side. And if the agreement key of the camera opening request is not triggered, waiting for a preset time interval and then retransmitting the camera opening request. And if the preset times are reached and the approval key of the camera opening request is still not triggered, exiting the learning page and sending a preposed camera opening failure signal to the server. And the user side sends the portrait data stream obtained by the camera to the server.
With continued reference to FIG. 2, a flow diagram of one embodiment of a user learned monitoring method according to the present application is shown. The monitoring method for user learning comprises the following steps:
s1: and receiving a portrait data stream, positioning positions of the eyeballs and the noses of the portrait in real time from the portrait data stream, and obtaining the coordinates of the eyeballs and the coordinates of the noses.
In this embodiment, the positions of the human figure eyeballs and the human figure nose are located in real time from the human figure data stream, and the eyeball coordinates and the nose coordinates are obtained according to a preset virtual coordinate system for the subsequent calculation process.
Specifically, in step S1, the step of obtaining the eye coordinates and the nose coordinates by locating the positions of the human image eyes and the human image nose in real time from the human image data stream includes:
inputting the portrait data stream into a pre-trained multi-task cascade convolution neural network to obtain an output eyeball position and a nose position;
and determining the eyeball coordinates and the nose coordinates corresponding to the eyeball positions and the nose positions based on a preset virtual coordinate system.
In this embodiment, MTCNN (multitask cascaded convolutional neural network) consists of three parts, P-Net (pro positive network), R-Net (refineNet), O-Net (outputNet). The three parts are three network structures which are independent from each other and are in series connection with each other. The network of each stage is a multitask network, and the processing tasks are three: judging the human face/non-human face, regression of a human face frame and positioning of feature points. MTCNN has two advantages of time and performance compared with other face recognition models.
Furthermore, in step S1, the step of locating the positions of the human image eyeballs and the human image nose in real time from the human image data stream, obtaining the eyeball coordinates and the nose coordinates comprises:
carrying out portrait intercepting operation on a portrait video frame of the portrait data stream to obtain a face image;
converting the face image into a binary image, and identifying a plurality of circular images in the binary image based on Hough transform;
taking the two circular images with the same gray value in a preset gray threshold interval as the portrait eyeball, and taking the circle centers of the two circular images as the corresponding eyeball coordinates;
and determining the nose coordinate of the face image based on a preset face proportion and the eyeball coordinate.
In this embodiment, a face of the portrait data stream is detected to obtain a face image; carrying out image binarization operation on the face image to obtain a binarized image; identifying a plurality of circular images in the binarized image based on Hough transform; hough Transform (Hough Transform) Hough Transform is one of basic methods for recognizing geometric shapes from images in image processing, is widely applied, and has a plurality of improved algorithms. Primarily to isolate geometric shapes (e.g., lines, circles, etc.) having certain identical characteristics from the image. After the face image is converted into the binary image, if the gray value of the eyeball is higher than that of the skin, the two circular images with the same gray value in a preset gray threshold interval are used as the human image eyeball, and the circle centers of the two circular images are used as the corresponding eyeball coordinates. And calculating the position of the nose in the face image based on a preset face proportion and the eyeball coordinates, and further determining the nose coordinates according to a virtual coordinate system. The specific process is described by combining an example: and taking the mean value of the horizontal coordinates of the two eyeball coordinates as the horizontal coordinate of the nose coordinate, calculating according to the two eyeball coordinates to obtain the pupil distance of the eyeballs as A, and determining the distance B from the middle point between the two eyeballs to the nose tip according to the face proportion and the pupil distance A. The ordinate of the nose coordinate is calculated from the ordinate of the eyeball coordinate and the distance B (for example, the ordinate of the eyeball coordinate minus the distance B is the ordinate of the nose coordinate), and the nose coordinate is obtained.
S2: and calculating the distance from the eyeballs of the portrait to the user side based on the eyeball coordinates to obtain a target distance, calculating the distance from the nose of the portrait to the user side based on the nose coordinates to obtain a standard distance.
In this embodiment, the distance between the nose and the device at the user end is used as the standard distance for calculating the target offset. Wherein, calculate the distance from nose to user's end through the nose coordinate, obtain the concrete step of standard distance and include: calculating the distance between two nostrils of the nose in the image to obtain the image nose distance; and calculating to obtain a standard distance according to the image nose distance, the front-end camera focal length and the actual nose distance obtained in advance according to the user login ID, wherein the specific calculation process is the following calculation process of the target distance, and the detailed description is omitted here. In addition, the method and the device can also detect the straight line between the camera and the eyes of the human face as a standard line according to the straight line between the camera and the nose of the human face as a pupil line, calculate the included angle between the standard line and the pupil line as the target offset degree, and are the same as the process.
Specifically, in step S2, the step of calculating the distance from the human image eyeball to the user end based on the eyeball coordinate to obtain the target distance includes:
calculating the interpupillary distance in the portrait data stream according to the eyeball coordinates to serve as the interpupillary distance of the portrait;
and acquiring an actual pupil distance and the front-end camera focal length, and calculating the distance from the eyeballs to the user side based on the actual pupil distance, the front-end camera focal length and the portrait pupil distance to obtain the target distance.
In the present embodiment, the distance of the two eyeballs is calculated from the two eyeball coordinates as the interpupillary distance of the human figure. And then based on the principle of similarity of triangles, calculating the distance from the eyeballs to the user side through the actual interpupillary distance, the front-end camera focal length and the portrait interpupillary distance. The concrete formula is as follows: the pupil distance/actual pupil distance of the portrait is equal to the focal length/target distance of the front-end camera. And the focal length of the front-end camera is the information which is called and prestored. The actual pupil distance is extracted from the user personal information which is called and correspondingly stored according to the user login ID.
Wherein, the step of obtaining actual interpupillary distance and front end camera focal length includes:
acquiring a user login name, calling prestored user personal information according to the user login name, and searching the mobile phone model of the user and the actual pupil distance of the user from the user personal information;
and searching the corresponding front-end camera focal length according to the user mobile phone model.
In this embodiment, the model of the mobile phone of the user and the actual interpupillary distance are found through the personal information of the user, and the focal length of the front-end camera of the mobile phone is found and determined in the webpage according to the model of the mobile phone of the user.
S3: and calculating the eyeball offset according to the target distance and the standard distance to obtain the target offset.
In this embodiment, the human image data stream is subjected to an eyeball positioning operation, and the positions of two eyeballs are positioned in real time. Calculating the eyeball displacement degree based on the eyeball coordinates and the nose coordinates, and collecting and storing the data at the user end (i.e. the mobile end). Specifically, positions of eyeballs and noses of each frame of image of the portrait data stream are located in real time, eyeball coordinates and nose coordinates of each frame of image are obtained respectively, eyeball offset degrees of each frame of image are calculated respectively on the basis of the eyeball coordinates and the nose coordinates, and target offset degrees of each frame of image are obtained respectively. The method and the device respectively calculate the target offset of each frame of image in the portrait data stream for subsequent calculation of the target score.
Specifically, in step S3, the step of calculating the eyeball displacement degree through the standard distance and the target distance to obtain the target displacement degree includes:
calculating the target offset degree according to the following formula:
wherein Ang represents the target offset degree, L1Represents the standard distance, L2Representing the target distance.
In this embodiment, by the calculation formula:
the value of cosAng can be calculated to obtain the specific angle value of Ang, i.e. the target offset.
S4: and scoring the learning condition of the user based on the portrait data stream and the target deviation degree to obtain a target score.
In this embodiment, the service center is a backend server or a cloud server of the present application. The service center receives the portrait data stream and the target offset, and further scores the whole portrait data stream through the target offset of each frame of image of the portrait data stream, namely scores the learning condition of the user to obtain a target score. The course configurator may adjust the quality of the course learned by the user based on the target score.
Specifically, in step S4, the step of scoring the situation learned by the user based on the portrait data stream and the target offset degree to obtain the target score includes:
splitting the portrait data stream into a plurality of portrait periods according to a preset percentage, wherein each portrait period corresponds to a preset period value;
counting the occupation ratio time of each target offset angle in each portrait period, and determining a preset time interval to which the occupation ratio time belongs, wherein each time interval corresponds to a preset angle base number, and each target offset angle corresponds to a preset offset base number;
calculating scores corresponding to the portrait periods based on the period scores, the angle base, and the shift base, and taking the sum of the scores of all the portrait periods as the target score.
In this embodiment, the calculation formula of the target score is as follows:
TABLE 1
TABLE 2
| Offset angle (%) | Offset cardinality |
| 10～20 | 0.9 |
| 20～30 | 0.8 |
| 30～40 | 0.7 |
| 40～50 | 0.6 |
| 50～60 | 0.5 |
| 60～70 | 0.4 |
| 70～80 | 0.3 |
| 80～90 | 0.2 |
| 90～100 | 0.1 |
TABLE 3
In this embodiment, the electronic device (for example, the service center shown in fig. 1) on which the monitoring method learned by the user operates may receive the portrait data stream and the target offset transmitted by the user terminal through a wired connection manner or a wireless connection manner. It should be noted that the wireless connection means may include, but is not limited to, a 3G/4G connection, a WiFi connection, a bluetooth connection, a WiMAX connection, a Zigbee connection, a uwb (ultra wideband) connection, and other wireless connection means now known or developed in the future.
The method further comprises the step of judging whether the target score exceeds a standard score corresponding to the course, and if so, determining that the quality of the course reaches the standard. According to the method, the quality of the course can be effectively and positively fed back according to the comprehensive score of the learning and training, for example, if the target score (9 points) exceeds the standard score (8 points) corresponding to the course, the content of the course is determined to be attractive, and the quality of the course reaches the standard. And if the target score does not exceed the standard score, determining that the course quality does not reach the standard. An effective passive feedback means is not available in a target online learning and training scene, and the quality control of courses is basically dependent on the postclass comment of a user or questionnaire content survey. No effective technology reflects the quality of a certain segment of the training course. According to the method and the device, data analysis is carried out by utilizing the human face eyeball excursion degree in the learning process of the user, an effective quality feedback scheme for covering the whole segment of a single course is provided, and subsequently, optimization and perfection can be carried out on a certain segment under the course so as to improve the overall quality of the course.
In addition, the method and the device can further refine to evaluate the quality of a certain course, for example, the score of the course time period (namely, the segmented video content) exceeds 80% of the preset standard score corresponding to the course time period, wherein the score of the course time period is the score of the portrait time period corresponding to the course time period.
According to the method and the device, models which need a large number of training samples and have high requirements on the accuracy of the models, such as the user limb models and the user expression models, do not need to be trained. The positions of eyeballs and noses are located in the portrait data stream to obtain eyeball coordinates and nose coordinates, and then target offset is calculated according to the eyeball coordinates and the nose coordinates, so that the learning condition of the user can be scored, the learning condition of the user can be monitored, and the effectiveness evaluation of the learning condition of the user can be realized through a rapid calculation mode.
It is emphasized that the portrait data stream may also be stored in a node of a blockchain in order to further ensure privacy and security of the portrait data stream.
The block chain referred by the application is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
This application can be applied to in the wisdom education field to promote the construction in wisdom city.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware associated with computer readable instructions, which can be stored in a computer readable storage medium, and when executed, can include processes of the embodiments of the methods described above. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
With further reference to fig. 3, as an implementation of the method shown in fig. 2, the present application provides an embodiment of a monitoring apparatus for user learning, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus may be applied to various electronic devices.
As shown in fig. 3, the monitoring apparatus 300 for user learning according to the present embodiment includes: the system comprises a receiving module 301, a distance calculating module 302, an offset calculating module 303 and a scoring module 304, wherein the receiving module 301 is used for receiving a portrait data stream, positioning positions of a portrait eyeball and a portrait nose in real time from the portrait data stream, and obtaining eyeball coordinates and nose coordinates; the distance calculation module 302 is configured to calculate a distance from each of the human figure eyeballs to a user side based on the eyeball coordinates to obtain a target distance, and calculate a distance from each of the human figure noses to the user side based on the nose coordinates to obtain a standard distance; the offset calculation module 303 is configured to calculate an eyeball offset according to the target distance and the standard distance to obtain a target offset; the scoring module 304 is configured to score a condition learned by a user based on the portrait data stream and the target offset to obtain a target score.
In the embodiment, models which need a large number of training samples and have high requirements on the accuracy of the models, such as a user limb model and a user expression model, do not need to be trained, the positions of eyeballs and noses only need to be positioned in a portrait data stream, the eyeball coordinates and the nose coordinates are obtained, and then the target offset is calculated according to the eyeball coordinates and the nose coordinates, so that the target offset is used for scoring the learning condition of the user, the learning condition of the user is monitored, and the effectiveness evaluation of the learning condition of the user is realized through a rapid calculation mode.
The receiving module 301 includes an input sub-module and a determination sub-module, where the input sub-module is configured to input the portrait data stream into a pre-trained multitask cascade convolution neural network to obtain an eyeball position and a nose position of the output; the determining submodule is used for determining the eyeball coordinates and the nose coordinates corresponding to the eyeball position and the nose position based on a preset virtual coordinate system.
The receiving module 301 further includes an intercepting submodule, a converting submodule, an eyeball coordinate generating submodule, and a nose coordinate generating submodule, where the intercepting submodule is configured to perform an image intercepting operation on an image video frame of the image data stream to obtain a face image; the conversion submodule is used for converting the face image into a binary image and identifying a plurality of circular images in the binary image based on Hough transform; the eyeball coordinate generation submodule is used for taking two circular images with the same gray value in a preset gray threshold interval as the portrait eyeball and taking the circle centers of the two circular images as the corresponding eyeball coordinates; the nose coordinate generation submodule is used for determining the nose coordinate of the face image based on a preset face proportion and the eyeball coordinate.
The distance calculation module 302 comprises a first calculation submodule and a second calculation submodule, wherein the first calculation submodule is used for calculating the interpupillary distance in the portrait data stream according to the eyeball coordinates to serve as the portrait interpupillary distance; and the second calculation submodule is used for acquiring an actual pupil distance and the front-end camera focal length, and calculating the distance from the eyeballs to the user side based on the actual pupil distance, the front-end camera focal length and the portrait pupil distance to obtain the target distance.
The second calculation submodule comprises an acquisition unit and a search unit, wherein the acquisition unit is used for acquiring a user login name, calling prestored user personal information according to the user login name, and searching the mobile phone model of the user and the actual pupil distance of the user from the user personal information; the searching unit is used for searching the corresponding front-end camera focal length according to the user mobile phone model.
In some optional implementations of the present embodiment, the offset calculation module 303 is further configured to: calculating the target offset degree according to the following formula:
wherein Ang represents the target offset degree, L1Represents the standard distance, L2Representing the target distance.
The scoring module 304 comprises a splitting submodule, a counting submodule and a target score calculating submodule, wherein the splitting submodule is used for splitting the portrait data stream into a plurality of portrait periods according to a preset percentage, and each portrait period corresponds to a preset period score; the counting submodule is used for counting the ratio duration of each target offset angle in each portrait period and determining a preset duration interval to which the ratio duration belongs, wherein each duration interval corresponds to a preset angle base number, and each target offset angle corresponds to a preset offset base number; the target score calculation sub-module is used for calculating scores corresponding to the portrait periods based on the period scores, the angle base and the offset base, and taking the sum of the scores of all the portrait periods as the target score.
According to the method and the device, models which need a large number of training samples and have high requirements on the accuracy of the models, such as the user limb models and the user expression models, do not need to be trained. The positions of eyeballs and noses are located in the portrait data stream to obtain eyeball coordinates and nose coordinates, and then target offset is calculated according to the eyeball coordinates and the nose coordinates, so that the learning condition of the user can be scored, the learning condition of the user can be monitored, and the effectiveness evaluation of the learning condition of the user can be realized through a rapid calculation mode.
In order to solve the technical problem, an embodiment of the present application further provides a computer device. Referring to fig. 4, fig. 4 is a block diagram of a basic structure of a computer device according to the present embodiment.
The computer device 200 comprises a memory 201, a processor 202, a network interface 203 communicatively connected to each other via a system bus. It is noted that only computer device 200 having components 201 and 203 is shown, but it is understood that not all of the illustrated components are required and that more or fewer components may alternatively be implemented. As will be understood by those skilled in the art, the computer device is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and the hardware includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.
The computer device can be a desktop computer, a notebook, a palm computer, a cloud server and other computing devices. The computer equipment can carry out man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch panel or voice control equipment and the like.
The memory 201 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 201 may be an internal storage unit of the computer device 200, such as a hard disk or a memory of the computer device 200. In other embodiments, the memory 201 may also be an external storage device of the computer device 200, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like, provided on the computer device 200. Of course, the memory 201 may also include both internal and external storage devices of the computer device 200. In this embodiment, the memory 201 is generally used for storing an operating system installed in the computer device 200 and various application software, such as computer readable instructions of a monitoring method learned by a user. Further, the memory 201 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 202 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 202 is generally operative to control overall operation of the computer device 200. In this embodiment, the processor 202 is configured to execute computer readable instructions stored in the memory 201 or process data, such as computer readable instructions for executing the monitoring method learned by the user.
The network interface 203 may comprise a wireless network interface or a wired network interface, and the network interface 203 is generally used for establishing communication connection between the computer device 200 and other electronic devices.
In this embodiment, positions of eyeballs and a nose are located in a portrait data stream, an eyeball coordinate and a nose coordinate are obtained, and then a target offset is calculated according to the eyeball coordinate and the nose coordinate, so that the learning condition of a user is scored, the learning condition of the user is monitored, and the effectiveness evaluation of the learning condition of the user is realized through a quick calculation mode.
The present application provides yet another embodiment, which is a computer-readable storage medium having computer-readable instructions stored thereon which are executable by at least one processor to cause the at least one processor to perform the steps of the user-learned monitoring method as described above.
In this embodiment, positions of eyeballs and a nose are located in a portrait data stream, an eyeball coordinate and a nose coordinate are obtained, and then a target offset is calculated according to the eyeball coordinate and the nose coordinate, so that the learning condition of a user is scored, the learning condition of the user is monitored, and the effectiveness evaluation of the learning condition of the user is realized through a quick calculation mode.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present application.
It is to be understood that the above-described embodiments are merely illustrative of some, but not restrictive, of the broad invention, and that the appended drawings illustrate preferred embodiments of the invention and do not limit the scope of the invention. This application is capable of embodiments in many different forms and is provided for the purpose of enabling a thorough understanding of the disclosure of the application. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to one skilled in the art that the present application may be practiced without modification or with equivalents of some of the features described in the foregoing embodiments. All equivalent structures made by using the contents of the specification and the drawings of the present application are directly or indirectly applied to other related technical fields and are within the protection scope of the present application.
Claims (10)
1. A method for monitoring user learning, comprising the steps of:
receiving a portrait data stream, and positioning positions of a portrait eyeball and a portrait nose in real time from the portrait data stream to obtain an eyeball coordinate and a nose coordinate;
calculating the distance from the eyeball of the portrait to the user side based on the eyeball coordinate to obtain a target distance, and calculating the distance from the nose of the portrait to the user side based on the nose coordinate to obtain a standard distance;
calculating eyeball offset according to the target distance and the standard distance to obtain target offset;
and scoring the learning condition of the user based on the portrait data stream and the target deviation degree to obtain a target score.
2. The method for monitoring learning of a user as claimed in claim 1, wherein the step of locating the positions of the human eye and the human nose in real time from the human data stream to obtain the eye coordinates and the nose coordinates comprises:
inputting the portrait data stream into a pre-trained multi-task cascade convolution neural network to obtain an output eyeball position and a nose position;
and determining the eyeball coordinates and the nose coordinates corresponding to the eyeball positions and the nose positions based on a preset virtual coordinate system.
3. The method for monitoring learning of a user as claimed in claim 1, wherein the step of locating the positions of the human eye and the human nose in real time from the human data stream to obtain the eye coordinates and the nose coordinates comprises:
carrying out portrait intercepting operation on a portrait video frame of the portrait data stream to obtain a face image;
converting the face image into a binary image, and identifying a plurality of circular images in the binary image based on Hough transform;
taking the two circular images with the same gray value in a preset gray threshold interval as the portrait eyeball, and taking the circle centers of the two circular images as the corresponding eyeball coordinates;
and determining the nose coordinate of the face image based on a preset face proportion and the eyeball coordinate.
4. The method as claimed in claim 1, wherein the step of calculating the degree of eyeball displacement from the standard distance and the target distance to obtain the degree of target displacement comprises:
calculating the target offset degree according to the following formula:
wherein Ang represents the target offset degree, L1Represents the standard distance, L2Representing the target distance.
5. The method as claimed in claim 1, wherein the step of calculating the distance between the human image eyeball and the user end based on the eyeball coordinate to obtain the target distance comprises:
calculating the interpupillary distance in the portrait data stream according to the eyeball coordinates to serve as the interpupillary distance of the portrait;
and acquiring an actual pupil distance and the front-end camera focal length, and calculating the distance from the eyeballs to the user side based on the actual pupil distance, the front-end camera focal length and the portrait pupil distance to obtain the target distance.
6. The method for monitoring learning of a user according to claim 5, wherein the step of obtaining the actual interpupillary distance and the front-end camera focal length comprises:
acquiring a user login name, calling prestored user personal information according to the user login name, and searching the mobile phone model of the user and the actual pupil distance of the user from the user personal information;
and searching the corresponding front-end camera focal length according to the user mobile phone model.
7. The method for monitoring user learning according to claim 1, wherein the step of scoring the user learning condition based on the human image data stream and the target bias degree to obtain a target score comprises:
splitting the portrait data stream into a plurality of portrait periods according to a preset percentage, wherein each portrait period corresponds to a preset period value;
counting the occupation ratio time of each target offset angle in each portrait period, and determining a preset time interval to which the occupation ratio time belongs, wherein each time interval corresponds to a preset angle base number, and each target offset angle corresponds to a preset offset base number;
calculating scores corresponding to the portrait periods based on the period scores, the angle base, and the shift base, and taking the sum of the scores of all the portrait periods as the target score.
8. A monitoring device for user learning, comprising:
the receiving module is used for receiving the portrait data stream, positioning the positions of the eyeballs and the nose of the portrait in real time from the portrait data stream and obtaining the coordinates of the eyeballs and the nose;
the distance calculation module is used for calculating the distance from the eyeballs of the portrait to the user side based on the eyeball coordinates to obtain a target distance, calculating the distance from the nose of the portrait to the user side based on the nose coordinates to obtain a standard distance;
the offset calculation module is used for calculating eyeball offset according to the target distance and the standard distance to obtain target offset;
and the scoring module is used for scoring the learning condition of the user based on the portrait data flow and the target deviation degree to obtain a target score.
9. A computer device comprising a memory having computer readable instructions stored therein and a processor which when executed implements the steps of the user learned monitoring method of any one of claims 1 to 7.
10. A computer readable storage medium having computer readable instructions stored thereon which, when executed by a processor, implement the steps of the user learned monitoring method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111522585.XA CN114202741A (en) | 2021-12-13 | 2021-12-13 | User learning monitoring method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111522585.XA CN114202741A (en) | 2021-12-13 | 2021-12-13 | User learning monitoring method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114202741A true CN114202741A (en) | 2022-03-18 |
Family
ID=80653287
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111522585.XA Pending CN114202741A (en) | 2021-12-13 | 2021-12-13 | User learning monitoring method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114202741A (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105759971A (en) * | 2016-03-08 | 2016-07-13 | 珠海全志科技股份有限公司 | Method and system for automatically prompting distance from human eyes to screen |
| CN111679736A (en) * | 2020-05-22 | 2020-09-18 | 中国长城科技集团股份有限公司 | Method and device for adjusting screen brightness of electronic equipment and terminal equipment |
-
2021
- 2021-12-13 CN CN202111522585.XA patent/CN114202741A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105759971A (en) * | 2016-03-08 | 2016-07-13 | 珠海全志科技股份有限公司 | Method and system for automatically prompting distance from human eyes to screen |
| CN111679736A (en) * | 2020-05-22 | 2020-09-18 | 中国长城科技集团股份有限公司 | Method and device for adjusting screen brightness of electronic equipment and terminal equipment |
Non-Patent Citations (1)
| Title |
|---|
| 田丰 等: "《人工智能与电影特效制作及应用》", vol. 1, 31 October 2021, 上海科学技术出版社, pages: 87 - 88 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10860839B2 (en) | Method and apparatus for generating training data for human face recognition, device and computer storage medium | |
| EP3477519B1 (en) | Identity authentication method, terminal device, and computer-readable storage medium | |
| CN107680019B (en) | Examination scheme implementation method, device, equipment and storage medium | |
| US11436863B2 (en) | Method and apparatus for outputting data | |
| US20180308107A1 (en) | Living-body detection based anti-cheating online research method, device and system | |
| WO2022105118A1 (en) | Image-based health status identification method and apparatus, device and storage medium | |
| WO2021139316A1 (en) | Method and apparatus for establishing expression recognition model, and computer device and storage medium | |
| CN113254491A (en) | Information recommendation method and device, computer equipment and storage medium | |
| WO2018000270A1 (en) | Personalized response generating method and system based on profile | |
| WO2022156084A1 (en) | Method for predicting behavior of target object on the basis of face and interactive text, and related device | |
| CN113707299A (en) | Auxiliary diagnosis method and device based on inquiry session and computer equipment | |
| CN112417121A (en) | Client intention recognition method and device, computer equipment and storage medium | |
| CN113743362A (en) | Method for correcting training action in real time based on deep learning and related equipment thereof | |
| CN113705534A (en) | Behavior prediction method, behavior prediction device, behavior prediction equipment and storage medium based on deep vision | |
| CN115510186A (en) | Instant question and answer method, device, equipment and storage medium based on intention recognition | |
| CN112507141A (en) | Investigation task generation method and device, computer equipment and storage medium | |
| CN116681045A (en) | Report generation method, report generation device, computer equipment and storage medium | |
| CN113343898B (en) | Mask shielding face recognition method, device and equipment based on knowledge distillation network | |
| CN111835861B (en) | Examination system data processing method and device, computer equipment and storage medium | |
| CN114202741A (en) | User learning monitoring method and device, computer equipment and storage medium | |
| CN114724072A (en) | Intelligent question pushing method, device, equipment and storage medium | |
| CN112905748A (en) | Speech effect evaluation system | |
| CN112733645A (en) | Handwritten signature verification method and device, computer equipment and storage medium | |
| CN114220055B (en) | Method, device, computer equipment and storage medium for assisting user interview | |
| TWI826767B (en) | Content-based dialogue method, computer-readable recording medium, and computer device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |