CN111341444B - Intelligent painting scoring method and system - Google Patents
Intelligent painting scoring method and system Download PDFInfo
- Publication number
- CN111341444B CN111341444B CN201911406860.4A CN201911406860A CN111341444B CN 111341444 B CN111341444 B CN 111341444B CN 201911406860 A CN201911406860 A CN 201911406860A CN 111341444 B CN111341444 B CN 111341444B
- Authority
- CN
- China
- Prior art keywords
- user
- eye movement
- client
- score
- painting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/30—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for calculating health indices; for individual health risk assessment
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
- G16H20/70—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to mental therapies, e.g. psychological therapy or autogenous training
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The invention discloses an intelligent painting scoring method and system, wherein the method comprises the following steps: the client provides a reference picture for the user to select; receiving and storing images and lines drawn by a user in a drawing area; collecting eye movement information of a user and facial features of the user; transmitting an image and a line drawn by a user in a drawing area, eye movement information of the user and facial features of the user to a processing center; after the processing center receives the information, analyzing to obtain a painting score, an eye movement tracking score and a mood score, and synthesizing the painting score, the eye movement tracking score and the mood score to obtain a risk assessment result; the processing center sends the painting score, the eye movement tracking score, the emotion state and the risk assessment result to the client; the client receives and displays. The invention can avoid the limitation of time and space to the user, so as to obtain the test result of the user in a relaxed state, and provide quantitative reference for various diseases related behaviors possibly occurring in the painting test process.
Description
Technical Field
The invention relates to the technical field of computer image processing, in particular to an intelligent painting scoring method and system.
Background
Three brain science diseases facing the world at present: autism (autism) in children, depression in adults, and Alzheimer's disease (commonly known as senile dementia) in the elderly. The brain science problem also reflects the social concern about mental diseases, childhood development and aging of the elderly, and the abnormality of the human brain is often reflected by human behavior.
The painting score itself is currently used in medical clinic to assist doctors in judging the development degree of children, or assist diagnosis of degenerative diseases and mental diseases of old people, such as vision problems, muscle control related diseases (such as parkinsonism, cerebral palsy and the like) and also evaluation of psychological states.
Eye tracking techniques are used in scientific and medical clinics for psychological and behavioral analysis. Eye movement tracking, assessment of the attention of a user, assessment of the degree of fit of the user to a pictorial test, may be used for diseases related to nystagmus (e.g. epilepsy, schizophrenia, eye tremors, etc.), attention-related diseases (e.g. ADHD (attention deficit hyperactivity disorder), depression, etc.), related dysplasia with a loss of social properties or psychological diseases (e.g. ASD (Autistic Spectrum Disorder, autism spectrum disorder), autism, etc.).
Analysis of expression the psychology is also useful in the recognition of expression and psychological analysis, for example, there is a tendency for ADHD hyperactivity to occur without tolerance, boredom, easy to be angry and other emotions, in addition facial paralysis, facial nerve inflammation and other problems of facial muscle control. There are also depression, mania, depression, bipolar disorder of depression, etc., which also reflect more negative expressions. Negative emotions may also be related to difficulties in meeting people, affecting the attention of children.
But either drawing scoring, expression analysis, or eye point analysis, if judged only by one or two clinicians on-the-fly, is often very subjective and is greatly affected by the outside world. At the same time, the user or patient is limited by time, geographical position and other factors, and can not be diagnosed and intervened in time.
Currently, in China, currently available developmental assessments and mental disease assessments are off-line institutions and organizations. Thus, the user who needs to participate in the drawing test scoring needs to draw out the corresponding time to the appointed place to participate in the drawing or other behavioral test activities, which is not very convenient for the user, and the user's familiar environment is changed, which may affect the user's attention and be unfavorable for the test effect. And offline assessment requires the presence of a specialized doctor, which has many limitations and difficulties in situations where regular psychologists are short-lived, the pediatric doctor is overloaded, and the geriatric care needs are out of standard. And currently existing assessment schemes have no way to provide a quantitative reference to various possible disease-related behaviors that a user may have during a pictorial test.
Disclosure of Invention
The invention provides an intelligent painting scoring method and system, which are used for solving the technical problem of avoiding the stress state of a user during time and space limitation and psychological evaluation.
In order to solve the technical problems, the technical scheme provided by the invention is as follows:
an intelligent painting scoring method, comprising the following steps:
the client provides a reference picture for the user to select;
the client displays the reference picture selected by the user, and receives and stores the image and the line drawn by the user in the drawing area; meanwhile, the client acquires eye movement information of a user and facial features of the user through the camera device; transmitting an image and a line drawn by a user in a drawing area, eye movement information of the user and facial features of the user to a processing center; the processing center is arranged on the server or the client;
the processing center receives images and lines drawn by a user in a drawing area from a client, eye movement information of the user and facial features of the user; analyzing the image and the line to obtain drawing scores, analyzing eye movement information of a user to obtain eye movement tracking scores of the user, and obtaining facial feature points and changes thereof when the user draws so as to obtain the expression of the user to judge the emotion state of the user, carrying out emotion scores according to the emotion state, and obtaining risk assessment results by integrating the drawing scores, the eye movement tracking scores and the emotion scores;
the processing center sends the painting score, the eye movement tracking score, the emotion score and the risk assessment result to the client;
the client receives and displays the painting score, eye movement tracking score, emotion score and risk assessment result from the processing center.
Preferably, the drawing scoring includes: accuracy, continuity, smoothness, line segment size and ratio;
the precision is used for judging whether the user accurately draws the reference graph according to the similarity degree of the image and the line of the user compared with the reference graph;
the continuity is used for judging whether the user continuously draws a reference graph according to whether the lines drawn by the user are continuous or not and whether the lines have a discontinuous phenomenon or not;
the smoothness is used for judging whether the user smoothly draws a reference graph according to whether the line drawn by the user is smooth or not and whether the line has obvious trembling phenomenon or not;
the line segment size and the proportion are used for judging the accuracy of hand muscle control of the user according to the thickness degree of the line drawn by the user and the overall drawing size.
The invention also provides an intelligent drawing scoring method which is applied to the client and comprises the following steps:
the client provides a reference picture for the user to select;
the client displays the reference picture selected by the user, and receives and stores the image and the line drawn by the user in the drawing area; meanwhile, the client acquires eye movement information of a user and facial features of the user through the camera device; transmitting an image and a line drawn by a user in a drawing area, eye movement information of the user and facial features of the user to a server;
the client receives and displays the painting score, eye tracking score, emotion score and risk assessment result from the server.
Preferably, collecting eye movement information of the user includes recording positions of a screen observed by both eyes of the user during a painting period to obtain a focus and a gaze track of the eyes; collecting facial features of a user, including facial feature points and changes thereof when the user draws;
the client also receives and displays an eye movement tracking result from the server, wherein the eye movement tracking result is an eye movement tracking heat map.
The invention also provides an intelligent painting scoring method which is applied to the server and comprises the following steps:
the server receives images and lines drawn by a user from the client in a drawing area, eye movement information of the user and facial features of the user; analyzing the images and lines to obtain drawing scores, analyzing eye movement information of the user to obtain eye movement tracking scores of the user, and obtaining facial feature points and changes thereof when the user draws so as to obtain expressions of the user to judge the emotional states of the user, carrying out emotion scores according to the emotional states, and obtaining risk assessment results by integrating the drawing scores, the eye movement tracking scores and the emotion scores;
and the server sends the painting score, the eye movement tracking score, the emotion score and the risk assessment result to the client.
Preferably, the drawing scoring includes: accuracy, continuity, smoothness, line segment size and ratio.
Preferably, the precision is used for judging whether the user accurately draws the reference graph according to the similarity degree of the image, the line and the reference graph of the user;
the continuity is used for judging whether the user continuously draws a reference graph according to whether the lines drawn by the user are continuous or not and whether the lines have a discontinuous phenomenon or not;
the smoothness is used for judging whether the user smoothly draws a reference graph according to whether the line drawn by the user is smooth or not and whether the line has obvious trembling phenomenon or not;
the line segment size and the proportion are used for judging the accuracy of hand muscle control of the user according to the thickness degree of the line drawn by the user and the overall drawing size.
Preferably, the eye movement information includes a focus point and a gaze track of eyes, which are obtained according to the recorded positions of the screen observed by the eyes of the user during the drawing period;
the server also evaluates and obtains the matching degree and eye movement tracking score of the user according to the attention points and the gaze tracks of the eyes.
The invention also provides a computer system comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of any of the methods described above when executing the computer program.
The invention has the following beneficial effects:
according to the intelligent drawing scoring method, through local test of the client or online test between the client and the server, (the intelligent drawing scoring method can be applied to diversified platforms such as webpage boards, computer boards and mobile boards) the limitation of time and space of a user can be avoided, and the test result of the user in a relaxed state can be obtained. And meanwhile, the actions of eyes, faces and hands of the user are collected, the drawing and eye movement and emotion during the drawing of the user are automatically evaluated by means of artificial intelligence and video analysis, and the behavioral characteristics, the attention allocation and the emotion state of the user in the drawing process are detected and evaluated to judge the psychological state of the user. The method provides quantitative reference for research, evaluation and prevention of children dysplasia, senile degenerative diseases, psychological problems, mental diseases and the like, and can perform psychological health evaluation on line at any time and any place.
In addition to the objects, features and advantages described above, the present invention has other objects, features and advantages. The invention will be described in further detail with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention. In the drawings:
FIG. 1 is a flow chart of a method for intelligent drawing scoring of a client according to a preferred embodiment of the present invention;
FIG. 2 is a flow chart of a method for intelligent painting scoring of a server according to a preferred embodiment of the present invention;
FIG. 3 is a schematic illustration of drawing scoring in accordance with a preferred embodiment of the present invention;
FIG. 4 is a schematic illustration of an eye tracking heat map in accordance with a preferred embodiment of the present invention;
FIG. 5 is a flow chart of a method for scoring intelligent painting in cooperation with a processing center by a client in accordance with a preferred embodiment of the present invention;
FIG. 6 is a graphical representation of the results of emotion scoring in accordance with a preferred embodiment of the present invention;
FIG. 7 is a schematic illustration of neutral emotion recognition in accordance with a preferred embodiment of the present invention;
FIG. 8 is a schematic diagram of a front emotion recognition of a preferred embodiment of the present invention;
FIG. 9 is a schematic diagram of negative emotion recognition in accordance with a preferred embodiment of the present invention;
FIG. 10 is a schematic diagram of the composite score for a preferred embodiment of the invention.
Detailed Description
Embodiments of the invention are described in detail below with reference to the attached drawings, but the invention can be implemented in a number of different ways, which are defined and covered by the claims.
Referring to fig. 1, the intelligent drawing scoring method of the present invention is applied to a client, and comprises the following steps:
the client provides a reference graph (which is uploaded by the user or randomly selected from a graph library by the system) for the user to select; the client comprises, but is not limited to, a webpage, a computer client, a tablet computer client or a tablet computer application program, a mobile phone client or a mobile phone application program, a WeChat applet and the like.
The client displays the reference picture selected by the user, and receives and stores the image and the line drawn by the user in the drawing area; meanwhile, the client acquires eye movement information of a user and facial features of the user through the camera device; transmitting an image and a line drawn by a user in a drawing area, eye movement information of the user and facial features of the user to a server;
the client receives and displays the painting score, eye tracking score, emotion score and risk assessment result from the server.
Through the online test between the client and the server, the limitation of the user by time and space can be avoided, so that the test result of the user in a relaxed state can be obtained. The method can perform on-line psychological health evaluation at any time and any place, and can provide reference for research, evaluation, prevention and evaluation of children dysplasia and senile degenerative diseases and mental diseases.
In practice, collecting eye movement information of a user includes recording the positions of the screen viewed by both eyes of the user during a painting session to obtain points of interest and gaze trajectories of the eyes. Eye tracking technology design can be adopted, for example, a RecordRTC (Record Real-Time Communication) frame and a face-api.js (JavaScript API for face detection and face recognition in the browser, used for face detection and face recognition interface in a browser) face recognition frame based on webRTC (Web Real-Time Communication) technology and derived from webpage instant messaging) are used, a computer camera is called, a mobile phone front-facing camera or a tablet computer front-facing camera is used, and the positions of the screens observed by the two eyes of a user in the drawing period are recorded. When presenting the results to the user, referring to fig. 4, the client also receives and presents the eye tracking results from the server, which in this embodiment is an eye tracking heat map.
In practice, the collection of facial features of the user includes facial feature points and their changes when the user draws. Facial feature recognition techniques may be used to design, for example, an open source face-api. Js (JavaScript API for face detection and face recognition in the browser, for face detection and face recognition interfaces in a browser) facial recognition framework. Invoking a computer camera, a mobile phone front-facing camera or a tablet computer front-facing camera, and recording facial feature changes of a user during drawing. Facial features of a human face are identified by using technology, such as eyes, corners of mouth and the like, and expression changes of the human face are recorded and analyzed. Can also provide references for the discovery, evaluation and rehabilitation of muscle control related diseases, nervous system diseases or muscle control related diseases.
Correspondingly, the invention also provides an intelligent painting scoring method, which is applied to a server and comprises the following steps of: the server receives images and lines drawn by a user from the client in a drawing area, eye movement information of the user and facial features of the user; analyzing the image and the line to obtain drawing scores, analyzing eye movement information of a user to obtain eye movement tracking scores of the user, and obtaining facial feature points and changes thereof when the user draws by facial features of the user, so as to obtain the expression of the user to judge the emotional state of the user, carrying out emotion scoring according to the emotional state, and synthesizing the drawing scores, the eye movement tracking scores and the emotion scores to obtain risk assessment results; and the server sends the painting score, the eye movement tracking score, the emotion score and the risk assessment result to the client.
The invention also provides an intelligent drawing scoring method, which is applied to a client and a processing center, and comprises the following steps of: the client provides a reference picture for the user to select; the client displays the reference picture selected by the user, and receives and stores the image and the line drawn by the user in the drawing area; meanwhile, the client acquires eye movement information of a user and facial features of the user through the camera device; transmitting an image and a line drawn by a user in a drawing area, eye movement information of the user and facial features of the user to a processing center; the processing center is arranged on the server or the client; the processing center receives images and lines drawn by a user in a drawing area from a client, eye movement information of the user and facial features of the user; analyzing the image and the line to obtain drawing scores, analyzing eye movement information of a user to obtain eye movement tracking scores of the user, and obtaining facial feature points and changes thereof when the user draws so as to obtain the expression of the user to judge the emotion state of the user, carrying out emotion scores according to the emotion state, and obtaining risk assessment results by integrating the drawing scores, the eye movement tracking scores and the emotion scores; the processing center sends the painting score, the eye movement tracking score, the emotion score and the risk assessment result to the client; the client receives and displays the painting score, eye movement tracking score, emotion score and risk assessment result from the processing center.
In this embodiment, referring to fig. 3, the drawing score includes: accuracy, continuity, smoothness, line segment size and ratio. The precision is used for judging whether the user accurately draws the reference graph according to the similarity degree of the image and the line of the user compared with the reference graph, and providing reference for the identification degree of the geometric graph in the user. And the continuity is used for judging whether the user continuously draws a reference graph according to whether the lines drawn by the user are continuous or not and whether the lines have a discontinuous phenomenon or not, and providing references for the discovery, evaluation and rehabilitation of brain diseases, nervous system diseases or muscle control related diseases. And the smoothness is used for judging whether the user smoothly draws a reference graph according to whether the line drawn by the user is smooth or whether the line has obvious tremor phenomenon, and providing references for the discovery, evaluation and rehabilitation of brain diseases, nervous system diseases (especially Parkinson) or muscle control related diseases. The line segment size and the proportion are used for judging the accuracy of the hand muscle control of the user according to the thickness degree of the line drawn by the user and the overall drawing size, and providing quantitative reference for the manual activity dexterity of the user. Also provides a quantitative reference for the discovery, evaluation and rehabilitation of brain diseases, nervous system diseases or muscle control related diseases. In practical application, the reference field can be used for writing by a user, and the score can be used for scoring several items by referring to the drawing score to obtain the writing score.
In this embodiment, the eye movement information includes the attention point and the gaze track of the eyes, which are obtained according to the recorded positions of the screen observed by the eyes of the user during the drawing period; the server also evaluates and obtains the matching degree and eye movement tracking score of the user according to the attention points and the gaze tracks of the eyes. And carrying out emotion scoring according to the matching degree of the user and the emotion state of the user, and integrating the drawing scoring, the eye movement tracking scoring and the emotion scoring to obtain a risk assessment result. The emotional state of the user includes: positive emotion, negative emotion, neutral emotion, and symmetrical expression. By acquiring the positions of the facial feature points and the movements of the connecting lines thereof, the movements of facial muscles of the user (such as the upward radian of corners of mouth and eyes, the lifting of eyebrows, and the like) can be known, so that whether the user is positive emotion (happy, relaxed) or negative emotion (difficult, boring, dysphoric), or neutral emotion (no obvious emotion change of facial expression) and the bilateral symmetry of the expression of the user can be calculated.
The invention can be implemented by multiple platforms such as web application programs, computer end clients, mobile end app applications, weChat applets and the like, and is convenient for users to select according to own conditions. For example, by using a web application program scheme, a user can use the invention only by using the tourist, thereby avoiding the user from installing additional software or plug-ins, reducing the use difficulty of the user and improving the compatibility and usability of the platform. The Web side may employ a resolution responsive design. The method and the system can be automatically adapted to different resolution displays, so that the limitation of the platform is broken through, a user can use the product of the invention at a computer end, and the method and the system can be used on mobile equipment such as a mobile phone, a tablet personal computer and the like, so that the user can conveniently select a plurality of types, and the method and the system can be used on different platforms. The invention can be used at any place at any time by using the browser or APP on the device under the condition of network, so that the use efficiency of the user is improved. The invention can also be set as a single edition independent of the network, thereby being convenient for users without network conditions to use.
In addition, in this embodiment, the data of the client and the processing center are stored and transmitted by using the blockchain technology. The blockchain technology can realize a blockchain storage and tracing scheme corresponding to the key information system infrastructure, and can carry out blockchain-based certification and transmission on key data of the key information system infrastructure in the data sharing process. The key data of malicious tampering can be found out in time, and the false data and illegal tampering malicious data can be found out and repaired in time in the using process.
Drawing technology provides a relatively simple way to collect social information from or related to children. Drawing is a good tool for evaluation purposes, as most children like drawing without any signs of tension. While many children do not like to answer questions, children can quickly, easily, and pleasurably complete drawing tests. In addition, the invention can avoid the limitation of time and space by the user through the single-machine test of the client or the online test between the client and the server (the invention can be applied to diversified platforms such as webpage version, computer version, mobile version, single-machine version and the like) so as to obtain the test result of the user in a relaxed state.
According to the invention, the imaging equipment of the computer platform is used for collecting and analyzing the fine movements of hands and eyes, the eye movement track, the attention distribution and the emotion psychological state simultaneously, drawing scoring (drawing scoring, eye movement tracking scoring and emotion scoring can be all carried out for a plurality of tests, the results of the plurality of tests are integrated, the accuracy of the tests is ensured), eye movement tracking scoring and emotion scoring are finally integrated to obtain a risk assessment result (integrated scoring), and the relevant analysis and auxiliary diagnosis are carried out by means of artificial intelligence. In implementation, the risk assessment results obtained by combining the three methods are as follows:
TABLE 1 mapping table of painting score, eye tracking score, expression score and risk assessment
Note that: 1. the risk assessment results are only used as reference and can be determined through further tests. 2. The risk assessment shown in table 1 is exemplary only and not all. 3. The high, normal, and medium threshold values of the scoring values can be set and fine-tuned empirically and experimentally.
FIG. 10 is a schematic diagram showing the output of a composite score, and in this embodiment, the risk assessment results are obtained from the composite score of FIG. 10, see Table 2:
table 2 comprehensive score evaluation example
The scoring details employed in the example of fig. 10 are given below:
(1) Drawing scoring:
drawing accuracy (full score 100), comparing the drawing image submitted by the user with a drawing reference image provided by a system, wherein the higher the shape accuracy is, the higher the score is;
drawing smoothness (full score 100), the user is analyzed for drawing line smoothness, and the smaller the line jitter, the higher the score;
drawing continuity (full score 100), analyzing whether a drawing line of a user is continuous or not, wherein the smaller the line broken condition is, the higher the score is;
drawing line size (full score 100), analyzing the thickness degree of the drawing line of a user, wherein the finer the drawing line is, the lower the score is; the lower the overall size of the plot, the lower the score.
(2) Eye movement tracking score:
eye movement score: (full score 100) the higher the heat of interest of the eye tracking result hot spot in the drawing region and the drawing reference region, the higher the score.
(3) Mood scoring:
emotion (full score 100) = (neutral emotion score + positive emotion score + negative emotion score); the emotion score represents the duty cycle of which emotion the user expression is more inclined to. Fig. 6 is an example of an emotion score, where the scores of three expressions represent the composition of the user's face that presents an expression, and the higher the score, the greater the probability that the user's face is that expression at the time. See the following table:
table 3 mood score content
The expression is symmetrical (full score 100), the connecting line of the characteristic points of the nose of the user is taken as the central axis, the symmetry degree of the characteristic points of the left face and the right face of the user is analyzed, and the score is higher as the symmetry degree is higher.
The results of emotion scoring were: positive emotion (see fig. 8), negative emotion (see fig. 9), neutral emotion (see fig. 7), and expression symmetry. By acquiring the positions of the facial feature points and the movements of the connecting lines thereof, the movements of facial muscles of the user (such as the upward radian of corners of mouth and eyes, the lifting of eyebrows, and the like) can be known, so that whether the user is positive emotion (happy, relaxed) or negative emotion (difficult, boring, dysphoric), or neutral emotion (no obvious emotion change of facial expression) and the bilateral symmetry of the expression of the user can be calculated.
According to the invention, the user drawing is scored by adopting an artificial intelligence means, and the computer algorithm automatically gives the score after the drawing analysis, so that the cost of manual scoring is saved, and the efficiency and objectivity of drawing test are improved. The process of drawing simultaneously can be used for measuring the fine exercise capacity of a user and the control capacity of hand muscle groups. By evaluating the result of drawing and the attention of a user through eye movement tracking, evaluating the matching degree of the user to drawing test and the emotion judgment of the user in the drawing process, the method can be comprehensively used for evaluating diseases related to abnormal discharge of brain areas (such as epilepsy, spasticity and the like), diseases related to muscle control (such as parkinsonism, cerebral palsy, eye tremor and the like), diseases related to attention (such as ADHD (hyperactivity disorder), depression and the like), developmental and social related diseases (ASD autism and the like) or other mental diseases (schizophrenia and the like), and the method is used for assisting in evaluation, and early intervention provides quantitative reference and improves practicability. The invention provides a quantitative reference for the discovery, evaluation and rehabilitation of muscle control related diseases, attention related diseases, development and social related diseases or other mental diseases.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (6)
1. An intelligent painting scoring method is characterized by comprising the following steps:
the client provides a reference picture for the user to select;
the client displays the reference picture selected by the user, and receives and stores the image and the line drawn by the user in the drawing area; meanwhile, the client acquires eye movement information of a user and facial features of the user through the camera device; transmitting an image and a line drawn by a user in a drawing area, eye movement information of the user and facial features of the user to a processing center; the processing center is arranged on the server or the client;
the processing center receives images and lines drawn by a user in a drawing area from a client, eye movement information of the user and facial features of the user; analyzing the images and the lines to obtain painting scores, analyzing eye movement information of the user to obtain eye movement tracking scores of the user, and obtaining facial feature points and changes thereof when the user draws facial features of the user, so as to obtain the expression of the user to judge the emotional state of the user, carrying out emotion scores according to the emotional state, and synthesizing the painting scores, the eye movement tracking scores and the emotion scores to obtain risk assessment results; the drawing scoring includes: accuracy, continuity, smoothness, line segment size and ratio;
the precision is used for judging whether the user accurately draws the reference graph according to the similarity degree of the image, the line and the reference graph of the user;
the consistency is used for judging whether the user continuously draws a reference graph according to whether the lines drawn by the user are continuous or not and whether the lines have a discontinuous phenomenon or not;
the smoothness is used for judging whether the user smoothly draws a reference graph according to whether the line drawn by the user is smooth or not and whether the line has obvious trembling phenomenon or not;
the line segment size and the proportion are used for judging the accuracy of hand muscle control of the user according to the thickness degree of the line drawn by the user and the overall drawing size;
the processing center sends the painting scores, the eye movement tracking scores, the emotional states and the risk assessment results to the client;
the client receives and displays the painting score, eye movement tracking score, emotion score and risk assessment result from the processing center.
2. An intelligent painting scoring method applied to a client side is characterized by comprising the following steps:
the client provides a reference picture for the user to select;
the client displays the reference picture selected by the user, and receives and stores the image and the line drawn by the user in the drawing area; meanwhile, the client acquires eye movement information of a user and facial features of the user through the camera device; transmitting an image and a line drawn by a user in a drawing area, eye movement information of the user and facial features of the user to a server;
the client receives and displays drawing scores, eye movement tracking scores, emotion scores and risk assessment results from the server;
the drawing scoring includes: accuracy, continuity, smoothness, line segment size and ratio;
the precision is used for judging whether the user accurately draws the reference graph according to the similarity degree of the image, the line and the reference graph of the user;
the consistency is used for judging whether the user continuously draws a reference graph according to whether the lines drawn by the user are continuous or not and whether the lines have a discontinuous phenomenon or not;
the smoothness is used for judging whether the user smoothly draws a reference graph according to whether the line drawn by the user is smooth or not and whether the line has obvious trembling phenomenon or not;
the line segment size and the proportion are used for judging the accuracy of hand muscle control of the user according to the thickness degree of the line drawn by the user and the overall drawing size.
3. The intelligent painting scoring method according to claim 2, wherein the collecting eye movement information of the user comprises recording positions of screens observed by both eyes of the user during painting period to obtain a focus and a gaze track of the eyes; the collecting facial features of the user comprises facial feature points and changes thereof when the user draws;
the client also receives and presents eye tracking results from the server.
4. An intelligent painting scoring method applied to a server is characterized by comprising the following steps:
the server receives images and lines drawn by a user from the client in a drawing area, eye movement information of the user and facial features of the user; analyzing the images and the lines to obtain painting scores, analyzing eye movement information of the user to obtain eye movement tracking scores of the user, and obtaining facial feature points and changes thereof when the user draws facial features of the user, so as to obtain the expression of the user to judge the emotional state of the user, carrying out emotion scores according to the emotional state, and synthesizing the painting scores, the eye movement tracking scores and the emotion scores to obtain risk assessment results; the drawing scoring includes: accuracy, continuity, smoothness, line segment size and ratio;
the precision is used for judging whether the user accurately draws the reference graph according to the similarity degree of the image, the line and the reference graph of the user;
the consistency is used for judging whether the user continuously draws a reference graph according to whether the lines drawn by the user are continuous or not and whether the lines have a discontinuous phenomenon or not;
the smoothness is used for judging whether the user smoothly draws a reference graph according to whether the line drawn by the user is smooth or not and whether the line has obvious trembling phenomenon or not;
the line segment size and the proportion are used for judging the accuracy of hand muscle control of the user according to the thickness degree of the line drawn by the user and the overall drawing size;
and the server sends the painting score, the eye movement tracking score, the emotion score and the risk assessment result to the client.
5. The intelligent painting scoring method according to claim 4, wherein the eye movement information includes a focus point and a gaze track of eyes, which are obtained according to the recorded positions of the screen observed by the eyes of the user during the painting period;
the server also evaluates and obtains the matching degree and eye tracking score of the user according to the attention points and the gaze tracks of the eyes.
6. A computer system comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method of any of the preceding claims 1 to 5 when the computer program is executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911406860.4A CN111341444B (en) | 2019-12-31 | 2019-12-31 | Intelligent painting scoring method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911406860.4A CN111341444B (en) | 2019-12-31 | 2019-12-31 | Intelligent painting scoring method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111341444A CN111341444A (en) | 2020-06-26 |
| CN111341444B true CN111341444B (en) | 2023-05-05 |
Family
ID=71183520
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911406860.4A Active CN111341444B (en) | 2019-12-31 | 2019-12-31 | Intelligent painting scoring method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111341444B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023219489A1 (en) * | 2022-05-13 | 2023-11-16 | Toybox Creations And Technology Sdn Bhd | Automated tool to assess child development |
| CN115394395B (en) * | 2022-10-27 | 2023-02-24 | 安徽星辰智跃科技有限责任公司 | Method, system and device for neuropsychological assessment and intervention based on drawing creation |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106339460A (en) * | 2016-08-26 | 2017-01-18 | 厦门优莱柏网络科技有限公司 | Online drawing processing system and online drawing processing method |
| CN107783945A (en) * | 2017-11-13 | 2018-03-09 | 山东师范大学 | A kind of search result web page notice assessment method and device based on the dynamic tracking of eye |
| CN109448848A (en) * | 2018-09-26 | 2019-03-08 | 长沙师范学院 | A kind of infantile psychology state evaluating method based on fuzzy evaluation |
-
2019
- 2019-12-31 CN CN201911406860.4A patent/CN111341444B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106339460A (en) * | 2016-08-26 | 2017-01-18 | 厦门优莱柏网络科技有限公司 | Online drawing processing system and online drawing processing method |
| CN107783945A (en) * | 2017-11-13 | 2018-03-09 | 山东师范大学 | A kind of search result web page notice assessment method and device based on the dynamic tracking of eye |
| CN109448848A (en) * | 2018-09-26 | 2019-03-08 | 长沙师范学院 | A kind of infantile psychology state evaluating method based on fuzzy evaluation |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111341444A (en) | 2020-06-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Niehorster et al. | The impact of slippage on the data quality of head-worn eye trackers | |
| Aigrain et al. | Multimodal stress detection from multiple assessments | |
| Künecke et al. | Facial EMG responses to emotional expressions are related to emotion perception ability | |
| Bindemann et al. | Viewpoint and center of gravity affect eye movements to human faces | |
| Hsiao et al. | Two fixations suffice in face recognition | |
| Wang et al. | Facial expression video analysis for depression detection in Chinese patients | |
| Bekele et al. | Assessing the utility of a virtual environment for enhancing facial affect recognition in adolescents with autism | |
| Cho | Rethinking eye-blink: Assessing task difficulty through physiological representation of spontaneous blinking | |
| US20150305662A1 (en) | Remote assessment of emotional status | |
| Dundas et al. | A lack of left visual field bias when individuals with autism process faces | |
| Hussain et al. | Automatic cognitive load detection from face, physiology, task performance and fusion during affective interference | |
| Brooks et al. | Looking at the figures: visual adaptation as a mechanism for body-size and-shape misperception | |
| Kastrati et al. | EEGEyeNet: a simultaneous electroencephalography and eye-tracking dataset and benchmark for eye movement prediction | |
| Sharma et al. | Sensing technologies and child–computer interaction: Opportunities, challenges and ethical considerations | |
| CN111341444B (en) | Intelligent painting scoring method and system | |
| Brunner et al. | BNCI horizon 2020–towards a roadmap for brain/neural computer interaction | |
| Cilia et al. | Eye-tracking dataset to support the research on autism spectrum disorder | |
| Skaramagkas et al. | eSEE-d: Emotional state estimation based on eye-tracking dataset | |
| Zhang et al. | Visual attention and cognitive process in construction hazard recognition: Study of fixation-related potential | |
| Chen | Cognitive load measurement from eye activity: acquisition, efficacy, and real-time system design | |
| Varela et al. | Looking at faces in the wild | |
| Zhang et al. | Human-centered intelligent healthcare: explore how to apply AI to assess cognitive health | |
| Hutt et al. | Feasibility of Longitudinal Eye-Gaze Tracking in the Workplace | |
| Jo et al. | Mocas: A multimodal dataset for objective cognitive workload assessment on simultaneous tasks | |
| Kim et al. | Facial Motion Capture System Based on Facial Electromyogram and Electrooculogram for Immersive Social Virtual Reality Applications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |