CN113591779B - Projection data acquisition system and method based on integrated multi-sensor handwriting - Google Patents
Projection data acquisition system and method based on integrated multi-sensor handwriting Download PDFInfo
- Publication number
- CN113591779B CN113591779B CN202110919114.6A CN202110919114A CN113591779B CN 113591779 B CN113591779 B CN 113591779B CN 202110919114 A CN202110919114 A CN 202110919114A CN 113591779 B CN113591779 B CN 113591779B
- Authority
- CN
- China
- Prior art keywords
- handwriting
- user
- electronic pen
- data
- electronic board
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000004458 analytical method Methods 0.000 claims abstract description 36
- 238000012360 testing method Methods 0.000 claims abstract description 20
- 238000012545 processing Methods 0.000 claims abstract description 12
- 239000011159 matrix material Substances 0.000 claims description 24
- 238000012986 modification Methods 0.000 claims description 16
- 230000004048 modification Effects 0.000 claims description 16
- QVGXLLKOCUKJST-UHFFFAOYSA-N atomic oxygen Chemical compound [O] QVGXLLKOCUKJST-UHFFFAOYSA-N 0.000 claims description 15
- 239000008280 blood Substances 0.000 claims description 15
- 210000004369 blood Anatomy 0.000 claims description 15
- 229910052760 oxygen Inorganic materials 0.000 claims description 15
- 239000001301 oxygen Substances 0.000 claims description 15
- 230000006870 function Effects 0.000 claims description 11
- 230000005611 electricity Effects 0.000 claims description 9
- 230000005540 biological transmission Effects 0.000 claims description 8
- 230000004927 fusion Effects 0.000 claims description 5
- 238000005259 measurement Methods 0.000 claims description 4
- 230000036772 blood pressure Effects 0.000 claims description 3
- 238000013500 data storage Methods 0.000 claims description 3
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 3
- 210000003811 finger Anatomy 0.000 claims description 3
- 238000010422 painting Methods 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- 238000003825 pressing Methods 0.000 claims description 3
- 230000008054 signal transmission Effects 0.000 claims description 3
- 210000003813 thumb Anatomy 0.000 claims description 3
- 238000013135 deep learning Methods 0.000 claims description 2
- 238000013480 data collection Methods 0.000 claims 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000013136 deep learning model Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 230000009191 jumping Effects 0.000 description 2
- 241001465754 Metazoa Species 0.000 description 1
- 101000579647 Penaeus vannamei Penaeidin-2a Proteins 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 208000020016 psychiatric disease Diseases 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F3/0414—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means using force sensing means to determine a position
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F3/0416—Control or interface arrangements specially adapted for digitisers
- G06F3/04162—Control or interface arrangements specially adapted for digitisers for exchanging data with external devices, e.g. smart pens, via the digitiser sensing hardware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Position Input By Displaying (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention relates to a system and a method for acquiring projection data based on handwriting of integrated multiple sensors, wherein the system comprises an electronic board and an electronic pen; the electronic pen with the integrated multiple sensors is used for writing or drawing on the electronic board by a user, physiological characteristic information of the user in the writing or drawing process is collected through the sensors on the electronic pen, meanwhile, dynamic processes of the user writing and drawing are collected through the electronic board in real time, data are stored and analyzed, picture content is identified, picture handwriting characteristics are further extracted, and comprehensive processing analysis is conducted by combining corresponding physiological characteristic information of the user, so that complete analysis can be conducted on the data of the whole testing process of the user, and objective and accurate testing results are facilitated.
Description
Technical Field
The invention relates to the field of biological information acquisition terminals, in particular to a system and a method for acquiring projection data based on integrated multi-sensor handwriting.
Background
The projection test is an important testing technique in the field of psychology, wherein the picture analysis technique in the projection test requires a subject to draw according to a certain task, and an evaluator to evaluate according to the content characteristics or form characteristics of the work thereof, such as a house-tree-person test. The assessor estimates the psychological characteristics of the subject or makes a diagnosis of a psychological disorder based on the pictorial representation analysis of the subject. Traditional house tree person testing is generally completed by using a paper pen testing method, and the traditional house tree person testing method has some defects and inconveniences in actual use. Therefore, there is also a technology for providing an electronic version house tree man technology, the traditional house tree man test is displayed on a computer in a software mode by using a processing mode of a multimedia technology, and then the characteristics of a picture are analyzed through the computer, so that the purpose of measurement is achieved. However, in the above-mentioned technology, only the picture completed by the user is analyzed, the collection and analysis of the physiological characteristics of the user drawing at the time are lacking, and the analysis of the dynamic parameters in the user drawing process is lacking, so that the data is very single, the subjectivity is strong, the influence of experience of the evaluator is great, and the conclusion is sometimes not objective enough.
Disclosure of Invention
In order to solve the technical problems, the invention provides a system and a method for acquiring projection data based on integrated multi-sensor handwriting, an electronic board and an electronic pen; the electronic pen is held by a user to write or draw on the electronic board, physiological characteristic information of the user in the writing or drawing process is collected through the sensor on the electronic pen, meanwhile, dynamic processes of the user writing and drawing are collected through the electronic board in real time, the data are stored and analyzed, picture content is identified, picture handwriting characteristics are further extracted, and comprehensive processing analysis is conducted by combining the corresponding physiological characteristic information of the user, so that complete analysis can be conducted on the data of the whole testing process of the user, and objective and accurate testing results are facilitated.
The technical scheme of the invention is as follows: a projection data acquisition system based on integrated multisensor handwriting, comprising:
the electronic board comprises a touch screen, an outer frame, a bracket and a signal transmission interface; a pressure sensor is arranged below the touch screen; the frame of the electronic board is also provided with a plurality of keys and contact indicator lamps, wherein the keys comprise a start key, an end key, a playback key, a data storage key and a data deriving key; the touch indicator light is turned on when the electronic pen is in effective contact with the touch screen of the electronic board, so that the touch indicator light indicates that the touch is effective, the data recording is normal, and the touch indicator light indicates that the touch is ineffective when the touch indicator light is not on, and the touch indicator light may be in bad contact or improper operation by a user;
a color selection area, a stroke width selection area and a back and erase button are arranged on a touch screen display area of the electronic board;
the electronic board also comprises a processor and a memory, wherein the processor is used for analyzing and processing handwriting, extracting the characteristics of the handwriting and specifically comprises the following steps:
the content recognition module is used for carrying out overall analysis on the written and drawn content through a machine vision algorithm so as to realize a main body target recognition function;
the handwriting characteristic analysis module is used for extracting and analyzing the painting handwriting characteristics;
and the fusion analysis module is used for fusing multidimensional features of handwriting on the basis of completing content identification to obtain an analysis result.
Further, triggering the touch screen to start recording handwriting when the start key is pressed; triggering the touch screen to end writing when the ending key is pressed; playing back the recorded handwriting when the playback key is pressed; and when the data export key is pressed, exporting handwriting to external equipment.
Further, the color selection area is used for selecting a color desired to be used by a user in the test; the stroke width selection area is used for selecting the stroke width which is expected to be used by a user in a test; the back button is used for executing withdrawal operation, the erasing button is used for erasing the track of a part of the area, and when the user is not satisfied with the track of the picture, the erasing function is used for erasing the content of the area.
Further, the electronic board is used for receiving writing and drawing of the electronic pen thereon, capturing and recording handwriting of the electronic pen through a touch screen on the electronic board, recording occurrence time of each track point in the handwriting, corresponding force and color information, and obtaining recorded handwriting points P (x, y, t, color, width and pressure), wherein x, y are an abscissa and an ordinate, t is the current recording moment, color is color, width is stroke, and pressure is pressure of the electronic pen; one handwriting formed by a series of points is l= { P1, P2, pi … … Pn }, n is the number of points on the handwriting;
the plurality of handwriting L form a whole picture A, A= { L1, L2, li … … LN }, and N is the number of handwriting;
the electronic board is internally provided with a timer for recording the time t of each point in the handwriting.
Further, the electronic board also comprises a data interface which is used for being connected with a data line of the electronic pen so as to receive the data of the electronic pen; or the electronic board is provided with a wireless transceiver module, and data transmission is carried out with the electronic pen through wireless signals;
the electronic board can also store the handwriting data, and the stored data can be read by the processor at any time and is replayed, analyzed and processed;
the electronic pen comprises an electronic pen body and a holding part arranged on the body, wherein a plurality of sensors are arranged at the holding part, and the sensors comprise a pressure sensor, a heart rate sensor, a blood oxygen saturation sensor, a skin electricity sensor and a Pi Wenchuan sensor; the tail end of the electronic pen is provided with a data transmission line, and the data transmission line is connected to the electronic board through an interface; or the electronic pen is provided with a wireless transceiver module, and sensor data are transmitted through the wireless transceiver module.
According to another aspect of the present invention, a method for acquiring handwritten projection data using the aforementioned system is provided, comprising the steps of:
step 3, when the electronic pen is detected to be in contact with the electronic board, the contact indicator is on, a pressure sensor on the electronic pen collects blood pressure data of the user in real time, a blood oxygen sensor collects blood oxygen concentration parameters of the user, a heart rate sensor collects heart rate of the user, and the physiological data of the user are transmitted to the electronic board;
step 4, the electronic board detects the handwriting of the electronic pen on the touch screen in real time, and the color parameters selected by the electronic pen, the pressure parameters of the electronic pen pressing the touch board, the position parameters of the handwriting point and the current time parameters;
step 5, carrying out content identification on the picture, and carrying out overall analysis on the written and drawn content through a machine vision algorithm to realize a main body target identification function;
step 6, based on the completion of content identification, merging multidimensional features of handwriting, and outputting collected and analyzed data; the multidimensional features of the handwriting mainly comprise handwriting features and physiological features.
Further, the step 5 is to identify the content of the picture, and to perform overall analysis on the written and drawn content through a machine vision algorithm, so as to realize the main body target identification function; the specific steps are as follows:
step 5.1, dividing an image on the electronic board into K x K dry square areas, and sequentially numbering all square areas from 0 to K x K areas in total;
step 5.2, constructing an area adjacency matrix, wherein the area adjacency matrix is used for representing adjacency relations among square areas, continuous handwriting is used when a user draws the same main body content, and the handwriting jump occurs when drawing different main bodies; the element of the ith row and jth column of the region adjacency matrix represents the total number of successive passes through the ith and jth image regions simultaneously; i. j is the serial number of the region;
step 5.3, merging the image with the adjacent matrix, firstly, according to the belonging relation between the region and the pixel, the dimension of the adjacent matrix of the region is changed from K 2 ×K 2 Extended to W x H x K 2 . Then the expanded area adjacent matrix is combined with the image data to obtain WXH× (C+K) 2 ) C represents the channel number, W is the image width, H is the image height, and is used as the input of a Darknet deep learning prediction model;
step 5.4, extracting image features by using a trained dark learning prediction model, wherein dark comprises a convolution layer, a residual layer and a pooling layer, and extracting depth features from image data;
and 5.5, outputting a prediction result, identifying a target subject in the region according to the image characteristics output by the Darknet by each region, and finally outputting the target type and the target size.
Further, step 6, on the basis of completing content recognition, merging multidimensional features of handwriting, and outputting collected and analyzed data; the method specifically comprises the following steps:
step 6.1, extracting all handwriting characteristics in the main content area;
step 6.2, extracting corresponding physiological characteristic information of the user according to the handwriting occurrence and ending time;
step 6.3, calculating average residence time, modification times, jump times and corresponding physiological characteristic change conditions of handwriting of the user in each main body area;
and 6, outputting acquisition and analysis results.
Further, drawing handwriting characteristics are extracted and analyzed by utilizing a handwriting characteristic analysis module; the handwriting characteristics comprise handwriting appearance characteristics, handwriting time phase characteristics, handwriting space phase characteristics and handwriting modification characteristics; specific:
the handwriting appearance characteristics comprise: handwriting thickness, regularity, size, weight, color; the handwriting thickness is obtained by selecting on a panel; the regularity refers to the degree of handwriting smoothness or flow field, for example, the regularity can be obtained by calculating the mean square error of a fitting curve; the handwriting size refers to the area size defined by the maximum width and the maximum height of the handwriting; the weight of the handwriting is obtained through a pressure sensor on the electronic pen;
the handwriting time stage characteristics comprise: the handwriting point sequence, the sequence among the handwriting, the pause time, the speed characteristic, the handwriting jump and the handwriting fluency; the handwriting point sequence comprises the sequence of each point in the handwriting, and as the time information is arranged on the handwriting points, the sequence of the points on the handwriting can be obtained through the time information; the sequence among different handwriting and the pause time can be obtained by comparing the time of the upper point of the different handwriting; calculating a speed characteristic by dividing the length of the handwriting by the time difference between the time of the first point and the time of the last point of the handwriting; handwriting jump refers to the distance between different handwriting, wherein the tail point of the previous handwriting is away from the starting point of the next handwriting;
the handwriting space features include: handwriting area distribution, breakpoint proportion, breakpoint number, area density and handwriting continuity;
according to the embodiment of the invention, the handwriting distribution area characteristics can be represented by a matrix, and the handwriting point distribution quantity of each area is used as a matrix corresponding position element; the number of break points is handwriting points with the length smaller than the preset length, and the number of break points is compared with the total number of handwriting points to obtain the proportion of break points; handwriting continuity refers to the ratio of the length of the handwriting to the average length of the handwriting; region density refers to the density of points per region;
the handwriting modification features comprise modification adjustment, repetition and deletion, and represent modification processing processes of users; a back and erasing button is displayed in a screen area of the electronic board, when a user clicks the back or erasing button in the drawing process, the system records the operation and modifies the corresponding handwriting to obtain a modified record (mode, L, t), and the mode represents back or erasing; l represents corresponding handwriting, and t represents current time;
further, the physiological characteristics are heart rate, pen holding pressure, blood oxygen saturation, skin electricity and skin temperature of a user during drawing.
The beneficial effects are that:
according to the projection data acquisition system and method based on the integrated multi-sensor handwriting, physiological characteristic information of a user in the writing or drawing process is acquired through various sensors on the electronic pen, meanwhile, dynamic processes of the user writing and drawing are acquired through the electronic board in real time, data are stored and analyzed, picture content is identified, picture handwriting characteristics are further extracted, and comprehensive processing analysis is performed by combining corresponding physiological characteristic information of the user, so that complete analysis can be performed on the data of the whole testing process of the user, and objective and accurate testing results are facilitated.
Drawings
FIG. 1 is a schematic diagram of an electronic board and electronic pen of a handwriting-based projection data acquisition system with integrated multiple sensors according to the present invention;
FIG. 2 is a schematic diagram of an electronic pen according to the present invention;
fig. 3 is a flowchart of a method for acquiring projection data based on handwriting of integrated multiple sensors according to the present invention.
Detailed Description
The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, but not all embodiments, and all other embodiments obtained by those skilled in the art without the inventive effort based on the embodiments of the present invention are within the scope of protection of the present invention.
According to an embodiment of the invention, an integrated multisensor handwriting-based projection data acquisition system is shown in fig. 1, and comprises an electronic board 1 and an integrated multisensor electronic pen 2, wherein the electronic board comprises a touch screen, an outer frame, a bracket and a signal transmission interface; a pressure sensor is arranged below the touch screen;
the frame of the electronic board is also provided with a plurality of keys and contact indicator lamps, wherein the keys comprise a start key, an end key, a playback key, a data storage key and a data deriving key;
triggering the touch screen to start recording handwriting when the start key is pressed;
triggering the touch screen to end writing when the ending key is pressed;
playing back the recorded handwriting when the playback key is pressed;
when the data export key is pressed, exporting handwriting to external equipment, such as a computer, a USB flash disk or the like;
the touch indicator light is turned on when the electronic pen is in effective contact with the touch screen of the electronic board, so that the touch indicator light indicates that the touch is effective, the data recording is normal, and the touch indicator light indicates that the touch is ineffective when the touch indicator light is not on, and the touch indicator light may be in bad contact or improper operation by a user;
the touch screen display area of the electronic board is provided with a color selection area, a stroke width selection area, buttons for backing, erasing and the like;
the color selection area is used for selecting the color which is expected to be used by a user in the test;
the stroke width selection area is used for selecting the stroke width which is expected to be used by a user in a test;
the back button is used for executing withdrawal operation, the erasing button is used for erasing the track of a part of the area, and when the user is dissatisfied with the track of the picture, the erasing function can be used for erasing the content of the area;
the electronic board is used for receiving writing and drawing of the electronic pen on the electronic board, capturing and recording handwriting of the electronic pen through a touch screen on the electronic board, and recording the occurrence time of each track point in the handwriting, and corresponding force and color information; for example, a point P (x, y, t, color, width, pressure), where x, y are abscissa and ordinate, t is the current recording time, color is color, width is stroke width, pressure is pressure of the electronic pen; one handwriting formed by a series of points is l= { P1, P2, pi … … Pn }, n is the number of points on the handwriting;
the plurality of handwriting L form a whole picture A, A= { L1, L2, li … … LN }, and N is the number of handwriting;
the electronic board internally comprises a timer which is used for recording the time t of each point in handwriting;
according to the embodiment of the invention, the electronic board further comprises a processor and a memory, wherein the processor is used for identifying picture content, analyzing and processing handwriting, extracting characteristics of the handwriting, such as time of each part of movement, strength of pen touch, track continuity/fluency, regularity of short points, jumping performance of strokes, repeated drawing, position (coordinate axis), area, symmetry and the like of the contour.
The electronic board also comprises a data interface which is used for being connected with a data line of the electronic pen so as to receive the data of the electronic pen;
or the electronic board is provided with a wireless transceiver module, and data transmission is carried out with the electronic pen through wireless signals;
the electronic board can also store the handwriting data, and the stored data can be read by the processor at any time and is replayed, analyzed and processed; so that data processing can be performed at any time;
as shown in fig. 2, the electronic pen comprises an electronic pen body and a holding part arranged on the body, wherein a plurality of sensors are arranged on the holding part, and the sensors comprise a pressure sensor, a heart rate sensor, a blood oxygen saturation sensor, a skin electricity sensor, a skin temperature sensor and the like; the black areas in the figures are sensors, which are only illustrative and can be arranged according to the actual required number;
further, a data transmission line is arranged at the tail end of the electronic pen and is connected to the electronic board through an interface; or the electronic pen is provided with a wireless transceiver module, and sensor data are transmitted through the wireless transceiver module;
according to an embodiment of the present invention, a method for acquiring handwritten projection data by using the above system is provided, which includes the following steps:
step 3, when the electronic pen is detected to be in contact with the electronic board, the contact indicator is on, a pressure sensor on the electronic pen collects blood pressure data of the user in real time, a blood oxygen sensor collects blood oxygen concentration parameters of the user, a heart rate sensor collects heart rate of the user, and the physiological data of the user are transmitted to the electronic board;
step 4, the electronic board detects the handwriting of the electronic pen on the touch screen in real time, and the color parameters selected by the electronic pen, the pressure parameters of the electronic pen pressing the touch board, the position parameters of the handwriting point and the current time parameters;
step 5, performing content recognition on the picture by utilizing a content recognition module, and performing overall analysis on the written and drawn content by utilizing a machine vision algorithm to realize a main body target recognition function;
step 6, extracting handwriting characteristics by utilizing a handwriting characteristic extraction module on the basis of completing content identification; the method comprises the steps of utilizing a fusion analysis module to fuse multidimensional features of handwriting and outputting collected and analyzed data; the multidimensional features of the handwriting mainly comprise handwriting features and physiological features.
The content recognition module performs overall analysis on the written and drawn content through a machine vision algorithm, and achieves a main body target recognition function. The content recognition process steps are shown in fig. 3. Comprises the following steps:
and step 1, dividing the image on the electronic board into areas. The image is divided into K x K dry square areas, for example, k=5, or k=6, etc., using expert experience. When the center point (black dot) of the content body falls within a certain square area, the area is responsible for identifying the content body and outputting the type and size of the final body target. All square areas are numbered sequentially starting from 0. For example, from the first region in the upper left corner to 0, numbered sequentially to K by zigzag, there are K total K regions;
and 2, constructing a region adjacency matrix. The area adjacency matrix is used to represent adjacency relations between square areas. Continuous handwriting is used when a user draws the same subject content, while jumping of handwriting occurs when drawing different subjects. Therefore, the construction of the pixel connection diagram by using the continuous handwriting is beneficial to improving the accuracy of the main body identification. The element of the ith row and jth column of the region adjacency matrix represents the total number of successive passes through the ith and jth image regions simultaneously; i. j is the sequence number of the region, and the total number of the regions is K 2 。
And 3, merging the image with the adjacent matrix. First according to the region and the pixelRelationship, dimension of area adjacency matrix from K 2 ×K 2 Extended to W x H x K 2 . The image data is then channel expanded, here channel expanded, using the expanded region adjacency matrix. The original image is w×h×c, and the channel expansion is changed to w×h (c+k) 2 ) W X H X (C+K) 2 ) C represents the number of channels, which generally refers to the values of three colors RGB (red, green and blue). However, there may be also cases of RGBA (red green blue alpha) 4 dimensions, or gray scale system 1 dimension, or CMYK. Depending on the image format. W is the image width, H is the image height, and is used as an input of the prediction model.
And 4, extracting image features by using a Darknet deep learning model. The dark net is a deep learning model constructed by a convolution layer, a residual layer, a pooling layer and the like, and can well extract depth features from image data. The dark deep learning model adopts known image data in advance to complete training.
And 5, outputting a prediction result. And each region identifies the target subject in the region according to the image characteristics output by the Darknet, and finally the target type and the target subject size are output. The target types include buildings, vehicles, people, animals, plants, etc.; the target size is the area occupied by the region (for example, coordinate values of the upper left corner and the lower right corner of the square are actually given, and the area thereof is calculated from the coordinate values).
The handwriting characteristic analysis module is used for extracting and analyzing the painting handwriting characteristics; the handwriting characteristic analysis module has the main functions of summarizing all historical handwriting in a target area on the basis of completing content recognition, and fusing multidimensional characteristics of the handwriting to obtain an analysis result, so that dynamic depiction of the user's mind is realized.
The multidimensional features of the handwriting mainly comprise handwriting features and physiological features.
The handwriting characteristics mainly comprise handwriting appearance characteristics, handwriting time phase characteristics, handwriting space phase characteristics and handwriting modification characteristics;
the handwriting appearance features include, for example: handwriting thickness, regularity, size, weight, color;
for example, the handwriting thickness can be selected on the panel, and the selected handwriting thickness is, for example, 5 pixels, or 10 pixels as the handwriting thickness; the regularity refers to the degree of handwriting smoothness or flow field, for example, the regularity can be obtained by calculating the mean square error of a fitting curve; the handwriting size refers to the area size defined by the maximum width and the maximum height of the handwriting; the weight of the handwriting is obtained through a pressure sensor on the electronic pen;
the handwriting time stage characteristics comprise: the handwriting point sequence, the sequence among the handwriting, the pause time, the speed characteristic, the handwriting jump and the handwriting fluency; the handwriting point sequence comprises the sequence of each point in the handwriting, and as the time information is arranged on the handwriting points, the sequence of the points on the handwriting can be obtained through the time information; the sequence among different handwriting and the pause time can be obtained by comparing the time of the upper point of the different handwriting; calculating a speed characteristic by dividing the length of the handwriting by the time difference between the time of the first point and the time of the last point of the handwriting; handwriting jump refers to the distance between different handwriting, wherein the tail point of the previous handwriting is away from the starting point of the next handwriting;
the handwriting modification features comprise modification adjustment, repetition, deletion and the like, and represent the modification processing process of a user;
according to the embodiment of the invention, the screen area of the electronic board is displayed with the back and erase buttons, when a user clicks the back or erase button in the drawing process, the system records the operation and modifies the corresponding handwriting to obtain a modified record edit (mode, L, t), and the mode represents back or erase; l represents corresponding handwriting, and t represents current time;
the handwriting space features include: handwriting area distribution, breakpoint proportion, breakpoint number, area density and handwriting continuity;
according to the embodiment of the invention, the handwriting distribution area characteristics can be represented by a matrix, and the handwriting point distribution quantity of each area is used as a matrix corresponding position element; the number of break points is handwriting points with the length smaller than the preset length, and the number of break points is compared with the total number of handwriting points to obtain the proportion of break points; handwriting continuity refers to the ratio of the length of the handwriting to the average length of the handwriting; region density refers to the density of points per region;
the physiological characteristics are heart rate, pen holding pressure, blood oxygen saturation, skin electricity, skin temperature and other physiological characteristics of a user during drawing.
According to the embodiment of the invention, the processor can further judge different stages of writing and drawing of an operator according to the handwriting, and stage division is carried out on the whole handwriting from time or space or combination of the two, and the stage division is divided into a plurality of track segments to obtain stage characteristics of the handwriting:
for example, in the process of writing and drawing, a user stops writing for a period of time in the middle and then drops the pen, so that the recording time of the handwriting points is discontinuous jump; representing the thinking of the user or completing one goal to the next, i.e. one phase to another;
or the user can finish drawing from one area and jump to another area to start drawing, and the space crossing in the user drawing process can be analyzed by analyzing the recording time of the handwriting points and the recording area; furthermore, the whole drawing process can be subjected to space crossing analysis to obtain the drawing sequence and stage of each region.
According to an alternative embodiment of the invention, the handwriting is also staged in time as follows:
the initial stage: positioning a handwriting area of a user, analyzing a layout area and content of the user, and calculating fluency of each handwriting line;
modification processing stage: extracting characteristics of detail adjustment, repeating, deleting and color modification of the picture content by a user;
in the completion phase: extracting the picture content layout of a user, and analyzing the main content and the main color, the background area and the color;
in the finalization stage: the user carries out written description and confirmation on the picture content, and extracts the keywords of the description as auxiliary data.
According to another aspect of the present invention, the content recognition module, handwriting feature extraction module, fusion analysis module may be provided as a separate chip module, such as an FPGA or other digital chip, or may be stored as a program module on a memory, the program module being executed by a processor to implement the content recognition module, etc
While the foregoing has been described in relation to illustrative embodiments thereof, so as to facilitate the understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the embodiments, but is to be construed as limited to the spirit and scope of the invention as defined and defined by the appended claims, as long as various changes are apparent to those skilled in the art, all within the scope of which the invention is defined by the appended claims.
Claims (9)
1. A projection data acquisition system based on integrated multisensor handwriting, comprising:
the electronic board comprises a touch screen, an outer frame, a bracket and a signal transmission interface; a pressure sensor is arranged below the touch screen; the frame of the electronic board is also provided with a plurality of keys and contact indicator lamps, wherein the keys comprise a start key, an end key, a playback key, a data storage key and a data deriving key; the touch indicator light is turned on when the electronic pen is in effective contact with the touch screen of the electronic board, the touch indicator light indicates that the touch is effective, the data record is normal, and the touch indicator light indicates that the touch is ineffective when the touch indicator light is not in the bright state; the electronic pen comprises an electronic pen body and a holding part arranged on the body, wherein a plurality of sensors are arranged at the holding part, and the sensors comprise a pressure sensor, a heart rate sensor, a blood oxygen saturation sensor, a skin electricity sensor and a Pi Wenchuan sensor; the tail end of the electronic pen is provided with a data transmission line, and the data transmission line is connected to the electronic board through an interface; or the electronic pen is provided with a wireless transceiver module, and sensor data are transmitted through the wireless transceiver module;
a color selection area, a stroke width selection area and a back and erase button are arranged on a touch screen display area of the electronic board;
the electronic board also comprises a content identification module, a handwriting feature analysis module and a fusion analysis module; analyzing and processing the handwriting, and extracting the multidimensional features of the handwriting; wherein,,
the content recognition module is used for carrying out overall analysis on the written and drawn content through a machine vision algorithm so as to realize a main body target recognition function; the specific steps are as follows:
step 5.1, dividing the image on the electronic board into K
K dry square areas, all square areas numbered sequentially from 0, K total +.>
K regions;
step 5.2, constructing an area adjacency matrix, wherein the area adjacency matrix is used for representing adjacency relations among square areas, continuous handwriting is used when a user draws the same main body content, and the handwriting jump occurs when drawing different main bodies; the element of the ith row and jth column of the region adjacency matrix represents the total number of successive passes through the ith and jth image regions simultaneously; i. j is the serial number of the region;
step 5.3, merging the image with the adjacent matrix, firstly, according to the belonging relation between the region and the pixel, the dimension of the adjacent matrix of the region is reduced from
Expansion to->
The method comprises the steps of carrying out a first treatment on the surface of the Then the expanded region adjacency matrix is combined with the image data to obtain +>
C represents the number of channels, W is the image width, H is the image height, as DarknetInput of a deep learning prediction model;
step 5.4, extracting image features by using a trained dark learning prediction model, wherein dark comprises a convolution layer, a residual layer and a pooling layer, and extracting depth features from image data;
step 5.5, outputting a prediction result, and identifying a target subject in each region according to image characteristics output by the Darknet, and finally outputting a target type and a target size;
the handwriting characteristic analysis module is used for extracting and analyzing the painting handwriting characteristics;
and the fusion analysis module is used for fusing the multidimensional features of the handwriting on the basis of completing the content identification to obtain an analysis result, wherein the multidimensional features of the handwriting comprise handwriting features and physiological features.
2. The integrated multisensor handwriting-based projection data collection system of claim 1, wherein the start button, when pressed, triggers the touch screen to start writing; triggering the touch screen to end writing when the ending key is pressed; playing back the recorded handwriting when the playback key is pressed; and when the data export key is pressed, exporting handwriting to external equipment.
3. The integrated multisensor handwriting-based projection data collection system of claim 1, wherein the color selection area is configured to select a desired color for use by a user during testing; the stroke width selection area is used for selecting the stroke width which is expected to be used by a user in a test; the back button is used for executing withdrawal operation, the erasing button is used for erasing the track of a part of the area, and when the user is not satisfied with the track of the picture, the erasing function is used for erasing the content of the area.
4. The integrated multisensor handwriting-based projection data acquisition system of claim 1, wherein the electronic board is used for receiving writing and drawing of an electronic pen on the electronic board, capturing and recording handwriting of the electronic pen through a touch screen on the electronic board, recording occurrence time of each track point in the handwriting, corresponding force and color information, and obtaining recorded handwriting points P (x, y, t, color, width and pressure), wherein x, y are an abscissa and an ordinate, t is a current recording time, color is a color, width is a stroke, and pressure is a pressure of the electronic pen; one handwriting formed by a series of points is l= { P1, P2, pi … … Pn }, n is the number of points on the handwriting;
the plurality of handwriting L form a whole picture A, A= { L1, L2, li … … LN }, and N is the number of handwriting;
the electronic board is internally provided with a timer for recording the time t of each point in the handwriting.
5. The integrated multisensor handwriting-based projection data collection system of claim 1, wherein the electronic board further comprises a data interface for connecting with a data line of the electronic pen to receive data of the electronic pen; or the electronic board is provided with a wireless transceiver module, and data transmission is carried out with the electronic pen through wireless signals;
the electronic board can also store the handwriting data, and the stored data can be read by the processor at any time and is replayed and analyzed.
6. A method of handwriting projection data acquisition using the system of any one of claims 1-5, comprising:
step 1, connecting an electronic pen with an electronic board in a wired or wireless mode, starting a power switch of a system, and running the system;
step 2, the electronic pen is held by a user, the thumb is contacted with the blood oxygen sensor, the skin electricity sensor and the Pi Wenchuan sensor, and the index finger is connected with the heart rate sensing period pressure sensor; clicking a start button, a user starts drawing on an electronic board by using an electronic pen fused by multiple sensors, and the electronic board records the drawing handwriting in real time and records the operation of the user on the editing of the picture pen, wherein the operation comprises withdrawal, erasure and measurement of physiological characteristic data of the user by the electronic pen; clicking an end key after drawing is completed;
step 3, when the electronic pen is detected to be in contact with the electronic board, the contact indicator is on, a pressure sensor on the electronic pen collects blood pressure data of the user in real time, a blood oxygen sensor collects blood oxygen concentration parameters of the user, a heart rate sensor collects heart rate of the user, and the physiological data of the user are transmitted to the electronic board;
step 4, the electronic board detects the handwriting of the electronic pen on the touch screen in real time, and the color parameters selected by the electronic pen, the pressure parameters of the electronic pen pressing the touch board, the position parameters of the handwriting point and the current time parameters;
step 5, carrying out content identification on the picture, and carrying out overall analysis on the written and drawn content through a machine vision algorithm to realize a main body target identification function;
step 6, based on the completion of content identification, merging multidimensional features of handwriting, and outputting collected and analyzed data; the multidimensional features of the handwriting mainly comprise handwriting features and physiological features.
7. The projection data acquisition method according to claim 6, wherein the step 6 of fusing multidimensional features of handwriting and outputting acquired and analyzed data on the basis of completing content recognition; the method specifically comprises the following steps:
step 6.1, extracting all handwriting characteristics in the main content area;
step 6.2, extracting corresponding physiological characteristic information of the user according to the handwriting occurrence and ending time;
step 6.3, calculating average residence time, modification times, jump times and corresponding physiological characteristic change conditions of handwriting of the user in each main body area;
and 6, outputting acquisition and analysis results.
8. The projection data collection method according to claim 6, wherein the drawing handwriting feature extraction and analysis is performed by using a handwriting feature analysis module; the handwriting characteristics comprise handwriting appearance characteristics, handwriting time phase characteristics, handwriting space phase characteristics and handwriting modification characteristics; specific:
the handwriting appearance characteristics comprise: handwriting thickness, regularity, size, weight, color; the handwriting thickness is obtained by selecting on a panel; the regularity refers to the degree of handwriting smoothness or flow field, and is obtained by calculating the mean square error of a fitting curve; the handwriting size refers to the area size defined by the maximum width and the maximum height of the handwriting; the weight of the handwriting is obtained through a pressure sensor on the electronic pen;
the handwriting time stage characteristics comprise: the handwriting point sequence, the sequence among the handwriting, the pause time, the speed characteristic, the handwriting jump and the handwriting fluency; the handwriting point sequence comprises the sequence of each point in the handwriting, and as the time information is arranged on the handwriting points, the sequence of the points on the handwriting can be obtained through the time information; the sequence among different handwriting and the pause time can be obtained by comparing the time of the upper point of the different handwriting; calculating a speed characteristic by dividing the length of the handwriting by the time difference between the time of the first point and the time of the last point of the handwriting; handwriting jump refers to the distance between different handwriting, wherein the tail point of the previous handwriting is away from the starting point of the next handwriting;
the handwriting space features include: handwriting area distribution, breakpoint proportion, breakpoint number, area density and handwriting continuity;
the handwriting distribution area features can be represented by a matrix, and the handwriting point distribution quantity of each area is used as a corresponding position element of the matrix; the number of break points is handwriting points with the length smaller than the preset length, and the number of break points is compared with the total number of handwriting points to obtain the proportion of break points; handwriting continuity refers to the ratio of the length of the handwriting to the average length of the handwriting; region density refers to the density of points per region;
the handwriting modification features comprise modification adjustment, repetition and deletion, and represent modification processing processes of users; a back and erasing button is displayed in a screen area of the electronic board, when a user clicks the back or erasing button in the drawing process, the system records the operation and modifies the corresponding handwriting to obtain a modified record (mode, L, t), and the mode represents back or erasing; l represents the corresponding handwriting, and t represents the current time.
9. The projection data acquisition method of claim 6, wherein the physiological characteristic is heart rate, pen holding pressure, blood oxygen saturation, skin electricity, skin temperature of the user at the time of drawing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110919114.6A CN113591779B (en) | 2021-08-11 | 2021-08-11 | Projection data acquisition system and method based on integrated multi-sensor handwriting |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110919114.6A CN113591779B (en) | 2021-08-11 | 2021-08-11 | Projection data acquisition system and method based on integrated multi-sensor handwriting |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113591779A CN113591779A (en) | 2021-11-02 |
| CN113591779B true CN113591779B (en) | 2023-04-28 |
Family
ID=78257200
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110919114.6A Active CN113591779B (en) | 2021-08-11 | 2021-08-11 | Projection data acquisition system and method based on integrated multi-sensor handwriting |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113591779B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114822113A (en) * | 2022-02-25 | 2022-07-29 | 杭州小伴熊科技有限公司 | Drawing teaching system for online education |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20140125207A (en) * | 2013-04-18 | 2014-10-28 | (주) 마이크로사이텍 | Method and apparatus for analyzing image for comprehending testee's psychology and recording medium thereof |
| CN211222736U (en) * | 2019-09-06 | 2020-08-11 | 四川爱联科技有限公司 | Healthy wisdom pen based on NB-IOT |
| CN111671446A (en) * | 2020-06-19 | 2020-09-18 | 智恩陪心(北京)科技有限公司 | Intelligent drawing psychological analysis system and method based on deep learning |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5058064B2 (en) * | 2008-01-29 | 2012-10-24 | 成 寅 金 | Picture therapy pattern selection and arrangement, color painting analysis method, psychological symptom diagnosis method based on picture therapy pattern analysis, and recording medium on which a program for doing this is recorded |
| CN113918072A (en) * | 2015-08-04 | 2022-01-11 | 株式会社和冠 | Display control method, computer, storage medium, and method |
-
2021
- 2021-08-11 CN CN202110919114.6A patent/CN113591779B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20140125207A (en) * | 2013-04-18 | 2014-10-28 | (주) 마이크로사이텍 | Method and apparatus for analyzing image for comprehending testee's psychology and recording medium thereof |
| CN211222736U (en) * | 2019-09-06 | 2020-08-11 | 四川爱联科技有限公司 | Healthy wisdom pen based on NB-IOT |
| CN111671446A (en) * | 2020-06-19 | 2020-09-18 | 智恩陪心(北京)科技有限公司 | Intelligent drawing psychological analysis system and method based on deep learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113591779A (en) | 2021-11-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10019690B2 (en) | Intellectual-productivity analysis apparatus and program | |
| CN111563887A (en) | Intelligent analysis method and device for oral cavity image | |
| CN107783718B (en) | Online homework/examination input method and device based on paper handwriting | |
| CN103154953A (en) | Measuring affective data for web-enabled applications | |
| JP2010131280A (en) | Method and apparatus for assisting mental state determination | |
| CN113591779B (en) | Projection data acquisition system and method based on integrated multi-sensor handwriting | |
| CN111938670B (en) | Depression identification method and system based on human skeleton kinematics characteristic information | |
| CN111597990B (en) | RSVP (resource reservation protocol) model-based brain-computer combined target detection method and system | |
| CN104156717A (en) | Method for recognizing rule breaking of phoning of driver during driving based on image processing technology | |
| US20240062582A1 (en) | Method and Device for Dynamic Recognition of Emotion Based on Facial Muscle Movement Monitoring | |
| CN109472257B (en) | Character layout determining method and device | |
| CN113822907B (en) | Image processing method and device | |
| KR102433303B1 (en) | Method, apparatus and program for drawing test using digital devices | |
| CN117095414A (en) | Handwriting recognition system and recognition method based on dot matrix paper pen | |
| CN109685143A (en) | A kind of thyroid gland technetium sweeps the identification model construction method and device of image | |
| US20230047530A1 (en) | Method and device for analyzing fine motor skills | |
| Zhao et al. | Depression detection based on human simple kinematic skeletal data | |
| CN115661491A (en) | Monitoring method for pest control in tea tree planting | |
| CN111831102A (en) | Expansion auxiliary system based on infant brain superiority assessment | |
| EP4339756A1 (en) | Intelligent drawing method and system for human machine interaction hmi design prototype | |
| CN112651347B (en) | Smoking behavior sample generation method and system based on double-spectrum imaging | |
| CN116311293B (en) | Test paper integral input method and system | |
| CN112906539B (en) | Object identification method based on EEG data | |
| CN112419112B (en) | Method and device for generating academic growth curve, electronic equipment and storage medium | |
| CN111383744B (en) | Medical microscopic image annotation information processing method and system and image analysis equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |