CN114067432A - Student classroom behavior recognition technology based on SSD-MobileNet V2 - Google Patents

Student classroom behavior recognition technology based on SSD-MobileNet V2 Download PDF

Info

Publication number
CN114067432A
CN114067432A CN202111314375.1A CN202111314375A CN114067432A CN 114067432 A CN114067432 A CN 114067432A CN 202111314375 A CN202111314375 A CN 202111314375A CN 114067432 A CN114067432 A CN 114067432A
Authority
CN
China
Prior art keywords
ssd
mobilenet
data set
students
student classroom
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111314375.1A
Other languages
Chinese (zh)
Inventor
尹淑娟
李洪星
张为
刘珂
刘浩
韩帅
李磊
张帅
武义超
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Inner Mongolia Uniersity Of Science And Technology Baotou Teachers' College
Original Assignee
Inner Mongolia Uniersity Of Science And Technology Baotou Teachers' College
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Inner Mongolia Uniersity Of Science And Technology Baotou Teachers' College filed Critical Inner Mongolia Uniersity Of Science And Technology Baotou Teachers' College
Priority to CN202111314375.1A priority Critical patent/CN114067432A/en
Publication of CN114067432A publication Critical patent/CN114067432A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10Services
    • G06Q50/20Education
    • G06Q50/205Education administration or guidance

Landscapes

  • Business, Economics & Management (AREA)
  • Engineering & Computer Science (AREA)
  • Educational Administration (AREA)
  • Educational Technology (AREA)
  • Strategic Management (AREA)
  • Tourism & Hospitality (AREA)
  • Health & Medical Sciences (AREA)
  • Economics (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Resources & Organizations (AREA)
  • Marketing (AREA)
  • Primary Health Care (AREA)
  • Physics & Mathematics (AREA)
  • General Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Electrically Operated Instructional Devices (AREA)

Abstract

The invention relates to the technical field of behavior recognition, and discloses a student classroom behavior recognition technology based on SSD-MobileNet V2, which comprises the following steps: constructing a student classroom behavior data set; step two: data preprocessing and data set partitioning; step three: using a MobileNet V2 pre-training model to perform transfer learning, and then loading the pre-processed VOC format data set into an SSD-MobileNet V2 multi-scale target detection network model for visual training; step four: and loading the data set to a MobileNet V2 feature extraction network to realize multi-feature extraction, outputting the target score and the bounding box regression parameter of the predicted picture in the SSD multi-scale target detection network, and finally outputting the prediction result through a maximum suppression algorithm NMS. The intelligent classroom teaching system accurately identifies abnormal behaviors of students, forms feedback data of classroom participation of the students, assists classroom teaching, realizes intelligent classrooms, promotes education intellectualization, and has important significance for improving teaching modes and improving teaching quality; the video can be observed manually, and the working efficiency and the analysis accuracy are improved.

Description

Student classroom behavior recognition technology based on SSD-MobileNet V2
Technical Field
The invention relates to the technical field of behavior recognition, in particular to student classroom behavior recognition technology based on SSD-MobileNet V2.
Background
In the modern society, industries with artificial intelligence technology as a core are spread over various industries. Indeed, the concept of artificial intelligence was proposed at the daltepo conference as early as 1956. Artificial intelligence is an important branch of computer science, which attempts to understand the intelligent connotation by simulating the thinking process of human brain, thereby realizing machine intelligence. Artificial intelligence has remarkable achievements in the fields of robots, image recognition, natural language processing, voice recognition, image segmentation and the like. In recent years, the research of artificial intelligence through deep learning becomes a more effective, more accurate and more adaptive way. With the continuous development of the times, the application of deep learning in the field of education is vigorous.
Classroom teaching activities are one of the most important and basic links of school research student education and are important forms for evaluating teaching levels of school teachers. The class behaviors of students can be divided into two types, one type is a learning behavior, and the learning behavior refers to a class participation state and an interaction state of the students under the guidance of teachers; the other is non-learning behavior. The interaction state specifically refers to whether the student holds hands to answer questions or not. The classroom participation state specifically means whether the student listens to the talk seriously, remembers the note seriously, holds a look in the east, sleeps, plays a mobile phone or not. The classroom behavior of the students can be further evaluated by deeply researching the classroom behavior of the students, the teaching behavior of the teacher can be evaluated simultaneously, the teaching mode can be timely improved by the teacher, and the teaching level of the teacher is further improved. Therefore, the observation of the classroom behavior of students is of great significance to the improvement of the teaching quality.
In order to realize classroom behavior analysis, teachers can work better. Traditional classroom behavior analysis relies mainly on the video of artifical observation record teacher and student's on class, and then the problem in the analysis teaching. The mode of manually observing video and analyzing the classroom behavior of students is time-consuming and labor-consuming and has low efficiency. Aiming at the requirement, the invention provides student classroom behavior recognition based on deep learning.
Disclosure of Invention
In view of the above-mentioned shortcomings in the prior art, the present invention aims to provide a student classroom behavior recognition technology based on SSD-MobileNetV 2.
In order to achieve the purpose, the invention adopts the following technical scheme:
a student classroom behavior recognition technology based on SSD-MobileNet V2 comprises the following steps:
the method comprises the following steps: configuring a Pytrich1.2 deep learning frame under windows, and automatically constructing a student classroom behavior data set;
step two: preprocessing data and dividing a data set, uniformly adjusting photos of the data set into a JPG format, marking out classroom behaviors to be detected by using a LabelImg tool, and manufacturing a VOC format data set;
step three: using a MobileNet V2 pre-training model to perform transfer learning, and then loading the pre-processed VOC format data set into an SSD-MobileNet V2 multi-scale target detection network model for visual training;
step four: and loading the data set to a MobileNet V2 feature extraction network to realize multi-feature extraction, outputting the target score and the bounding box regression parameter of the predicted picture in the SSD multi-scale target detection network, and finally outputting the prediction result through a maximum suppression algorithm NMS.
Further, a student classroom behavior recognition system is designed based on PyQt5, and student classroom behavior recognition visualization is realized.
Preferably, in the first step, eight classes of student classroom behaviors, namely ' reading, sleeping ', playing mobile phone ', ' standing up ', ' lifting hand ', ' listening class ', ' distracting ' and ' writing ', are selected, and the judgment standard is as follows: reading-students watch on textbooks, posture correcting, sleeping-students lie on their upper half on the desktop, playing mobile phones-students play mobile phones on the desktop, standing up-students stand up behind their desks, and can observe upper half of their bodies, raising their hands-students lift their left or right hands, listening-in-class-students watch on the blackboard or PPT, posture correcting, distracting-students observe left and right, look around, write-students hold pen and write.
Further, the size of the VOC format data set is 640, Batch _ size is 16, and Iteration is 12000, so that 300 epochs are completed after completing one Iteration of Iteration, where Batch _ size is the number of video memory pictures read in at one time, Iteration is the number of iterations, and Epoch is the number of times of traversing the entire training set.
Further, in the third step, the training parameters include Learning rate, Momentum, attenuation coefficient, Batch _ size, num _ classes, Learning _ rate, Momentum, and Decay _ steps.
In a preferred embodiment of the invention, in the first step, a total of 800 photos of eight classes of student classroom behaviors are screened out from the collected data, and a student classroom behavior data set is constructed.
In a preferred embodiment of the present invention, in the second step, the photos of the data set are uniformly adjusted to the JPG format, and the size is 500 × 375.
Furthermore, the GUI interface of the student classroom behavior recognition system comprises an input picture, an output prediction picture, a recognition result, accuracy and an FPS, and the back-end core algorithm adopts an SSD-Mobile Net V2 multi-scale target detection algorithm to perform eight classes of classroom behavior recognition on the input picture.
Compared with the prior art, the invention has the following beneficial effects:
(1) the method and the device have the advantages that the classroom behaviors of the students are identified based on the single image, the abnormal behaviors of the students are accurately identified, the feedback data of the classroom participation degree of the students are formed, the classroom teaching is assisted, an intelligent classroom is realized, the intellectualization of education is promoted, and the method and the device have important significance for improving the teaching mode and improving the teaching quality;
(2) the invention can replace manual video observation, analyze the classroom behavior of students and improve the working efficiency and the analysis accuracy.
(3) The invention relates to deep learning, a convolutional neural network, a transfer learning theory and a Pythrch deep learning framework and lays a foundation for subsequently realizing a student classroom behavior recognition algorithm.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is an illustration of eight classes of classroom behavior, wherein 1-1 is reading, 1-2 is sleeping, 1-3 is standing, 1-4 is raising hands, 1-5 is playing mobile phones, 1-6 is vague, 1-7 is writing, and 1-8 is listening;
FIG. 2 is a training flow diagram;
FIG. 3 is a prediction flow chart;
FIG. 4 is a classroom behavior recognition system GUI interface;
fig. 5 shows the recognition accuracy (AP values and maps values) of the eight classes of classroom behavior;
FIG. 6 is a graph showing the recognition results, in which 6-1 is reading, 6-2 is sleeping, 6-3 is vague, 6-4 is learning, 6-5 is standing, 6-6 is playing mobile phone, 6-7 is lifting hand, and 6-8 is writing.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications can be made by persons skilled in the art without departing from the spirit of the invention. All falling within the scope of the present invention.
A student classroom behavior recognition technology based on SSD-MobileNet V2 comprises the following steps:
the method comprises the following steps: configuring a Pytrich1.2 deep learning frame under windows, screening out 800 pictures of eight classes of student classroom behaviors from the collected data, and automatically constructing a student classroom behavior data set;
the data set is important for designing and verifying the network model, and the quantity and quality of the data set can influence the training effect of the network model to a great extent.
The classroom behaviors of students are rich and diverse, and can be divided into learning states and non-learning states according to different properties. In the teaching process, the problem interaction between the teacher and the students can directly reflect the activity degree of the classroom and indirectly reflect the advantages and disadvantages of the teaching activities of the teacher. The definition of the Student classroom behavior refers to an S-T (Student-Teacher) analysis method, and the Student classroom behavior is presented in a single-frame photo for analysis, so that quantitative analysis of teaching activities is formed.
Finally, according to an S-T analysis method, eight classes of student classroom behavior researches of reading, sleeping, playing mobile phones, standing up, lifting hands, listening, distracting and writing are selected, the judgment standards are shown in a table 1, and an example diagram of the eight classes of classroom behaviors is shown in a figure 1.
TABLE 1 eight classes of classroom behavior
Action name Judgment criteria
Reading (reading) Student staring at textbook and correcting posture
Sleep (sleeping) The upper body of the student lies on the desk top
Mobile phone (Phone) Mobile phone for students playing on desktop
Standing up (standing) Standing up after the student desk can observe the upper part of the body
Hand lifting (raise) Student lifts left or right hand
Teaching (learning) The student looks at the blackboard or PPT and has correct posture
Nerve (distraction) The students observe left and right and look around
Writing (writing) Writing pen holding posture for students
Step two: preprocessing data and dividing a data set, uniformly adjusting photos of the data set into a JPG format with the size of 500 multiplied by 375, marking out classroom behaviors to be detected by using a LabelImg tool, and manufacturing a VOC format data set.
Step three: and (3) performing transfer learning by using a MobileNet V2 pre-training model, and then loading the pre-processed VOC format data set into an SSD-MobileNet V2 multi-scale target detection network model for visual training. And observing the variation of the Loss value in the training process, and storing the training model each time. And adjusting the number of the epochs and fine-tuning the training parameters and the network model structure according to the actual training requirements.
Training process and parameter setting:
the student classroom behavior recognition network training process is as shown in fig. 2, and before starting model training, the preprocessed data set needs to be stored in the format of paschaloc data set, so that the SSD-MobilNetV2 network can effectively read the training data set. The Batch _ size is the number of pictures of the video memory read at one time and is limited by the size of the video memory. Iteration is the number of iterations and Epoch is the number of traversals through the entire training set. Since the data set size is 640, Batch _ size is 16, and operation is 12000, the execution of the operation completes 300 epochs.
The training parameters include learning rate, momentum, attenuation coefficient, Batch _ size, num _ classes, and the like. Updating the learning rate influence weight; momentum can improve the convergence speed of the model; the weight attenuation coefficient can effectively reduce the phenomenon of model overfitting; when training is terminated in a special condition, time waste is avoided, parameters need to be set to enable the training script to load the most recent pre-training weight for continuous training, and the training parameters are shown in table 2.
TABLE 2
Parameter name Initial value
num_class 9
Batch_size 16
Learning_rate 0.5
Momentum 0.9
Decay_steps 3000
Step four: the student classroom behavior prediction process is as shown in fig. 3, and a prediction data set is loaded to a MobileNetV2 feature extraction network to realize multi-feature extraction, so that a foundation is laid for subsequent multi-scale target detection. The feature extraction network is formed by stacking inverse residual error structures, target scores and boundary frame regression parameters of predicted pictures are output in a deep SSD multi-scale target detection network, and finally, prediction results are output through a maximum value suppression algorithm NMS.
Further, a student classroom behavior recognition system is designed based on PyQt5, namely a student classroom behavior recognition GUI interface is designed, and student classroom behavior recognition visualization is realized.
In order to more conveniently realize the prediction of the student classroom behavior, a student classroom behavior recognition system is designed based on PyQt 5. The GUI interface of the system comprises an input picture, an output prediction picture, an identification result, accuracy and FPS, the back-end core algorithm adopts the SSD-MobileNet V2 multi-scale target detection algorithm to identify eight classes of classroom behaviors of students on the input picture, and the GUI interface is shown in figure 4.
Further, the inventors performed model evaluation and performed prediction experiments, and the model evaluation results and prediction results were as follows:
and (3) model evaluation:
one of the most important evaluation indexes of the target detection network model is AP and mAP, wherein the AP value represents the recognition effect of the trained network model on each category; the mAP value represents the recognition effect of the trained network model on all classes. After the SSD-MobileNetV2 network model is visually trained, Precision and Recall are used to calculate AP and mAP values corresponding to eight classes (reading, sleeping, playing mobile phone, standing up, raising hands, listening, writing, and moving) of student classroom behavior, as shown in fig. 5.
And (4) predicting results:
the student classroom behavior recognition system uploads an image to be recognized, and the image is preprocessed, so that the feature extraction network transmits multi-scale feature information into the target detection network to recognize the multi-scale feature information. The recognition result is shown in fig. 6.
Therefore, the student classroom behavior recognition system designed by the invention can visually display and analyze the recognition result, so that the purposes of assisting teachers in classroom teaching, constructing a smart classroom and further promoting the intellectualization of the education field are achieved.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention.

Claims (8)

1. A student classroom behavior identification technology based on SSD-MobileNet V2 is characterized by comprising the following steps:
the method comprises the following steps: configuring a Pytrich1.2 deep learning frame under windows, and automatically constructing a student classroom behavior data set;
step two: preprocessing data and dividing a data set, uniformly adjusting photos of the data set into a JPG format, marking out classroom behaviors to be detected by using a LabelImg tool, and manufacturing a VOC format data set;
step three: using a MobileNet V2 pre-training model to perform transfer learning, and then loading the pre-processed VOC format data set into an SSD-MobileNet V2 multi-scale target detection network model for visual training;
step four: and loading the data set to a MobileNet V2 feature extraction network to realize multi-feature extraction, outputting the target score and the bounding box regression parameter of the predicted picture in the SSD multi-scale target detection network, and finally outputting the prediction result through a maximum suppression algorithm NMS.
2. The student classroom behavior recognition technology based on SSD-MobileNetV2 of claim 1, wherein the student classroom behavior recognition system is designed based on PyQt5 to realize student classroom behavior recognition visualization.
3. The student classroom behavior recognition technology based on SSD-MobileNetV2 as claimed in claim 1, wherein in step one, eight classes of student classroom behaviors of reading, sleeping, playing mobile phone, standing up, lifting hands, listening class, leaving nerve and writing are selected, and the judgment criteria are: reading-students watch on textbooks, posture correcting, sleeping-students lie on their upper half on the desktop, playing mobile phones-students play mobile phones on the desktop, standing up-students stand up behind their desks, and can observe upper half of their bodies, raising their hands-students lift their left or right hands, listening-in-class-students watch on the blackboard or PPT, posture correcting, distracting-students observe left and right, look around, write-students hold pen and write.
4. The student classroom behavior recognition technology based on SSD-MobileNetV2 of claim 1, wherein the VOC format data set size is 640, Batch _ size is 16, and Iteration is 12000, so that performing one Iteration completes 300 epochs, where Batch _ size is the number of video pictures read in at one time, Iteration is the number of iterations, and Epoch is the number of times of traversing the entire training set.
5. The student class behavior recognition technology based on SSD-MobileNet V2 of claim 1, wherein in step three, the training parameters comprise Learning rate, Momentum, attenuation coefficient, Batch _ size, num _ classes, Learning _ rate, Momentum, and Decay _ steps.
6. The SSD-MobileNetV 2-based student classroom behavior recognition technology of claim 1, wherein in step one, eight classes of student classroom behavior photos are screened out from the collected data, and a student classroom behavior data set is constructed in a total of 800 photos.
7. The student classroom behavior recognition technology based on SSD-MobileNetV2 of claim 1, wherein in step two, photos of data sets are uniformly adjusted to JPG format with a size of 500 x 375.
8. The SSD-MobileNet V2-based student classroom behavior recognition technology according to claim 2, wherein the GUI interface of the student classroom behavior recognition system comprises an input picture, an output predicted picture, a recognition result, accuracy and FPS, and the back-end core algorithm adopts SSD-MobileNet V2 multi-scale object detection algorithm to recognize eight classes of classroom behaviors of the input picture.
CN202111314375.1A 2021-11-08 2021-11-08 Student classroom behavior recognition technology based on SSD-MobileNet V2 Pending CN114067432A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111314375.1A CN114067432A (en) 2021-11-08 2021-11-08 Student classroom behavior recognition technology based on SSD-MobileNet V2

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111314375.1A CN114067432A (en) 2021-11-08 2021-11-08 Student classroom behavior recognition technology based on SSD-MobileNet V2

Publications (1)

Publication Number Publication Date
CN114067432A true CN114067432A (en) 2022-02-18

Family

ID=80274206

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111314375.1A Pending CN114067432A (en) 2021-11-08 2021-11-08 Student classroom behavior recognition technology based on SSD-MobileNet V2

Country Status (1)

Country Link
CN (1) CN114067432A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116596719A (en) * 2023-07-18 2023-08-15 江西科技学院 Computer room computer teaching quality management system and method

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116596719A (en) * 2023-07-18 2023-08-15 江西科技学院 Computer room computer teaching quality management system and method
CN116596719B (en) * 2023-07-18 2023-09-19 江西科技学院 Computer room computer teaching quality management system and method

Similar Documents

Publication Publication Date Title
CN108648757A (en) A kind of analysis method based on various dimensions Classroom Information
CN110009210B (en) Comprehensive assessment method for student class listening level based on attention degree and concentration degree
CN112163162A (en) Portrait recognition-based selected course recommendation method, storage medium and electronic equipment
Yurtseven et al. Understanding by Design (UbD) in EFL Teaching: The Investigation of Students' Foreign Language Learning Motivation and Views.
CN106205254A (en) Guiding learning system and method
CN106897384A (en) One kind will bring out the theme automatic evaluation method and device
CN114067432A (en) Student classroom behavior recognition technology based on SSD-MobileNet V2
CN113657723A (en) Knowledge point mastering condition diagnosis method, test question recommendation method and device
Yildirim et al. Experience of traditional teaching methods in architectural design education:“mimesis technique”
CN109284944A (en) A kind of classroom instruction interaction liveness evaluation system based on machine vision
JP2022014890A (en) Concentration determination program
CN108009537A (en) A kind of method for identifying questionnaire and correcting score
CN112614033A (en) System for be used for college physics experiment teaching
CN109448464A (en) A kind of English- word spelling exercising method
Kozminsky et al. Concept mapping and writing
CN105528505A (en) Online automatic learning style diagnosis system and method
Yi et al. Effect of Using Logo on Pupils' Learning in Two-Dimensional Shapes.
CN112382138A (en) Teaching experiment of internet + experiment teaching method
Stillman et al. Empirical Evidence for Niss'" Implemented Anticipation" in Mathematising Realistic Situations.
Chen et al. Student classroom behavior recognition based on openpose and deep learning
CN112634100A (en) Behavior analysis education system based on big data
Herold et al. Characterizing students handwritten self-explanations
Liu The Detection of English Students’ Classroom Learning State in Higher Vocational Colleges Based on Improved SSD Algorithm
Erten Technological Pedagogical Content Knowledge (TPACK) Proficiency of Preservice Teachers Who Will Perform Language Teaching as a Foreign Language
Lei The Curriculum Goal Achievement Evaluation Design of IC Design and Integrated System Excellent Engineer CDIO Training Mode

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination