WO2022100358A1 - 一种便携式多模态学习分析智能眼镜 - Google Patents

一种便携式多模态学习分析智能眼镜 Download PDF

Info

Publication number
WO2022100358A1
WO2022100358A1 PCT/CN2021/123942 CN2021123942W WO2022100358A1 WO 2022100358 A1 WO2022100358 A1 WO 2022100358A1 CN 2021123942 W CN2021123942 W CN 2021123942W WO 2022100358 A1 WO2022100358 A1 WO 2022100358A1
Authority
WO
WIPO (PCT)
Prior art keywords
analysis
data
learning
module
modal
Prior art date
Application number
PCT/CN2021/123942
Other languages
English (en)
French (fr)
Inventor
欧阳璠
焦鹏程
Original Assignee
浙江大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 浙江大学 filed Critical 浙江大学
Publication of WO2022100358A1 publication Critical patent/WO2022100358A1/zh
Priority to US18/107,490 priority Critical patent/US20230296923A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02C—SPECTACLES; SUNGLASSES OR GOGGLES INSOFAR AS THEY HAVE THE SAME FEATURES AS SPECTACLES; CONTACT LENSES
    • G02C11/00—Non-optical adjuncts; Attachment thereof
    • G02C11/10—Electronic devices other than hearing aids
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B5/00—Electrically-operated educational appliances
    • G09B5/08—Electrically-operated educational appliances providing for individual presentation of information to a plurality of student stations
    • G09B5/12—Electrically-operated educational appliances providing for individual presentation of information to a plurality of student stations different stations being capable of presenting different information simultaneously
    • G09B5/125—Electrically-operated educational appliances providing for individual presentation of information to a plurality of student stations different stations being capable of presenting different information simultaneously the stations being mobile
    • G—PHYSICS
    • G02—OPTICS
    • G02C—SPECTACLES; SUNGLASSES OR GOGGLES INSOFAR AS THEY HAVE THE SAME FEATURES AS SPECTACLES; CONTACT LENSES
    • G02C5/00—Constructions of non-optical parts
    • G02C5/001—Constructions of non-optical parts specially adapted for particular purposes, not otherwise provided for or not fully classifiable according to technical characteristics, e.g. therapeutic glasses
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/94—Hardware or software architectures specially adapted for image or video understanding
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/70—Multimodal biometrics, e.g. combining information from different biometric modalities
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B5/00—Electrically-operated educational appliances
    • G09B5/06—Electrically-operated educational appliances with both visual and audible presentation of the material to be studied
    • G—PHYSICS
    • G02—OPTICS
    • G02C—SPECTACLES; SUNGLASSES OR GOGGLES INSOFAR AS THEY HAVE THE SAME FEATURES AS SPECTACLES; CONTACT LENSES
    • G02C2200/00—Generic mechanical aspects applicable to one or more of the groups G02C1/00 - G02C5/00 and G02C9/00 - G02C13/00 and their subgroups
    • G02C2200/18—Adjustment ridges or notches
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/26—Speech to text systems

Definitions

  • the invention relates to the technical field of online education, in particular to portable multimodal learning and analysis smart glasses.
  • Learning analytics an emerging subfield in pedagogy - has had a major impact on computer-supported learning and teaching.
  • Learning analytics is the measurement, collection, analysis and reporting of data on learners and their learning environment; its purpose is to better understand and optimize the learning process and learning environment.
  • the research and application of multimodal learning analysis is one of the important branches in the field of learning analysis.
  • Multimodal learning analysis requires the use of various analysis methods and technical means to collect, analyze and mine multimodal data (such as physiological indicators, voice, micro-expressions, etc.) Reporting to users to help users understand and master learning-related information.
  • Multimodal learning analysis can acquire the data in the learning process of learners through sensors and other methods, then model the multimodal data through artificial intelligence algorithms (such as machine learning), and finally interpret and visualize the results of the modal data.
  • a learning analysis module which uses image acquisition peripherals and audio acquisition peripherals to monitor the dynamics of student users in the learning process in real time; on the other hand, the learning analysis method can be programmed through the chip module.
  • the innovatively designed learning analysis peripheral auxiliary tools are used to carry out more precise, optimized and targeted dynamic monitoring process of micro-expressions, physiological indicators and other factors of student users in the learning process.
  • the present invention proposes a portable multimodal learning analysis smart glasses for real-time and accurate multimodal information acquisition, analysis and feedback, thereby realizing the innovation of intelligent learning analysis.
  • the present invention innovatively designs a portable multi-modal learning and analysis smart glasses, which captures and analyzes multi-modal data for students' learning process, so as to realize optimal dynamic monitoring and feedback.
  • the present invention is a student-centered, student-oriented multi-modal learning analysis device, which can more directly provide students with opportunities to explore, understand and reflect on the learning process, thereby mobilizing students' learning autonomy. , dynamism and innovation.
  • the multi-modal portable device can solve the problems that the current analysis and presentation of learning analysis results in the field of education is lagging, and the learning status and learning process cannot be presented in multiple dimensions, and it is difficult for students to interpret the learning analysis results from multiple angles and guide their next learning. .
  • the portable multi-modal learning and analysis smart glasses utilize data reading and analysis functions to dynamically read data from students' real-time learning process, process and analyze them, and finally display the data analysis results to students.
  • the portable multi-modal learning and analysis smart glasses use sensors and other means to obtain data in the learning process of learners, establish an artificial intelligence algorithm model to model the multi-modal data, and finally visualize the data analysis results. Presented to help students understand their dynamically changing learning states and processes.
  • the purpose of the present invention is to propose a portable multi-modal learning and analysis smart glasses aiming at the deficiencies of the prior art.
  • a portable multi-modal learning and analysis smart glasses the smart glasses are composed of a frame module, a smart screen module, and a glasses carrier module; and the multi-modal learning and analysis method is used in real time. Capture and analyze multimodal data generated by learner users, and feed back the results of visual analysis to users;
  • the mirror frame module includes a front compartment, a mobile compartment and a rear compartment.
  • the front compartment is provided with an audio device and a chip module outside, and a gyroscope module is provided inside.
  • the audio device provides a voice recognition function, and the language data is converted into text in real time through the audio device, and the head movement is monitored through the gyroscope module.
  • the chip module provides data analysis function, reads student data in real time for analysis, and stores the data in a preset data structure;
  • the mobile compartment is provided with a vertical downward base camera, a roller, a flexible screen, an exit slot and a flexible screen guide rail; wherein, the function of monitoring eye movements and facial expressions is realized by the vertical downward base camera; the roller is connected with the flexible screen in rotation , the flexible screen is connected to the frame module through the flexible screen guide rail, and the flexible screen can slide out from the exit slot of the frame module along the flexible screen guide rail;
  • the smart glasses visualize the processing results of the data reading and analysis functions, and feed them back to the students through flexible screens in various ways of visualizing graphics.
  • the rear compartment is arranged outside the frame module, and a physiological index sensor is installed on the rear compartment; the physiological index sensor is used to monitor the physiological index.
  • the frame module, the smart screen module and the glasses carrier module are connected by a circuit arranged in the frame, and a chip module is arranged in the front compartment to process the physical sign data collected in real time.
  • the front compartment is also provided with a charging interface, a switch and a wireless transmitter outside;
  • the interior is also provided with a micro laser limit switch, a micro gear, a chip module, a cable, a battery and a micro rack.
  • the components are installed on the front compartment in the order of the labels; the battery is connected to the charging interface and the switch through the cable, and supplies power to the entire glasses, and the wireless transmitter is connected to the chip module; at the same time, the micro laser limit switch is connected to the micro gear, and the micro gear In cooperation with the miniature rack, the miniature rack is connected with the mobile bin, and the micro laser limit switch constrains the front and rear position limit of the mobile bin by scanning the movement of the mobile bin.
  • the rear compartment is also provided with a clamp, a wireless earphone, an ear hook and a hanging ring.
  • the clips are clamped on the temples of the smart glasses, and fixed on the wearer's ears through ear hooks and rings, and receive voice information through wireless earphones.
  • the audio device converts the language data into text in real time by analyzing and encoding the speech content by using technologies such as natural language processing.
  • the flexible screen can be used not only to assist eyesight in daily life, but also to display the processing result of the chip module in time, so as to realize the function of the display screen.
  • the speech recognition and analysis results can extract keywords through algorithms such as stuttering word segmentation, and use visual graphics such as keyword clustering to display the characteristics of learners' conversations.
  • the results of head motion analysis can show the learner's current movements such as raising, lowering or turning his head, and feedback the relevant frequency to the learner.
  • the base camera recognizes the facial expressions, analyzes the results through the chip module, and then displays the learners' current psychological and emotional characteristics through text.
  • the recognition and analysis results of the facial expressions include different psychological states such as the learner's concentration, pleasure, and confusion; the eye movement results monitored by the base camera can display the learning objects that the learners are currently paying attention to, and will focus on the duration, objects, and the number of eye movements. The results are fed back to the learner.
  • the multimodal data includes multimodal type data including expressions, voice, physiology, eyeball and head movement and other indicators.
  • results of physiological indicators such as heart rate and galvanic skin indicators are displayed to the learners to display their current physical characteristics and conditions.
  • the portable multi-modal learning analysis smart glasses proposed by the present invention directly provide learners with opportunities to explore, understand and reflect on the learning process and state, which can solve the learning analysis problems existing in the existing learning analysis tools in the education field.
  • the results showed lag, inability to present the learning process in multiple dimensions, and difficulties for students to interpret the results of the learning analysis and guide their next learning.
  • This technology uses data reading and analysis functions to acquire multi-modal data of learners in real time, automatically analyze data, and display the results of data analysis, so as to mobilize students' learning autonomy, initiative and innovation,
  • Figure 1 is a schematic structural diagram of the present invention.
  • Fig. 2-Fig. 5 are structural details diagrams of the present invention.
  • FIG. 6 is a keyword clustering diagram of an embodiment of the present invention.
  • the present invention provides a portable multi-modal learning and analysis smart glasses.
  • the smart glasses are composed of a frame module 1, a smart screen module 2, and a glasses carrier module 3.
  • the module 2 and the glasses carrier module 3 are connected by a circuit arranged in the frame; and the multimodal data generated by the learner user is captured and analyzed in real time by the multimodal learning analysis method, and the visual analysis result is fed back to the user;
  • the described Multimodal data includes multimodal data including expressions, speech, physiology, eye and head movements.
  • the frame module 1 includes a front compartment 4 , a moving compartment 5 and a rear compartment 6 .
  • the front compartment 4 is provided with an audio device 401 and a chip module 408 outside, and a gyroscope module 407 is provided inside.
  • the audio device 401 provides a voice recognition function, and the language data is converted into text in real time through the audio device 401. to monitor head movement.
  • the chip module 408 provides a data analysis function to process the vital sign data collected in real time, analyze the data, and store the data in a preset data structure;
  • the audio device 401 converts the language data into text in real time by analyzing and encoding the speech content using technologies such as natural language processing.
  • the speech recognition and analysis results can be used to extract keywords through algorithms such as stuttering word segmentation, as shown in Figure 6, and visual graphics such as keyword clustering are used to display the characteristics of learners' conversations.
  • the results of head motion analysis can show the learner's current movements such as raising, lowering or turning his head, and feedback the relevant frequency to the learner.
  • the mobile compartment 5 is provided with a vertically downward base camera 501, a roller 502, a flexible screen 503, an exit slot 504 and a flexible screen guide rail 505; wherein, the vertical downward base camera 501 realizes the function of monitoring eye movement and facial expressions;
  • the roller 502 is rotatably connected to the flexible screen 503, the flexible screen 503 is connected to the frame module 1 through the flexible screen guide rail 505, and the flexible screen 503 can slide out from the outlet slot 504 of the frame module 1 along the flexible screen guide rail 505;
  • the base camera 501 recognizes the facial expression, analyzes the result through the chip module 408, and then displays the current psychological and emotional characteristics of the learner through text.
  • the recognition and analysis results of the facial expressions include different psychological states such as the learner's concentration, joy, confusion, etc.; the eye movement results monitored by the base camera 501 can display the learning object that the learner is currently paying attention to, and will focus on the duration, object, and eye movement times. Wait for the results to be fed back to the learners.
  • the flexible screen 503 can be used not only to assist eyesight in daily life, but also to display the results processed by the chip module 408 in time to realize the function of a display screen.
  • the smart glasses visualize the processing results of the data reading and analysis functions, and feed them back to the students through the flexible screen 503 in various ways of visualizing graphics.
  • the rear compartment 6 is disposed outside the frame module 1, and a physiological index sensor 602 is installed on the rear compartment 6; the physiological index sensor 602 is used to monitor the physiological index.
  • Physiological indicators such as heart rate and electrodermal index results are displayed to learners to show their current physical characteristics and conditions.
  • the front compartment 4 is also provided with a charging interface 402, a switch 403 and a wireless transmitter 404 on the outside; a micro laser limit switch 405, a micro gear 406, a chip module 408, a cable 409, a battery 410 and a micro gear are also provided inside.
  • the components are sequentially installed on the front compartment 4 in the order of the labels; the battery 410 is connected to the charging interface 402 and the switch 403 through the cable 409, and supplies power to the entire glasses, and the wireless transmitter 404 is connected to the chip module 408; At the same time, the micro laser limit switch 405 is connected with the micro gear 406 , the micro gear 406 is matched with the micro rack 411 , the micro rack 411 is connected with the moving bin 5 , and the micro laser limit switch 405 scans the movement of the moving bin 5 , and then restricts The front and rear position limit of the mobile bin 5.
  • the rear compartment 6 is also provided with a clamp 601 , a wireless earphone 603 , an ear hook 604 and a hanging ring 605 .
  • the clip 601 is clamped on the temple of the smart glasses, and is fixed on the wearer's ear through the ear hook 604 and the hanging ring 605 , and the voice information is received through the wireless earphone 603 .
  • the charging interface 402 is connected to an external power source for charging, and the electrical energy is stored in the battery 410 for the dynamic monitoring process of the frame module and the smart screen module.
  • the audio device 401 converts the language data into text in real time, and then uses technologies such as natural language processing to analyze and encode the text content, topic modeling and clustering technology to study the thinking process of the students.
  • the face shows the psychological state through expressions, and the eye movement can reflect the learner's gaze target and duration.
  • the vertical bottom camera 501 can realize the function of monitoring eye movement and facial expression by capturing the subtle movements of the face.
  • the gyroscope is an angular motion detection device that uses the moment-of-momentum-sensitive shell of the high-speed revolving body relative to the inertial space around one or two axes that are orthogonal to the rotation axis.
  • the gyroscope module 407 can accurately monitor the learner's head movement.
  • the physiological indicator sensor 602 can measure various physiological indicators including pulse, heartbeat, etc. through built-in sensors, and can obtain blood pressure values by analyzing the corresponding electrical signals generated by the pulse of the arteries, which is suitable for long-term continuous monitoring.
  • the wireless transmitter 404 performs data transmission; the wireless earphone 603 can be used to receive information feedback in real time. It is worth noting that this design is a learning aid tool based on multimodal learning analysis technology, a supplementary tool for the application of the learning analysis technology, but the learning analysis technology and application are not limited to such tools.

Abstract

一种便携式多模态学习分析智能眼镜，可在学习者学习过程中实时监测、分析并反馈包括表情、语音、生理、眼球及头部运动等指标在内的多模态数据及数据分析结果。该智能眼镜的芯片集成了实时数据监测功能、多模态数据分析功能以及数据可视化功能。数据监测功能通过实时获取学习者用户学习过程中表情、语音、生理、眼球运动和头部运动的变化情况；多模态数据分析功能将实时获取的数据存入预先设置的数据结构中进行多模态学习分析；数据可视化功能将数据分析的处理结果以可视化图形的方式显示给学习者用户。通过便携的智能眼镜实时捕捉学习者用户学习过程中产生的多模态数据、自动分析多模态数据、并可视化地显示数据分析结果。

Description

一种便携式多模态学习分析智能眼镜 技术领域
本发明涉及在线教育技术领域，尤其涉及一种便携式多模态学习分析智能眼镜。
背景技术
学习分析——教育学中一个新兴的分支领域——对计算机支持的学习和教学产生了重要影响。学习分析是对学习者及其学习环境的数据测量、收集、分析和汇报；其目的是为了更好地理解和优化学习过程以及学习环境。其中，多模态学习分析研究和应用是学习分析领域的重要分支之一。多模态学习分析需要利用各种分析方法和技术手段采集、分析和挖掘学生学习过程中产生的多模态数据(如生理指标、语音、微表情等)，并以一定的方式将数据分析结果汇报给用户以帮助用户了解和掌握与学习相关的信息。多模态学习分析的开发和应用——包含多模态数据采集、数据分析和结果解读及呈现——是将学习分析运用到教学实践中来支持并促进学习和教学的主要手段之一。
基于多模态学习分析方法，研究人员可通过设计多模态智能设备对学习和教学过程进行辅助。多模态学习分析可以通过传感器等方式获取学习者学习过程中的数据、再通过人工智能算法(如机器学习)对多模数据进行建模，最后对模态数据结果进行合理解读和可视化呈现。为了实现该目标，一方面需要建立学习分析模块，采用图像获取外设与音频获取外设同时对学生用户在学习过程中的动态进行实时监测；另一方面，可以通过芯片模块程控对学习分析方法进行建模，同时采用创新设计的学习分析周边辅助工具对学生用户在学习过程中的微表情、生理指标等因素进行更加精密、优化和针对的动态监测过程。目前智能眼镜或镜架的相关发展，均未涉及结合智能学习分析方法进行的设计发明：一方面，传统电脑无法针对用户个体的体征指标进行个性化信息获取；另一方面，针对利用眼镜作为载体进行创新设计以辅助智能学习分析方法的研究在国内研究中未见述及。因此，本发明提出了一种便携式多模态学习分析智能眼镜实时和准确多模态信息获取、分析和反馈，从而实现智能化学习分析的创新。
与现有技术相比，本发明创新地设计了一种便携式多模态学习分析智能眼镜，对学生学习过程进行多模态数据捕捉和分析，以实现最优化动态监测及反馈。与现有技术相比，本发明是以学生为中心的、面向学生的多模态学习分析设备，能更直接的提供给学生探索、理解和反思学习过程的机会，从而调动学生的学习自主性、能动性和创新性。该多模态便携设备可解决教育领域现有的学习分析结果分析和呈现滞后，且不能多维度呈现学习状态和学习过 程，以及学生难以从多角度解释学习分析结果并指导其下一步学习等难题。与现有技术相比，该便携式多模态学习分析智能眼镜利用数据读取和分析功能从学生实时学习过程中动态地读取数据并进行处理分析，最后给学生展示数据分析结果。与现有技术相比，这种便携式多模态学习分析智能眼镜利用传感器等方式获取学习者学习过程中的数据、建立人工智能算法模型对多模数据进行建模，最后将数据分析结果进行可视化呈现以帮助学生理解其动态变化的学习状态和过程。
发明内容
本发明目的在于针对现有技术的不足，提出一种便携式多模态学习分析智能眼镜。
本发明的目的是通过以下技术方案来实现的：一种便携式多模态学习分析智能眼镜，该智能眼镜由镜架模块、智能屏幕模块、眼镜载体模块组成；并利用多模态学习分析方法实时捕捉和分析学习者用户产生的多模态数据，并将可视化分析结果反馈给用户；
所述的镜架模块包括前仓、移动仓和后仓。所述的前仓外部设置有音频装置和芯片模块，内部设置有陀螺仪模块，音频装置提供语音识别功能，通过音频装置将语言数据实时转化为文本，通过陀螺仪模块来实现头部运动的监测。芯片模块提供数据分析功能，实时读取学生数据进行分析，并将数据存入预设的数据结构中；
所述的移动仓设置有垂直向下底座摄像头、滚轴、柔性屏幕、出口槽和柔性屏导轨；其中，通过垂向下底座摄像头实现监测眼球运动和面部表情功能；滚轴与柔性屏幕转动连接，柔性屏幕通过柔性屏导轨与镜架模块连接，柔性屏幕可延柔性屏导轨从镜架模块的出口槽处滑出；
智能眼镜将数据读取和分析功能的处理结果可视化，以多种可视化图形的方式通过柔性屏幕反馈给学生。
所述的后仓设置在镜架模块外部，后仓上安装有生理指标传感器；通过生理指标传感器来实现对生理指标的监测。
进一步地，所述的镜架模块、智能屏幕模块和眼镜载体模块通过设置在镜架内的线路连接，且前仓中设置芯片模块以处理实时采集的体征数据。
进一步地，所述的前仓外部还设置有充电接口、开关和无线传输器；内部还设置有微型激光限位开关、微型齿轮、芯片模块、排线、电池和微型齿条，所述各零部件按照标号顺序依次安装在前仓上；电池通过排线与充电接口、开关连接，并为整个眼镜供电，无线传输器与芯片模块连接；同时，微型激光限位开关与微型齿轮连接，微型齿轮与微型齿条配合，微型齿条与移动仓连接，微型激光限位开关通过扫描移动仓的运动，进而约束移动仓的前后位置极限。
进一步地，所述的后仓还设置有夹具、无线耳机、耳挂和挂环。所述夹具夹在智能眼镜的镜腿上，并通过耳挂和挂环固定在佩戴者的耳朵上，通过无线耳机接收语音信息。
进一步地，所述音频装置将语言数据实时转化为文本是利用自然语言处理等技术对语音内容进行分析与编码。
进一步地，柔性屏幕既可用以日常辅助视力，也可及时显示芯片模块处理的结果，实现显示屏的功能。
进一步地，语音识别和分析结果可通过结巴分词等算法提取关键词，利用关键词聚类等可视化图形方式展示学习者谈话特征。头部运动分析结果可展示学习者当前的抬头、低头或转头等动作，并将相关频率反馈给学习者。
进一步地，底座摄像头对到面部表情进行识别，通过芯片模块分析结果，进而通过文字方式展示学习者当前心理情绪特征。所述面部表情的识别分析结果包括学习者专注、愉悦、困惑等不同心理状态；底座摄像头监测到的眼球运动结果可展示学习者当前关注的学习对象，并将关注时长、对象、眼球运动次数等结果反馈给学习者。
进一步地，所述多模态数据包括表情、语音、生理、眼球及头部运动等指标在内的多模态类型数据。
进一步地，将生理指标如心率及皮肤电指标结果展示给学习者以显示其当前身体特征及状况。
本发明的有益效果：本发明提出的一种便携式多模态学习分析智能眼镜直接给学习者提供探索、理解和反思学习过程和状态的机会，可解决教育领域现有学习分析工具存在的学习分析结果显示滞后，不能多维度呈现学习过程，以及学生难以解释学习分析结果并指导其下一步学习等难题。此技术利用数据读取和分析功能实时获取学习者多模态数据、自动分析数据、并显示数据分析结果，从而调动学生的学习自主性、能动性和创新性，
附图说明
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本发明结构示意图。
图2-图5为本发明结构细节图；
图6为本发明实施例的关键词聚类图；
图中，1-镜架模块；2-智能屏幕模块；3-眼镜载体模块；4-前仓；401-音频装置；402- 充电接口；403-开关；404-无线传输器；405-微型激光限位开关；406-微型齿轮；407-陀螺仪模块；408-芯片模块；409-排线；410-电池；411-微型齿条；5-移动仓；501-垂向下底座摄像头；502-滚轴；503-柔性屏幕；504-出口槽；505-柔性屏导轨；6-后仓；601-夹具；602-生理指标传感器；603-无线耳机；604-耳挂；605-挂环。
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
如图1所示，本发明提供的一便携式多模态学习分析智能眼镜，该智能眼镜由镜架模块1、智能屏幕模块2、眼镜载体模块3组成，所述的镜架模块1、智能屏幕模块2和眼镜载体模块3通过设置在镜架内的线路连接；并利用多模态学习分析方法实时捕捉和分析学习者用户产生的多模态数据，并将可视化分析结果反馈给用户；所述多模态数据包括表情、语音、生理、眼球及头部运动等指标在内的多模态类型数据。
如图2、图3、图4和图5所示，所述的镜架模块1包括前仓4、移动仓5和后仓6。所述的前仓4外部设置有音频装置401和芯片模块408，内部设置有陀螺仪模块407，音频装置401提供语音识别功能，通过音频装置401将语言数据实时转化为文本，通过陀螺仪模块407来实现头部运动的监测。芯片模块408提供数据分析功能，以处理实时采集的体征数据，并对数据进行分析，将数据存入预设的数据结构中；
所述音频装置401将语言数据实时转化为文本是利用自然语言处理等技术对语音内容进行分析与编码。
语音识别和分析结果可通过结巴分词等算法提取关键词，如图6所示，利用关键词聚类等可视化图形方式展示学习者谈话特征。
头部运动分析结果可展示学习者当前的抬头、低头或转头等动作，并将相关频率反馈给学习者。
所述的移动仓5设置有垂直向下底座摄像头501、滚轴502、柔性屏幕503、出口槽504和柔性屏导轨505；其中，通过垂向下底座摄像头501实现监测眼球运动和面部表情功能；滚轴502与柔性屏幕503转动连接，柔性屏幕503通过柔性屏导轨505与镜架模块1连接，柔性屏幕503可延柔性屏导轨505从镜架模块1的出口槽504处滑出；
底座摄像头501对到面部表情进行识别，通过芯片模块408分析结果，进而通过文字方式展示学习者当前心理情绪特征。所述面部表情的识别分析结果包括学习者专注、愉悦、困 惑等不同心理状态；底座摄像头501监测到的眼球运动结果可展示学习者当前关注的学习对象，并将关注时长、对象、眼球运动次数等结果反馈给学习者。
柔性屏幕503既可用以日常辅助视力，也可及时显示芯片模块408处理的结果，实现显示屏的功能。
智能眼镜将数据读取和分析功能的处理结果可视化，以多种可视化图形的方式通过柔性屏幕503反馈给学生。
所述的后仓6设置在镜架模块1外部，后仓6上安装有生理指标传感器602；通过生理指标传感器602来实现对生理指标的监测。将生理指标如心率及皮肤电指标结果展示给学习者以显示其当前身体特征及状况。
所述的前仓4外部还设置有充电接口402、开关403和无线传输器404；内部还设置有微型激光限位开关405、微型齿轮406、芯片模块408、排线409、电池410和微型齿条411，所述各零部件按照标号顺序依次安装在前仓4上；电池410通过排线409与充电接口402、开关403连接，并为整个眼镜供电，无线传输器404与芯片模块408连接；同时，微型激光限位开关405与微型齿轮406连接，微型齿轮406与微型齿条411配合，微型齿条411与移动仓5连接，微型激光限位开关405通过扫描移动仓5的运动，进而约束移动仓5的前后位置极限。
所述的后仓6还设置有夹具601、无线耳机603、耳挂604和挂环605。所述夹具601夹在智能眼镜的镜腿上，并通过耳挂604和挂环605固定在佩戴者的耳朵上，通过无线耳机603接收语音信息。
以图1-图5为例，解释一种便携式多模态学习分析智能眼镜操作方法。首先通过充电接口402连接外部电源进行充电，电能储存在电池410中用于镜架模块、智能屏幕模块的动态监测过程。音频装置401将语言数据实时转化为文本，再利用自然语言处理等技术对文本内容分析与编码、主题建模和聚类技术研究学生的思维过程。人脸通过表情展示心理状态，眼球运动可以反映学习者的注视目标和时长，垂向下底座摄像头501通过对脸部细微动作的捕捉可以实现监测眼球运动和面部表情的功能。陀螺仪是用高速回转体的动量矩敏感壳体相对惯性空间绕正交于自转轴的一个或二个轴的角运动检测装置，利用陀螺仪模块407可以精确监测学习者的头部运动。生理指标传感器602通过内置传感器可以测量包括脉搏、心跳等多种生理指标，对动脉的搏动产生相应的电信号进行分析可以得到血压值，适合于长期连续监测。无线传输器404进行数据传输；无线耳机603可以用来实时接受信息反馈。值得注意的是，本设计为基于多模态学习分析技术的学习辅助工具，为所述的学习分析技术应用的补充工具，但学习分析技术及应用不仅限于该类工具。
需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
以上各实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述各实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的范围。

Claims (8)

  1. 一种便携式多模态学习分析智能眼镜，其特征在于，该智能眼镜由镜架模块(1)、智能屏幕模块(2)、眼镜载体模块(3)组成；并利用多模态学习分析方法实时捕捉和分析学习者用户产生的多模态数据，并将可视化分析结果反馈给用户；
    所述的镜架模块(1)包括前仓(4)、移动仓(5)和后仓(6)；所述的前仓(4)外部设置有音频装置(401)和芯片模块(408)，内部设置有陀螺仪模块(407)，音频装置(401)提供语音识别功能，通过音频装置(401)将语言数据实时转化为文本，通过陀螺仪模块(407)来实现头部运动的监测；芯片模块(408)提供数据分析功能，实时读取学生数据进行分析，并将数据存入预设的数据结构中；
    所述的前仓(4)外部还设置有充电接口(402)、开关(403)和无线传输器(404)；内部还设置有微型激光限位开关(405)、微型齿轮(406)、芯片模块(408)、排线(409)、电池(410)和微型齿条(411)，所述微型激光限位开关(405)、微型齿轮(406)、芯片模块(408)、排线(409)、电池(410)和微型齿条(411)依次安装在前仓(4)上；电池(410)通过排线(409)与充电接口(402)、开关(403)连接，并为整个眼镜供电，无线传输器(404)与芯片模块(408)连接；同时，微型激光限位开关(405)与微型齿轮(406)连接，微型齿轮(406)与微型齿条(411)配合，微型齿条(411)与移动仓(5)连接，微型激光限位开关(405)通过扫描移动仓(5)的运动，进而约束移动仓(5)的前后位置极限；
    所述的移动仓(5)设置有垂直向下底座摄像头(501)、滚轴(502)、柔性屏幕(503)、出口槽(504)和柔性屏导轨(505)；其中，通过垂向下底座摄像头(501)实现监测眼球运动和面部表情功能；滚轴(502)与柔性屏幕(503)转动连接，柔性屏幕(503)通过柔性屏导轨(505)与镜架模块(1)连接，柔性屏幕(503)可延柔性屏导轨(505)从镜架模块(1)的出口槽(504)处滑出；
    智能眼镜将数据读取和分析功能的处理结果可视化，以多种可视化图形的方式通过柔性屏幕(503)反馈给学生，具体为：语音识别和分析结果通过结巴分词算法提取关键词，利用关键词聚类可视化图形方式展示学习者谈话特征；头部运动分析结果可展示学习者当前的抬头、低头或转头动作，并将相关频率反馈给学习者；
    所述的后仓(6)设置在镜架模块(1)外部，后仓(6)上安装有生理指标传感器(602)；通过生理指标传感器(602)来实现对生理指标的监测。
  2. 如权利要求1所述的一种便携式多模态学习分析智能眼镜，其特征在于，所述的镜架模块(1)、智能屏幕模块(2)和眼镜载体模块(3)通过设置在镜架内的线路连接，且前仓 (4)中设置芯片模块(408)以处理实时采集的体征数据。
  3. 如权利要求1所述的一种便携式多模态学习分析智能眼镜，其特征在于，所述的后仓(6)还设置有夹具(601)、无线耳机(603)、耳挂(604)和挂环(605)；所述夹具(601)夹在智能眼镜的镜腿上，并通过耳挂(604)和挂环(605)固定在佩戴者的耳朵上，通过无线耳机(603)接收语音信息。
  4. 如权利要求1所述的一种便携式多模态学习分析智能眼镜，其特征在于，所述音频装置(401)将语言数据实时转化为文本是利用自然语言处理技术对语音内容进行分析与编码。
  5. 如权利要求1所述的一种便携式多模态学习分析智能眼镜，其特征在于，柔性屏幕(503)既可用以日常辅助视力，也可及时显示芯片模块(408)处理的结果，实现显示屏的功能。
  6. 如权利要求1所述的一种便携式多模态学习分析智能眼镜，其特征在于，底座摄像头(501)对面部表情进行识别，通过芯片模块(408)分析结果，进而通过文字方式展示学习者当前心理情绪特征；所述面部表情的识别分析结果包括学习者专注、愉悦、困惑不同心理状态；底座摄像头(501)监测到的眼球运动结果展示学习者当前关注的学习对象，并将关注时长、对象、眼球运动次数结果反馈给学习者。
  7. 如权利要求1所述的一种便携式多模态学习分析智能眼镜，其特征在于，所述多模态数据包括表情、语音、生理、眼球及头部运动指标在内的多模态类型数据。
  8. 如权利要求1所述的一种便携式多模态学习分析智能眼镜，其特征在于，将生理指标结果展示给学习者以显示其当前身体特征及状况，所述生理指标包括心率及皮肤电指标。
PCT/CN2021/123942 2020-11-10 2021-10-14 一种便携式多模态学习分析智能眼镜 WO2022100358A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/107,490 US20230296923A1 (en) 2020-11-10 2023-02-08 Portable multimodal learning analytics smart glasses

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011247103.XA CN112419808B (zh) 2020-11-10 2020-11-10 一种便携式多模态学习分析智能眼镜
CN202011247103.X 2020-11-10

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/107,490 Continuation US20230296923A1 (en) 2020-11-10 2023-02-08 Portable multimodal learning analytics smart glasses

Publications (1)

Publication Number Publication Date
WO2022100358A1 true WO2022100358A1 (zh) 2022-05-19

Family

ID=74781030

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/123942 WO2022100358A1 (zh) 2020-11-10 2021-10-14 一种便携式多模态学习分析智能眼镜

Country Status (3)

Country Link
US (1) US20230296923A1 (zh)
CN (1) CN112419808B (zh)
WO (1) WO2022100358A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112419808B (zh) * 2020-11-10 2021-11-02 浙江大学 一种便携式多模态学习分析智能眼镜

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105954878A (zh) * 2016-07-19 2016-09-21 苏州市景荣科技有限公司 一种多功能智能眼镜
WO2017061753A1 (ko) * 2015-10-05 2017-04-13 주식회사 이르테크 텍스트 시각화와 학습자 말뭉치를 이용한 언어학습 시스템
CN108594471A (zh) * 2018-04-28 2018-09-28 中国计量大学 一种基于智能眼镜的学习监督系统及方法
CN110251146A (zh) * 2019-05-31 2019-09-20 郑州外思创造力文化传播有限公司 一种自主学习辅助装置
CN112419808A (zh) * 2020-11-10 2021-02-26 浙江大学 一种便携式多模态学习分析智能眼镜

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3005280B1 (en) * 2013-05-30 2019-05-08 Atlas Wearables, Inc. Portable computing device and analyses of personal data captured therefrom
CN104730731B (zh) * 2013-12-18 2019-03-15 昆山工研院新型平板显示技术中心有限公司 眼镜
KR102349452B1 (ko) * 2015-03-05 2022-01-12 삼성전자주식회사 사용자 인증 방법 및 이를 지원하는 머리 착용형 장치
CN105528577B (zh) * 2015-12-04 2019-02-12 深圳大学 基于智能眼镜的识别方法
CN109902904B (zh) * 2017-12-11 2021-06-29 清华大学 创新性能力分析系统及方法
CN207380922U (zh) * 2018-02-01 2018-05-18 哈尔滨理工大学 一种便于随时复习日语单词的眼镜框
CN108319039A (zh) * 2018-03-26 2018-07-24 南安初盼商贸有限公司 一种具有收音机功能的智能穿戴设备
CN109407858A (zh) * 2018-09-29 2019-03-01 深圳前海格物致知科技有限公司 一种智能眼镜
CN110807471B (zh) * 2019-10-12 2024-02-02 深圳大学 一种多模态传感器的行为识别系统及识别方法
CN111709640A (zh) * 2020-06-15 2020-09-25 浙江大学 一种针对教育场景的特征均衡智能分组方法
CN111736364A (zh) * 2020-07-24 2020-10-02 东莞新溢眼镜制造有限公司 一种智能眼镜框

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017061753A1 (ko) * 2015-10-05 2017-04-13 주식회사 이르테크 텍스트 시각화와 학습자 말뭉치를 이용한 언어학습 시스템
CN105954878A (zh) * 2016-07-19 2016-09-21 苏州市景荣科技有限公司 一种多功能智能眼镜
CN108594471A (zh) * 2018-04-28 2018-09-28 中国计量大学 一种基于智能眼镜的学习监督系统及方法
CN110251146A (zh) * 2019-05-31 2019-09-20 郑州外思创造力文化传播有限公司 一种自主学习辅助装置
CN112419808A (zh) * 2020-11-10 2021-02-26 浙江大学 一种便携式多模态学习分析智能眼镜

Also Published As

Publication number Publication date
CN112419808A (zh) 2021-02-26
CN112419808B (zh) 2021-11-02
US20230296923A1 (en) 2023-09-21

Similar Documents

Publication Publication Date Title
CN110070944B (zh) 基于虚拟环境和虚拟角色的社会功能评估训练系统
US8937650B2 (en) Systems and methods for performing a triggered action
US9910298B1 (en) Systems and methods for a computerized temple for use with eyewear
CN109155837A (zh) 一种情绪感知的可穿戴的电话会议系统
US20180301061A1 (en) Gesture recognition and communication
CN103853071B (zh) 基于生物信号的人机面部表情交互系统
CN100418498C (zh) 导盲器
CN105528577A (zh) 基于智能眼镜的识别方法
WO2022100358A1 (zh) 一种便携式多模态学习分析智能眼镜
CN208255530U (zh) 智能颈戴设备
CN106444084A (zh) 一种新型智能眼镜
CN109426653A (zh) 心理咨询机器人
CN107113390A (zh) 控制装置
CN108960023A (zh) 一种便携式情绪识别装置
CN112002186A (zh) 一种基于增强现实技术的信息无障碍系统及方法
CN211633236U (zh) 一种vr设备
CN210166754U (zh) 一种虚拟现实头戴交流装置及虚拟现实头戴交流系统
CN213423727U (zh) 一种基于tgam的智能家居控制装置
CN113995411A (zh) 一种小型便携式多模态鉴赏评估系统及方法
CN210606227U (zh) 一种增强现实头戴交流装置及增强现实头戴交流系统
CN215841519U (zh) 基于健身手环的健身镜接收模块
Elsherbini et al. Towards A Novel Prototype for Superpower Glass for Autistic Kids
CN214804654U (zh) 基于ai智能的静脉可视便携式穿刺移动设备
CN213183151U (zh) 一种智能演讲训练装置
CN212724719U (zh) 一种听证室语音识别系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21890885

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21890885

Country of ref document: EP

Kind code of ref document: A1