CN116149470A - Practical training method and device for depth fusion of XR virtual simulation and vocational education - Google Patents
Practical training method and device for depth fusion of XR virtual simulation and vocational education Download PDFInfo
- Publication number
- CN116149470A CN116149470A CN202211622267.5A CN202211622267A CN116149470A CN 116149470 A CN116149470 A CN 116149470A CN 202211622267 A CN202211622267 A CN 202211622267A CN 116149470 A CN116149470 A CN 116149470A
- Authority
- CN
- China
- Prior art keywords
- information
- area
- current
- determining
- dynamic image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 31
- 238000012549 training Methods 0.000 title claims abstract description 27
- 230000001755 vocal effect Effects 0.000 title claims abstract description 26
- 230000004927 fusion Effects 0.000 title claims abstract description 18
- 230000008859 change Effects 0.000 claims description 17
- 210000001061 forehead Anatomy 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 11
- 230000003068 static effect Effects 0.000 claims description 8
- 230000005855 radiation Effects 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 3
- 230000010354 integration Effects 0.000 abstract 1
- 230000008569 process Effects 0.000 description 4
- 206010012218 Delirium Diseases 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000001815 facial effect Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000012854 evaluation process Methods 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/20—Education
- G06Q50/205—Education administration or guidance
- G06Q50/2057—Career enhancement or continuing education service
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Educational Technology (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Educational Administration (AREA)
- Strategic Management (AREA)
- General Engineering & Computer Science (AREA)
- Tourism & Hospitality (AREA)
- Oral & Maxillofacial Surgery (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Human Resources & Organizations (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Economics (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- General Business, Economics & Management (AREA)
- Electrically Operated Instructional Devices (AREA)
Abstract
The application relates to an XR virtual simulation and vocational education depth fusion training method, which comprises the following steps: acquiring courseware information, wherein the courseware information comprises video information and voice information; acquiring the current region information corresponding to the dynamic image information in the video information; acquiring face information of a listener, analyzing an area observed by the listener based on the face information, and acquiring observation area information; determining an explanation area based on the voice information, wherein the explanation area comprises explanation contents in the dynamic image information corresponding to the current voice information; determining the distance between the observation area information and the explanation area of a listener, and determining the mutual inclusion relation between the current area information corresponding to the observation area information and the dynamic image information; determining the attention degree of a listener based on the distance, the mutual inclusion relationship and a preset attention degree rule; simultaneously discloses real device, electronic equipment and storage medium of instructing of virtual emulation of XR and depth of vocational education integration, this application has the effect of being convenient for realize remote teaching and knowing student's study condition.
Description
Technical Field
The application relates to the technical field of practical training of XR virtual simulation and vocational education depth fusion, in particular to a practical training method and device of XR virtual simulation and vocational education depth fusion.
Background
The traditional classroom teaching mainly adopts a mode of combining the traditional classroom teaching with the network remote education, and has the advantages that in the teaching process, a teacher can interact with students in a language, gesture, eye contact and other modes to answer the questions of the students in time, and meanwhile, discussion, communication and answering can be carried out in the same environment, and the students cooperate to learn without problems, but the classroom teaching has the defects of itself, for example, the students are expanded and recruited with the further expansion of schools, the number of the students is increased at a high speed, but the schools and teacher resources are relatively tense, so that the education quality is reduced; in addition, classroom teaching is greatly influenced by time and space, such as teacher business trip, student illness and the like, and can bring great influence to learning. The network remote education brought by the information technology can well make up the deficiency of classroom teaching, so that learning is not limited by space and time, resources can be shared, and the problem of shortage of educational resources is solved, but the network education also has some defects, such as difficulty in creating a vivid teaching scene, difficulty in real-time interaction between a teacher and students and difficulty in establishing collaborative support learning between learning peers and the like.
Therefore, how to make up the defects of the traditional classroom teaching and the network remote education and know the learning condition of students in the remote teaching process is a problem to be solved urgently.
Disclosure of Invention
In order to solve the problems that remote interaction cannot be realized and learning conditions cannot be known during remote teaching, the application provides an XR virtual simulation and vocational education deep fusion training method, device, electronic equipment and storage medium.
In a first aspect, the present application provides an XR virtual simulation and vocational education deep fusion training method, which adopts the following technical scheme:
an XR virtual simulation and vocational education depth fusion training method comprises the following steps:
acquiring courseware information, wherein the courseware information comprises video information and voice information;
acquiring current dynamic image information in the video information and a current area corresponding to the current dynamic image information;
collecting face information of a listener, analyzing the face information to obtain an area observed by the listener, and defining the area as an observation area;
determining an explanation area based on the voice information, wherein the explanation area comprises an area corresponding to explanation content in the current voice information in the dynamic image information;
determining the distance between the observation area of the audience and the explanation area, and determining the mutual inclusion relation between the observation area and the current area corresponding to the dynamic image information;
and determining the attention degree of the audience based on the distance, the mutual inclusion relationship and a preset attention degree rule.
By adopting the technical scheme, the face information of the audience is acquired, the observation view angle of the audience can be known through the face information, the actual observation area range of the audience can be determined, and then the areas corresponding to the voice information and the dynamic image information are analyzed to form three specific areas; after the preliminary analysis, the voice information brings the lecturer to explain a certain position in the dynamic image information, and the attention of the audience is further judged by judging the distance between the explanation area and the observation area, so that the lecturer can know the learning condition of the student conveniently.
Preferably, the obtaining the current moving image information in the video information and the current area corresponding to the current moving image information includes:
acquiring current pattern information and historical pattern information before a preset interval time in video information, wherein the pattern information comprises static image information and dynamic image information;
comparing the current pattern information with the historical pattern information, judging the change information of the current pattern information relative to the historical pattern information, and determining the current dynamic image information in the current pattern information;
the change information includes position change information or pattern change information in the pattern information;
and determining an area covered by the current dynamic image information, and defining the area as the current area corresponding to the dynamic image information.
Preferably, the acquiring the face information of the listener, and analyzing the region observed by the listener based on the face information to acquire the observation region information; comprising the following steps:
collecting face information of a listener, and extracting feature information in the face information, wherein the feature information comprises cheek feature information, forehead feature information and eye position information;
acquiring face section information of a listener according to the cheek characteristic information and the forehead characteristic information;
based on the face section information, the eye position information and the preset eye radiation range, an observation area of a listener is obtained.
Preferably, the determining, based on the voice information, an explanation area in the moving image information corresponding to an explanation content in the current voice information includes:
determining a plurality of characteristic image areas in the dynamic image information according to a preset rule;
establishing a corresponding relation between each characteristic image area and name information;
analyzing the voice information to obtain text information corresponding to the voice information;
and matching the text information with the name information, and acquiring an explanation area corresponding to the explanation content in the current voice information according to the corresponding relation between the characteristic image area and the name information.
Preferably, determining the distance between the audience's viewing area and the interpretation area includes:
determining edge information of an observation area of a listener and forming edge area lines;
determining a current interpretation area corresponding to the current language information in the plurality of interpretation areas based on the time point;
determining the shortest distance between an edge area line and the current explanation area; and defines this shortest distance as the distance of the audience's viewing area from the interpretation area.
Preferably, a mutual inclusion relation between the observation area information and the current area information corresponding to the dynamic image information is determined; comprising the following steps:
determining a dynamic area line of the current area information; and matching the edge area line with the dynamic area line, and determining the mutual inclusion relationship according to the mutual relationship between the area formed by the edge area line and the area formed by the dynamic area line.
In a second aspect, the application provides an XR virtual simulation and vocational education depth fusion training device, which adopts the following technical scheme:
practical training device for fusing XR virtual simulation and vocational education depth, which is characterized by comprising:
the first acquisition module is used for acquiring courseware information, wherein the courseware information comprises video information and voice information;
the first processing module is used for acquiring the dynamic image information in the video information and the current area information corresponding to the dynamic image information;
the second acquisition module is used for acquiring the face information of the audience, analyzing the area observed by the audience based on the face information and acquiring the information of the observed area;
the second processing module is used for determining an explanation area based on the voice information, wherein the explanation area comprises explanation contents in the dynamic image information corresponding to the current voice information; and
the matching module is used for determining the distance between the observation area information of the audience and the explanation area and determining the mutual inclusion relation between the observation area information and the current area information corresponding to the dynamic image information; and determining the attention degree of the audience based on the distance, the mutual inclusion relationship and a preset attention degree rule.
Preferably, the face information of the audience is collected, the observation view angle of the audience can be known through the face information, the actual observation area range of the audience can be determined, and then three specific areas are formed by analyzing the areas corresponding to the voice information and the dynamic image information, when the observation area of the audience is overlapped with the current area corresponding to the dynamic image, the attention degree is determined according to the overlapped condition, if the observation area of the audience is not overlapped, the audience is proved to have no class, and when the observation area is positioned in the current area corresponding to the dynamic image, the attention degree of the audience is proved to be very high; after the preliminary analysis, the voice information brings the lecturer to explain a certain position in the dynamic image information, and the attention of the audience is further judged by judging the distance between the explanation area and the observation area, so that the lecturer can know the learning condition of the student conveniently.
In a third aspect, the present application provides an electronic device, which adopts the following technical scheme:
an electronic device, the electronic device comprising:
one or more processors;
a memory;
one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications configured to: a practical training method of XR virtual simulation and depth fusion of vocational education is performed as shown in any one of the possible implementations according to the first aspect.
In a fourth aspect, the present application provides a computer readable storage medium, which adopts the following technical scheme:
a computer readable storage medium, which when executed in a computer, causes the computer to perform a practical training method of fusion of XR virtual simulation and vocational education depth as claimed in any one of the first aspects.
In summary, the present application includes at least one of the following beneficial technical effects:
1. the method comprises the steps of acquiring the face information of a listener, knowing the observation visual angle of the listener through the face information, determining the actual observation area range of the listener, analyzing the areas corresponding to the voice information and the dynamic image information to form three specific areas, determining the attention degree according to the superposition condition when the observation area of the listener is superposed with the current area corresponding to the dynamic image, and proving that the listener does not listen to class if the observation area is not superposed, wherein the attention degree of the listener is proved to be very high when the observation area is positioned in the current area corresponding to the dynamic image; after the preliminary analysis, the voice information brings the lecturer to explain a certain position in the dynamic image information, and the attention of the audience is further judged by judging the distance between the explanation area and the observation area, so that the lecturer can know the learning condition of the student conveniently.
Drawings
FIG. 1 is a schematic flow diagram of a method of an embodiment.
Fig. 2 is a schematic flow chart showing acquisition of a current dynamic image area in the embodiment.
FIG. 3 is a schematic diagram of a practical training device embodying XR virtual simulation and vocational education depth fusion in an embodiment.
Detailed Description
The present application is described in further detail below in conjunction with figures 1-3.
The embodiment of the application provides a practical training method for fusing XR virtual simulation and vocational education depth, which is executed by electronic equipment, wherein the electronic equipment can be a server or terminal equipment, and the server can be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server for providing cloud computing service. The terminal device may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, etc., and the terminal device and the server may be directly or indirectly connected through wired or wireless communication, which is not limited herein, and as shown in fig. 1, the method includes step S101, step S102, step S103, step S104, and step S105, where,
s101, acquiring courseware information, wherein the courseware information comprises video information and voice information;
the practical training method of XR virtual simulation and vocational education depth fusion is applied to course teaching of naked eye 3D or conventional two-dimensional images, courseware information comprises video information, the video information comprises images watched by a tablet or other terminal equipment, and the video information in the application can be used for remotely sharing a screen, recorded videos or live scenes shot by shooting equipment; the voice information corresponds to video information, and is complementary explanation of the content in the video information, for example, when introducing a refrigerator door of a refrigerator device, when an image is adjusted to an image of the refrigerator door, the voice information should also be voice information for explaining the refrigerator door.
S102, acquiring current dynamic image information in video information and a current area corresponding to the current dynamic image information;
the video information is formed by overlapping multiple frames of images, in the teaching process of a lecturer, one frame of image corresponding to a certain moment can be called for analysis in order to clearly know the attention condition of the audience, each frame of image comprises a dynamic image and a static image, the dynamic image information can be extracted from the current frame of image, then the boundary of the dynamic image information is determined by processing the dynamic image information, and the area formed by encircling the boundary is the current area corresponding to the current dynamic image information.
For example, the video information may be a combination of multi-frame pictures including three-dimensional refrigerator equipment, each frame of picture includes a dynamic image and a static image, the dynamic image information refers to an image which changes in a short time interval, the change includes a position change or a shape change of the image, at this time, the refrigerator equipment in the video information can be captured, the refrigerator equipment needs to be turned over during explanation, and then a multi-dimensional explanation is formed, at this time, other images in the image information cannot be changed, the multi-frame images formed by the refrigerator equipment are all dynamic images, and the image formed by the refrigerator equipment in the current frame is the current dynamic image information; and the image forming area of the current refrigerator device is the corresponding current area.
S103, collecting the face information of the audience, analyzing the face information to obtain the area observed by the audience, and defining the area as an observation area;
the face information of the listener comprises the face outline and the information of each characteristic point of the face, such as the forehead, cheek, nose bridge, eyes and the like of the face, the observation area of the listener can be known conveniently by analyzing the face orientation information, for example, the dynamic image information in the video information is a refrigerator device, the position of the refrigerator device in the whole video information is the left lower corner area, the listener should face to the left lower corner area according to the convention, meanwhile, each eye of each person has a corresponding attention area, the area observed by the general eyes has attention area and a blurring area, when one person carefully observes a certain object, the image in a certain area is clear, but other areas are blurred, the area corresponding to the clear image is the observation area, and the information which can be absorbed into the area by the listener is proved.
S104, determining an explanation area based on the voice information, wherein the explanation area comprises an area corresponding to explanation content in the current voice information in the dynamic image information;
in any frame of image of each video information, a plurality of different components are included, for example, the refrigerator equipment comprises a refrigerator main body, a refrigerator door, a refrigerator drawer and other structures, when the lower components cover areas which are all representing different areas, each area is an explanation area, when courseware is played, the video information and the voice information based on time axis points correspond to each other, so that the area currently explained by an instructor can be determined by analyzing the content in the voice information, and the area is defined as the area to be explained.
S105, determining the distance between the observation area and the explanation area of the audience, and determining the mutual inclusion relation between the observation area and the current area corresponding to the dynamic image information; and determining the attention degree of the audience based on the distance, the mutual inclusion relationship and a preset attention degree rule.
The current region corresponding to the dynamic image information is obtained through the step S102, the observation region is obtained through the step S103, and the explanation region is obtained through the step S104; the current area corresponding to the dynamic image information comprises an explanation area, if the audience obtains the observation area and can form superposition with the explanation area, even the audience is located in the explanation area, the audience is proved to carefully hear the explanation knowledge in the explanation area, if the superposition is not present or the audience is not located in the explanation area, whether the observation area is located in the current area corresponding to the dynamic image information or has an overlapped part with the current area corresponding to the dynamic image information is determined, if the audience is located in the current area corresponding to the dynamic image information, the audience is proved to be possibly observing other parts in the whole dynamic image information, and if the audience is proved to be possibly observing other parts in the whole dynamic image information or not observed. The scoring condition may be determined based on a distance of the pair between the interpretation zone and the outer boundary of the observation zone.
Examples: when the observation area is positioned outside the current area corresponding to the dynamic image information, the attention degree is 1 minute; when the overlapping part exists between the observation area and the current area corresponding to the dynamic image information, the attention degree is 2 minutes at the moment; when the observation area is positioned in the current area corresponding to the dynamic image information, the attention degree is 3 minutes at the moment; further, when the observation area is located in the explanation area or there is a superimposed portion, the attention degree is given 3.5 minutes at this time, and if the observation area is not superimposed or located in the explanation area, the attention degree is given according to the distance between the observation area and the explanation area at this time, for example, the observation area and the explanation area may be changed by 0.05 minutes every 1 cm; the score is between 3 minutes and 3.5 minutes, excluding 3 minutes or 3.5 minutes; if the observation area is located outside the current area corresponding to the moving image information or if there is a superposition portion between the observation area and the current area corresponding to the moving image information, the corresponding 1 or 2 points are directly assigned, and the judgment of the distance is not participated.
In one possible implementation manner of the embodiment of the present application, obtaining current dynamic image information in video information and a current area corresponding to the current dynamic image information includes:
s1021, acquiring current pattern information and historical pattern information before a preset interval time in video information, wherein the pattern information comprises static image information and dynamic image information;
for the present embodiment, all frame image information in video information is acquired; respectively calling historical pattern information and current pattern information in corresponding video information according to a front preset interval and a rear preset interval; the history pattern information and the current pattern information each include still image information and moving image information.
S1022, comparing the current pattern information with the historical pattern information, judging the change information of the current pattern information relative to the historical pattern information, and determining the current dynamic image information in the current pattern information; the change information includes position change information or pattern change information in the pattern information;
the current pattern information comprises dynamic image information and static image information; according to the comparison between the historical pattern information and the current pattern information, determining the change condition in the current pattern information, for example, three-dimensional explanation is carried out on the overturning of the refrigerator equipment, at the moment, the refrigerator equipment in the current frame pattern information is changed, other parts are not changed, and meanwhile, if the refrigerator equipment in the pattern information is dragged, the position of the refrigerator equipment is changed, and the current image of the refrigerator equipment can be known to be changed; the application aims at teaching the real-time change of the image, and a lecturer can conduct real-time explanation according to the changed image; at this time, it is determined that the current refrigerator device is current moving image information.
S1023, determining the area covered by the current dynamic image information, and defining the area as the current area corresponding to the dynamic image information.
And determining the boundary of the current dynamic image information, wherein the area in the boundary is the current area.
In one possible implementation manner of the embodiment of the present application, facial information of a viewer is collected, and an area observed by the viewer is analyzed based on the facial information, so as to obtain observation area information; comprising the following steps:
s1031, collecting face information of a listener, and extracting feature information in the face information, wherein the feature information comprises cheek feature information, forehead feature information and eye position information; acquiring face section information of a listener according to cheek characteristic information and forehead characteristic information;
the face information of the audience can be acquired through acquisition, the acquisition device is arranged on the courseware display device, the acquisition device can comprise a camera device and a camera device of a computer, the face information of the audience can be acquired, the face information can be identified after being acquired, the cheek characteristic information, the forehead characteristic information and the eye position information can be identified, and the cheek characteristic information comprises the distance information of the cheek distance acquisition device and the cheek position information; the forehead characteristic information comprises distance information of the forehead distance acquisition device and position information of the forehead;
according to the distance between the forehead and the acquisition device and the distance between the cheeks at two sides and the acquisition device, the plane formed by the cheeks and the forehead can be judged, and then the face section information of the relative acquisition device is determined, and the face section information is used for representing the inclination degree of the face of the listener.
S1032, obtaining the observation area of the audience based on the face section information, the eye position information and the preset eye radiation range.
Firstly, determining the tangent plane information of the face, determining a straight line which passes through the acquisition device and is perpendicular to the face tangent plane information through the face tangent plane information and the acquisition device, and then making another straight line which passes through the eye position information according to the determined eye position information, wherein the two straight lines are parallel to each other; at this time, a connection point of a straight line passing through the eye position information and a display plane of the terminal device is set as a base point, the eye radiation range is preset, and is expanded based on the reference point, and finally, an observation area of the listener is formed.
According to one possible implementation manner of the embodiment of the application, the explanation area corresponding to the explanation content in the current voice information in the dynamic image information is determined based on the voice information; comprising the following steps:
s1041, determining a plurality of characteristic image areas in the dynamic image information according to a preset rule; and establishing a corresponding relation between each characteristic image area and name information;
the dynamic image information generally includes a plurality of areas, each of which is used for representing a feature, for example, the refrigerator device includes a refrigerator body, a refrigerator door and a refrigerator base, wherein the refrigerator body is located in a specific area, and the refrigerator door is also located in a specific area; each area is named as a characteristic image area, and meanwhile, each area has respective unique name information, for example, the image area formed by a refrigerator door corresponds to the name information of a refrigerator door, and a corresponding relation between the characteristic image area and the name information is established.
S1042, analyzing the voice information to obtain text information corresponding to the voice information; and matching the text information with the name information, and acquiring an explanation area corresponding to the explanation content in the current voice information according to the corresponding relation between the characteristic image area and the name information.
Forming text information corresponding to the voice information by recording the voice information; the text information and the voice information have consistency; then inquiring the determined name information in the text information; and then according to the corresponding relation between the name information and the characteristic image area, the explanation area corresponding to the explanation content in the current voice can be known.
In one possible implementation manner of the embodiment of the present application, determining a distance between an observation area and an explanation area of a viewer includes:
s1051, determining edge information of an observation area of an observer, and forming edge area lines;
s1052, determining a current interpretation area corresponding to current language information in the plurality of interpretation areas based on the time point;
s1053, determining the shortest distance between the edge area line and the current explanation area; and defines the shortest distance as the distance between the observation area and the interpretation area of the observer.
The observation area and the current explanation area are provided with edge lines, and the shortest distance between the edge lines and the current explanation area can be determined through image processing.
In one possible implementation manner of the embodiment of the present application, a mutual inclusion relationship between current region information corresponding to observation region information and dynamic image information is determined; comprising the following steps:
s1054, determining a dynamic region line of the current region information; and matching the edge area line with the dynamic area line, and determining the mutual inclusion relationship according to the mutual relationship between the area formed by the edge area line and the dynamic area line forming area.
The current area corresponding to the dynamic image information comprises an explanation area, if the audience obtains the observation area and can form superposition with the explanation area, even the audience is located in the explanation area, the audience is proved to carefully hear the explanation knowledge in the explanation area, if the superposition is not present or the audience is not located in the explanation area, whether the observation area is located in the current area corresponding to the dynamic image information or has an overlapped part with the current area corresponding to the dynamic image information is determined, if the audience is located in the current area corresponding to the dynamic image information, the audience is proved to be possibly observing other parts in the whole dynamic image information, and if the audience is proved to be possibly observing other parts in the whole dynamic image information or not observed.
The embodiment of the application provides a real device of instructing of virtual emulation of XR and depth fusion of vocational education, include:
the first acquisition module is used for acquiring courseware information, wherein the courseware information comprises video information and voice information;
the first processing module is used for acquiring dynamic image information in the video information and current area information corresponding to the dynamic image information;
the second acquisition module is used for acquiring the face information of the audience, analyzing the area observed by the audience based on the face information and acquiring the information of the observed area;
the second processing module is used for determining an explanation area based on the voice information, wherein the explanation area comprises explanation contents in the dynamic image information corresponding to the current voice information; and
the matching module is used for determining the distance between the observation area information and the explanation area of the audience and determining the mutual inclusion relation between the current area information corresponding to the observation area information and the dynamic image information; and determining the attention degree of the audience based on the distance, the mutual inclusion relationship and a preset attention degree rule.
In this embodiment, courseware information is obtained through a first acquisition module, the courseware information is analyzed through a first processing module to obtain dynamic image information, the dynamic image information refers to a changed part in an image, then face information of a listener is acquired through a second acquisition module, the observation view angle of the listener can be known through the face information, the actual observation area range of the listener can be further determined, and then the voice information and an area corresponding to the dynamic image information are analyzed through a second processing module to form three specific areas, when the observation area of the listener is overlapped with the current area corresponding to the dynamic image, the attention degree of the listener is determined according to the overlapped condition, if the observation area of the listener is not overlapped, the listener is proved to have no class, and when the observation area is positioned in the current area corresponding to the dynamic image, the attention degree of the listener is proved to be very high; after the preliminary analysis, the voice information brings the lecturer to explain a certain position in the dynamic image information, and the attention of the audience is further judged by judging the distance between the explanation area and the observation area, so that the lecturer can know the learning condition of the student conveniently.
In an embodiment of the present application, as shown in fig. 3, an electronic device 30 shown in fig. 3 includes: a processor 301 and a memory 303. Wherein the processor 301 is coupled to the memory 303, such as via a bus 302. Optionally, the electronic device 30 may also include a transceiver 304. It should be noted that, in practical applications, the transceiver 304 is not limited to one, and the structure of the electronic device 30 is not limited to the embodiment of the present application.
The processor 301 may be a CPU (central processing unit), general purpose processor, DSP (digital signal processor), ASIC (application specific integrated circuit), FPGA (field programmable gate array) or other programmable logic device, transistor logic device, hardware component, or any combination thereof. Which may implement or perform the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. Processor 301 may also be a combination that implements computing functionality, e.g., comprising one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
Bus 302 may include a path to transfer information between the components. The bus 302 may be a PCI (peripheral component interconnect) bus or an EISA (extended industrial standard architecture) bus, or the like. Bus 302 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in fig. 3, but not only one bus or type of bus.
The memory 303 may be, but is not limited to, a ROM (read only memory) or other type of static storage device that can store static information and instructions, a RAM (random access memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (electrically erasable programmable read only memory), a CD-ROM (compact disc read only memory) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
The memory 303 is used for storing application program codes for executing the present application and is controlled to be executed by the processor 301. The processor 301 is configured to execute the application code stored in the memory 303 to implement what is shown in the foregoing method embodiments.
Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. But may also be a server or the like. The electronic device shown in fig. 3 is only an example and should not be construed as limiting the functionality and scope of use of the embodiments herein.
The present application provides a computer readable storage medium having a computer program stored thereon, which when run on a computer, causes the computer to perform the corresponding method embodiments described above. Compared with the related art, in the embodiment of the application, when the instruction of starting the evaluation is detected, the preset audio is output according to the preset sequence, and the aim of outputting test questions to the patient is achieved by playing the preset audio. And simultaneously acquiring video information of the patient, thereby achieving the effect of recording the evaluation process, and recording feedback made by the patient on each preset audio in sequence in the video information, and determining videos to be analyzed corresponding to each preset audio respectively from the video information, wherein the videos to be analyzed are video fragments of the feedback process made by the patient. Determining feedback video and/or feedback audio of each preset audio according to a preset feedback mode corresponding to each preset audio, analyzing the feedback video and/or feedback audio corresponding to each preset audio, thereby obtaining a feedback result corresponding to each preset audio, and determining whether a patient is delirium or not according to the feedback result corresponding to each preset audio. The feedback video and/or the feedback audio which are/is fed back by the patient are obtained through outputting the preset audio, and the feedback result is obtained according to the feedback video and the feedback audio, so that whether delirium of the patient is caused or not is judged, and the manpower resource waste during delirium assessment is reduced.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein. Moreover, at least some of the steps in the flowcharts of the figures may include a plurality of sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order of their execution not necessarily being sequential, but may be performed in turn or alternately with other steps or at least a portion of the other steps or stages.
The foregoing is only a partial embodiment of the present application, and it should be noted that, for a person skilled in the art, several improvements and modifications can be made without departing from the principle of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.
Claims (9)
1. An XR virtual simulation and vocational education depth fusion training method is characterized by comprising the following steps of:
acquiring courseware information, wherein the courseware information comprises video information and voice information;
acquiring current dynamic image information in the video information and a current area corresponding to the current dynamic image information;
collecting face information of a listener, analyzing the face information to obtain an area observed by the listener, and defining the area as an observation area;
determining an explanation area based on the voice information, wherein the explanation area comprises an area corresponding to explanation content in the current voice information in the dynamic image information;
determining the distance between the observation area of the audience and the explanation area, and determining the mutual inclusion relation between the observation area and the current area corresponding to the dynamic image information;
and determining the attention degree of the audience based on the distance, the mutual inclusion relationship and a preset attention degree rule.
2. The method for training XR virtual simulation and vocational education depth fusion according to claim 1, wherein the obtaining the current dynamic image information in the video information and the current area corresponding to the current dynamic image information comprises:
acquiring current pattern information and historical pattern information before a preset interval time in video information, wherein the pattern information comprises static image information and dynamic image information;
comparing the current pattern information with the historical pattern information, judging the change information of the current pattern information relative to the historical pattern information, and determining the current dynamic image information in the current pattern information;
the change information includes position change information or pattern change information in the pattern information;
and determining an area covered by the current dynamic image information, and defining the area as the current area corresponding to the dynamic image information.
3. The practical training method of the fusion of XR virtual simulation and vocational education depth according to claim 1, wherein the face information of the audience is collected, and the observed area information is obtained by analyzing the area observed by the audience based on the face information; comprising the following steps:
collecting face information of a listener, and extracting feature information in the face information, wherein the feature information comprises cheek feature information, forehead feature information and eye position information;
acquiring face section information of a listener according to the cheek characteristic information and the forehead characteristic information;
based on the face section information, the eye position information and the preset eye radiation range, an observation area of a listener is obtained.
4. The practical training method for fusing virtual simulation and depth of vocational education according to claim 1, wherein the practical training method comprises the following steps of: the determining, based on the voice information, an explanation area in the moving image information corresponding to an explanation content in the current voice information includes:
determining a plurality of characteristic image areas in the dynamic image information according to a preset rule;
establishing a corresponding relation between each characteristic image area and name information;
analyzing the voice information to obtain text information corresponding to the voice information;
and matching the text information with the name information, and acquiring an explanation area corresponding to the explanation content in the current voice information according to the corresponding relation between the characteristic image area and the name information.
5. The practical training method for fusing virtual simulation and depth of vocational education according to claim 1, wherein the practical training method comprises the following steps of: determining a distance of an observation area of the audience from the interpretation area, comprising:
determining edge information of an observation area of a listener and forming edge area lines;
determining a current interpretation area corresponding to the current language information in the plurality of interpretation areas based on the time point;
determining the shortest distance between an edge area line and the current explanation area; and defines this shortest distance as the distance of the audience's viewing area from the interpretation area.
6. The practical training method for fusing virtual simulation and depth of vocational education according to claim 1, wherein the practical training method comprises the following steps of: determining the mutual inclusion relation of the current region information corresponding to the observation region information and the dynamic image information; comprising the following steps:
determining a dynamic area line of the current area information; and matching the edge area line with the dynamic area line, and determining the mutual inclusion relationship according to the mutual relationship between the area formed by the edge area line and the area formed by the dynamic area line.
7. Practical training device for fusing XR virtual simulation and vocational education depth, which is characterized by comprising:
the first acquisition module is used for acquiring courseware information, wherein the courseware information comprises video information and voice information;
the first processing module is used for acquiring the dynamic image information in the video information and the current area information corresponding to the dynamic image information;
the second acquisition module is used for acquiring the face information of the audience, analyzing the area observed by the audience based on the face information and acquiring the information of the observed area;
the second processing module is used for determining an explanation area based on the voice information, wherein the explanation area comprises explanation contents in the dynamic image information corresponding to the current voice information; and
the matching module is used for determining the distance between the observation area information of the audience and the explanation area and determining the mutual inclusion relation between the observation area information and the current area information corresponding to the dynamic image information; and determining the attention degree of the audience based on the distance, the mutual inclusion relationship and a preset attention degree rule.
8. An electronic device, comprising:
one or more processors;
a memory;
one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications configured to: a practical training method of fusing XR virtual simulation with vocational education depth according to any one of claims 1 to 7.
9. A computer readable storage medium having stored thereon a computer program, which when executed in a computer causes the computer to perform an XR virtual simulation and vocational education depth fusion training method as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211622267.5A CN116149470A (en) | 2022-12-16 | 2022-12-16 | Practical training method and device for depth fusion of XR virtual simulation and vocational education |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211622267.5A CN116149470A (en) | 2022-12-16 | 2022-12-16 | Practical training method and device for depth fusion of XR virtual simulation and vocational education |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116149470A true CN116149470A (en) | 2023-05-23 |
Family
ID=86357393
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211622267.5A Pending CN116149470A (en) | 2022-12-16 | 2022-12-16 | Practical training method and device for depth fusion of XR virtual simulation and vocational education |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116149470A (en) |
-
2022
- 2022-12-16 CN CN202211622267.5A patent/CN116149470A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Sannikov et al. | Interactive educational content based on augmented reality and 3D visualization | |
| Hall | Videorecording as theory | |
| Reyna | The potential of 360-degree videos for teaching, learning and research | |
| CN111242962A (en) | Method, device and equipment for generating remote training video and storage medium | |
| NL2026359B1 (en) | Method for multi-channel fusion and presentation of virtual learning environment oriented to field practice teaching | |
| WO2019019403A1 (en) | Interactive situational teaching system for use in k12 stage | |
| CN111144356A (en) | Teacher sight following method and device for remote teaching | |
| Omlor et al. | Comparison of immersive and non-immersive virtual reality videos as substitute for in-hospital teaching during coronavirus lockdown: a survey with graduate medical students in Germany | |
| CN203588489U (en) | A situational teaching device | |
| CN111325853B (en) | Remote coaching system and method based on augmented reality glasses | |
| KR101757420B1 (en) | The system for remote video communication and lecture based on interaction using transparent display | |
| Rummukainen et al. | Audio quality evaluation in virtual reality: multiple stimulus ranking with behavior tracking | |
| Güler et al. | Stereoscopic 3D teaching material usability analysis for interactive boards | |
| Kuester et al. | A virtualized laboratory for earthquake engineering education | |
| CN109076251B (en) | Teleconferencing transmission | |
| CN116149470A (en) | Practical training method and device for depth fusion of XR virtual simulation and vocational education | |
| CN114095747B (en) | Live broadcast interaction system and method | |
| Shibata et al. | Utilization of stereoscopic 3D images in elementary school social studies classes | |
| CN112712738B (en) | Student display processing method and device and electronic device | |
| JP2016114684A (en) | Degree-of-comprehension estimation system, degree-of-comprehension estimation method, and degree-of-comprehension estimation program | |
| Miyai et al. | New approach to camerawork skills education for 3D/S3D computer graphics animation | |
| Jin | A Constructivism-based 3D Scene Display Teaching System for Wushu Teaching. | |
| WO2024001560A1 (en) | Augmented reality-based drawing teaching method and system, display terminal, and storage medium | |
| US20220343783A1 (en) | Content control system, content control method, and content control program | |
| Qian et al. | A systematic scoping review of 360-degree videos in teacher education |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |