WO2015000286A1

WO2015000286A1 - Three-dimensional interactive learning system and method based on augmented reality

Info

Publication number: WO2015000286A1
Application number: PCT/CN2014/000642
Authority: WO
Inventors: 熊剑明
Original assignee: 央数文化(上海)股份有限公司
Priority date: 2013-07-03
Filing date: 2014-07-04
Publication date: 2015-01-08
Also published as: CN103366610B; CN103366610A

Abstract

Provided are a three-dimensional interactive learning system and method based on augmented reality. The system comprises an information processing device, a photographing device, a display device, and at least one physical teaching tool. The physical teaching tool is provided with identification information. The photographing device is used to capture a video of the real environment after an augmented reality application is activated. The information processing device comprises: an identifying module, used to identify the identification information on the physical teaching tool; an orientation calculation module, used to calculate spatial orientation information of the physical teaching tool; and a three-dimensional rendering module, used to acquire a three-dimensional model corresponding to the identification information, render the model, generate a corresponding virtual object, place the virtual object in a corresponding position of a video image according to the spatial orientation information of the physical teaching tool, and display the virtual object. In this way, the present invention can superpose the real environment and the virtual object in a same scenario in real time, provide a more vivid sense experience for the user, and improve a teaching effect by using the human instinct of cognizing the three-dimensional space.

Description

3D interactive learning system and method based on augmented reality

[0001] The present invention relates to the field of multimedia interactive teaching devices, and in particular, to a three-dimensional interactive learning system and method based on augmented reality. Background technique

[0002] At present, the more mainstream multimedia teaching devices on the market include early education machines, dot reading machines, point reading pens, e-books/book bags, literacy cards, and computer education software. Existing multimedia teaching devices have the following drawbacks:

[0003] 1. The traditional multimedia teaching device has low interest and it is difficult to mobilize the initiative of children.

Say

[0004] 2. The multimedia teaching device of the high-tech education class is relatively complicated in operation and has a high user threshold.

[0005] 3. There is no good multimedia teaching device that can drive family interaction.

[0006] With the continuous advancement of the computer age, the application of texts and pictures in learning has become more and more important, and even children aged 3 to 6 have begun to contact a large amount of text and image materials. The past operation of the book is through the mouse and keyboard. Although the touch-type tablet changes the interaction between the keyboard and the mouse, it still stays on the two-dimensional plane interaction.

[0007] AR (Augmented Reality) is one of the hotspots of scientific research in recent years. Augmented reality is also called mixed reality. It applies virtual information to the real world through computer technology. The real environment and virtual objects are superimposed on the same picture or space in real time. Augmented reality provides information that is different from human perception in general. It not only displays the information of the real world, but also displays the virtual information at the same time, and the two kinds of information complement each other and superimpose. In visual augmented reality, users can use the helmet display to combine the real world with computer graphics to see the real world around it. Augmented reality uses computer graphics technology and visualization technology to generate virtual objects that do not exist in the real environment, and accurately places the virtual objects in the real environment through the sensing technology, and integrates the virtual objects with the real environment by means of the display device, and presents them to the real environment. The user has a new environment with realistic sensory effects. Therefore, the augmented reality system has new features of virtual and real integration, real-time interaction, and three-dimensional registration. Augmented reality uses a computer to generate a virtual environment with realistic perceptions of sight, sound, force, touch, and motion. The user is immersed in the environment through various sensing devices to achieve direct natural interaction between the user and the environment. However, there is no precedent for introducing augmented reality technology into multimedia teaching.

[0008] In summary, the prior art obviously has inconveniences and defects in practical use, so it is necessary to improve. Summary of the invention

[0009] In view of the above drawbacks, an object of the present invention is to provide a three-dimensional interactive learning system and method based on augmented reality, which can superimpose a real environment and a virtual object into a same scene in real time, thereby providing a more vivid user. The sensory experience and the use of human instinct for three-dimensional space enhance the teaching effect.

[0010] In order to achieve the above object, the present invention provides a three-dimensional interactive learning system based on augmented reality, including an information processing device, a camera device, a display device, and at least one entity teaching aid;

[001 1] each of the entity teaching aids is provided with an identification information, and each of the identification information corresponds to a three-dimensional model;

[0012] the camera device is configured to perform video collection on a real environment after the augmented reality application is started;

[0013] the display device is configured to display a video image of the real environment; Instruction manual

[0014] The information processing apparatus further includes:

[0015] an identification module, configured to: when the user moves the physical teaching aid to a shooting range of the camera device, identify the identification information on the physical teaching aid, and analyze the corresponding information corresponding to the identification information Three-dimensional model

[0016] an orientation calculation module, configured to calculate spatial orientation information of the physical teaching aid;

[0017] a three-dimensional rendering module, configured to acquire the three-dimensional model corresponding to the identification information, and render the three-dimensional model to generate a corresponding virtual object, and according to the spatial orientation information of the physical teaching aid, A virtual object is placed at a corresponding location in the video image for display.

[0018] According to the three-dimensional interactive learning system of the present invention, the orientation calculation module is configured to perform moving object tracking on the physical teaching aid when the user moves the physical teaching aid in a real environment, and calculate the physical teaching aid in real time. Current spatial orientation information;

[0019] The three-dimensional rendering module is configured to control, according to the current spatial orientation information, the virtual object to perform synchronous display in a corresponding position in the video image.

[0020] According to the three-dimensional interactive learning system of the present invention, after the augmented reality application is started, various multimedia resources are used according to specific application logic.

[0021] According to the three-dimensional interactive learning system of the present invention, the augmented reality application employs multi-thread programming, including a background thread for running a video capture operation and a main thread for running other functional operations.

[0022] According to the three-dimensional interactive learning system of the present invention, the orientation calculation module is configured to calculate two-dimensional spatial orientation information of the physical teaching aid in a physical teaching aid space coordinate system; and according to calibration parameters of the camera device Converting the two-dimensional spatial orientation information into three-dimensional spatial orientation information of the space coordinate system of the camera;

[0023] The three-dimensional rendering module is configured to display the virtual object in a corresponding position in the video image according to the three-dimensional spatial orientation information.

[0024] According to the three-dimensional interactive learning system of the present invention, the three-dimensional rendering module is based on a real-time rendering frame rate, and each time an M frame is rendered, the camera device acquires an image, by the identification module and the The orientation calculation module processes the image; from the 0th frame to the first period of the N*M frame, the orientation calculation module obtains the spatial orientation information of the physical teaching aid N times; the three-dimensional rendering module is The first period does not activate any of the virtual objects; from the NM frame to the second period of the 2N*M frame, the three-dimensional rendering module constructs an N-time Bezier curve by using the results of the first N times of visual capture to estimate Obtaining the spatial orientation information of the physical teaching aid of any one frame and acting on the virtual object.

[0025] According to the three-dimensional interactive learning system of the present invention, the physical teaching aids are teaching cards, teaching books, and teaching dies.

According to the three-dimensional interactive learning system of the present invention, the identification information is a two-dimensional code.

[0027] According to the three-dimensional interactive learning system of the present invention, the three-dimensional interactive learning system prestores at least one template pattern, each of the template patterns corresponding to one of the three-dimensional models; and the identification module is configured to extract the Determining, by the predetermined image matching algorithm, whether the two-dimensional code matches the template pattern by using a predetermined image matching algorithm, and if yes, determining that the matching three-dimensional model corresponding to the template pattern is the physical teaching aid Corresponding said three-dimensional model -

[0028] According to the dimensional interactive learning system, the formula of the image matching algorithm is -

Instruction manual

[0030] wherein 1 is a pixel value of the template pattern, which is a pixel value of the two-dimensional code; when S _; approaches 1, r approaches 1 and the extracted two-dimensional code is The template patterns are compared in N different orientations respectively; or

[0031] The formula of the image matching algorithm is:

Where E _s is the average gray level of the two-dimensional code, and E _t is the average gray level of the template pattern.

[0034] The present invention also provides a three-dimensional interactive learning method based on augmented reality, including steps having -

[0035] setting an identification information on each of the physical teaching aids, each of the identification information corresponding to a three-dimensional model;

[0036] after the augmented reality application is started, video capture is performed on the real environment by the camera device;

[0037] the display device displays a video image of the real environment;

[0038] when the user moves the physical teaching aid to the shooting range of the imaging device, the information processing device identifies the identification information on the physical teaching aid, and analyzes the three-dimensional model corresponding to the identification information. ;

[0039] the information processing device calculates spatial orientation information of the entity teaching aid;

[0040] the information processing device acquires the three-dimensional model corresponding to the identification information, and renders the three-dimensional model to generate a corresponding virtual object, and according to the spatial orientation information of the physical teaching aid, the virtual An object is placed at a corresponding location in the video image for display.

[0041] According to the three-dimensional interactive learning method of the present invention, the step of the information processing apparatus calculating the spatial orientation information of the physical teaching aid further includes:

[0042] when the user moves the entity teaching aid in a real environment, the information processing device performs moving object tracking on the entity teaching aid, and calculates current spatial orientation information of the entity teaching aid in real time;

[0043] the step of placing the virtual object in a corresponding position in the video image according to the spatial orientation information of the physical teaching aid further includes:

And [0044] the information processing device controls the virtual object to perform synchronous display on a corresponding position in the video image according to the current spatial orientation information.

[0045] According to the three-dimensional interactive learning method of the present invention, after the augmented reality application is started, various multimedia resources are used according to specific application logic.

[0046] According to the three-dimensional interactive learning method of the present invention, the augmented reality application employs multi-thread programming, including a background thread for running a video capture operation and a main thread for running other functional operations.

[0047] According to the three-dimensional interactive learning method of the present invention, the step of the information processing device calculating the spatial orientation information of the physical teaching aid further includes:

[0048] the information processing device calculates two-dimensional spatial orientation information of the physical teaching aid in the physical teaching aid space coordinate system; and converts the two-dimensional spatial orientation information into an imaging device according to the calibration parameter of the imaging device Three-dimensional spatial orientation information of the spatial coordinate system;

And the step of displaying the virtual object in a corresponding position in the video image according to the spatial orientation information of the physical teaching aid, further comprising:

[0050] The information processing device displays the virtual object at a corresponding position in the video image according to the three-dimensional spatial orientation information. [0051] According to the three-dimensional interactive learning method of the present invention, the information processing device root is based on a real-time rendering frame rate, and each time an M frame is rendered, the camera device acquires an image, which is processed by the information processing device. Processing the image; from the 0th frame to the first period of the N*M frame, the information processing apparatus obtains the spatial orientation information of the physical teaching aid N times; the information processing apparatus is at the first During the second period from the NM frame to the 2N*M frame, the information processing apparatus constructs the N-time Bezier curve by using the result of the first N times of visual capture, thereby estimating any frame. The spatial orientation information of the physical teaching aid is applied to the virtual object.

[0052] According to the three-dimensional interactive learning method of the present invention, the physical teaching aids are teaching cards, teaching books, and teaching dies.

[0053] According to the three-dimensional interactive learning method of the present invention, the identification information is a two-dimensional code.

[0054] According to the three-dimensional interactive learning method of the present invention, the step of the information processing apparatus identifying the identification information on the entity teaching aid and analyzing the three-dimensional model corresponding to the identification information includes :

[0055] pre-storing at least one template pattern, each of the template patterns corresponding to one of the three-dimensional models;

[0056] the information processing device extracts the two-dimensional code on the entity teaching aid, and identifies whether the two-dimensional code matches the template pattern by a predetermined image matching algorithm;

[0057] If yes, determining that the three-dimensional model of the book corresponding to the matched template pattern is the three-dimensional model corresponding to the entity teaching aid.

[0058] According to the three-dimensional interactive learning method of the present invention, the formula of the image matching algorithm is:

Where T\ is the pixel value of the template pattern, S _; is the pixel value of the two-dimensional code; when approaching 1, r approaches 1 and the extracted two-dimensional code Comparing with the template pattern in N different orientations; or

[0062]

[0064] The present invention provides a three-dimensional interactive learning system and method based on augmented reality in order to solve the problem of low interest and poor teaching effect of the existing multimedia teaching device. When the user starts the augmented reality application, the system passes the camera. The device performs video capture to display the real environment on the display device; at this time, the user can move the physical teaching aid with the identification information to the shooting range of the camera device, and the system identifies the identification information and the spatial orientation information of the physical teaching aid, the entity The teaching aid is preferably a teaching card, and the identification information is preferably a two-dimensional code; then the system acquires the three-dimensional model corresponding to the identification information and renders it into a corresponding virtual object, and places the virtual object on the video image according to the spatial orientation information of the physical teaching aid. The corresponding position in the display is mixed with the real environment. Thereby, the invention introduces the augmented reality technology into the multimedia teaching device, and superimposes the real environment and the virtual object into the same scene in real time, and the two kinds of information complement and superimpose each other, thereby bringing a new experience of the sensory effect to the user, and simultaneously utilizing Human instinct for three-dimensional spatial cognition to improve the user's learning ability and memory ability, thereby improving the teaching effect. More preferably, the user can move the entity teaching aids at will, the system will track the position of the physical teaching aids, and control the virtual objects to be synchronized according to the real space movements. Description

In order to achieve the purpose of free movement of the virtual object controlled by the user, a rich interactive experience is generated, and the effect of teaching and learning is realized. The invention is especially suitable for multimedia interactive teaching of children and adolescents. DRAWINGS

1 is a schematic structural diagram of a three-dimensional interactive learning system based on augmented reality according to the present invention;

2 is a schematic diagram showing a calculation principle of spatial orientation information of a physical teaching aid according to an embodiment of the present invention;

3 is a schematic diagram showing the principle of a contour extraction algorithm in an embodiment of the present invention;

4 is a schematic diagram showing the principle of a quadrilateral detection algorithm in an embodiment of the present invention;

5 is a schematic diagram showing the principle of a quadrilateral coordinate algorithm in an embodiment of the present invention;

6 is a corresponding relationship diagram between a template plane and an imaging plane in the embodiment of the present invention -

7 is a schematic diagram showing the principle of comparing a two-dimensional code and a template pattern in four different orientations in an embodiment of the present invention;

8 is a schematic diagram of a principle of delay rendering in an embodiment of the present invention;

9 is a schematic diagram showing the principle of multi-thread operation of an augmented reality application according to an embodiment of the present invention;

10 is a flowchart of a three-dimensional interactive learning method based on augmented reality according to the present invention;

11 is a flow chart of a preferred augmented reality based three-dimensional interactive learning method of the present invention. Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

1 is a schematic structural diagram of a three-dimensional interactive learning system based on augmented reality according to the present invention. The three-dimensional interactive learning system 100 includes an information processing device 10, a camera device 20, a display device 30, and at least one physical teaching aid 40, wherein:

[0078] Each of the entity teaching aids 40 is provided with an identification information, and each of the identification information corresponds to a three-dimensional model.

[0079] The physical teaching aid 40 is preferably a teaching card, such as a literacy card and an identification card. Of course, the physical teaching aid 40 can also be a teaching book, a teaching mold, and the like. The identification information is preferably a two-dimensional code, and the two-dimensional code can be composed of various characters, patterns, and the like. The physical teaching aid 40 corresponds to at least one vocabulary, object, phenomenon and the like.

[0080] The camera device 20 is configured to perform video collection on a real environment after the augmented reality application is started.

[0081] The imaging device 20 preferably employs a camera. What is more, at the same time, the ordinary camera and the infrared camera are used as the camera device 20, which form complementary advantages and solve the fatal problem that the two-dimensional code input fails when the light in the real environment is weak.

[0082] The augmented reality application refers to various types of application categories of knowledge education, such as nature, ocean, universe, and the like. Each augmented reality application can have specific interactions and game logic. Preferably, after the augmented reality application is started, various multimedia resources are used according to specific application logic, including three-dimensional models, images, sound effects, animations, and other special effects (for example, when the literacy card appears in the video image, the corresponding three-dimensional model is displayed and Trigger smoke effects). Users can learn basic information about each augmented reality application through the application browser, and select and execute the augmented reality application of interest.

[0083] The display device 30 is configured to display a video image of a real environment. The display device 30 can be a screen of a computer, a communication terminal, or a television.

[0084] The information processing device 10 may be a computer, a communication terminal, a television, etc., and the communication terminal may be Description

Mobile phones, PDAs (Personal Digital Assistant, personal digital assistants), tablets, etc. The information processing device 10 further includes -

The identification module 11 is configured to identify the identification information on the physical teaching aid 40 when the user moves the physical teaching aid 40 to the shooting range of the imaging device 20, and analyze the three-dimensional model corresponding to the identification information. The three-dimensional model is pre-existing in the three-dimensional interactive learning system 100.

[0086] The orientation calculation module 12 is configured to calculate spatial orientation information of the physical teaching aid 40. Preferably, the orientation calculation module 12 is configured to calculate the two-dimensional spatial orientation information of the physical teaching aid 40 in the physical teaching aid spatial coordinate system. And according to the calibration parameters of the camera device 20, the two-dimensional spatial orientation information is converted into a three-dimensional spatial orientation of the space coordinate system of the camera device.

[0087] The three-dimensional rendering module 13 is configured to render a virtual three-dimensional model and other virtual environments, preferably for acquiring a three-dimensional model corresponding to the identification information, and rendering the three-dimensional model to generate a corresponding virtual object, and according to the spatial orientation information of the physical teaching aid 40. The virtual object is placed at a corresponding position in the video image, and the virtual object and the real environment are mixed and displayed in the display device 30. Preferably, the three-dimensional rendering module 13 is configured to display the virtual object in a corresponding position in the video image according to the three-dimensional spatial orientation information of the physical teaching aid 40.

[0088] Preferably, the orientation calculation module 12 of the information processing apparatus 10 is configured to perform moving object tracking on the physical teaching aid 40 when the user moves the physical teaching aid 40 in the real environment, and calculate the current spatial orientation of the physical teaching aid 40 in real time. information. The three-dimensional rendering module 13 is configured to control the synchronous display of the virtual object in the corresponding position in the video image according to the current spatial orientation information. In this way, the user can move the entity teaching aid 40 at will, the system 100 will perform position tracking on the physical teaching aid 40, and control the virtual object to be synchronously displayed according to the action of the physical teaching aid 40 in the real space, so as to achieve the purpose of the user to control the free movement of the virtual object. Generate a rich interactive experience.

[0089] The invention is derived from computer graphics recognition control technology and three-dimensional model instant rendering technology.

[0090] First, computer graphics recognition control technology: computer vision technology provides an important application premise for augmented reality systems, camera calibration technology, moving object tracking technology and spatial registration technology of 3D objects for the consistency and real-time interaction of virtual and real space Provided the possibility.

[0091] 1. Using the camera as a video capture device, displaying the real environment on the screen, and being able to identify the identification information such as the two-dimensional code in the real environment, thereby obtaining code value information and spatial position information;

[0092] 2. The pre-created 3D virtual object is placed according to the spatial position information provided by the two-dimensional code, and is mixed and displayed with the real environment;

[0093] 3. The user can control the free movement of the virtual object by moving the two-dimensional code (ie, the physical teaching aid). The user moves the physical teaching aid (such as a literacy card with a two-dimensional code) in real space, and the virtual objects projected on the image captured by the camera on the screen are synchronized according to the real space motion.

[0094] Second, three-dimensional model instant rendering technology:

[0095] 1) real-time rendering of the virtual object, so that it can achieve a realistic image level, so that the virtual object has a high visual consistency with the real environment;

[0096] 2) through the three-dimensional modeling technology of the image, the interaction between the virtual object and the real environment can be made possible;

[0097] 3) Through the geometric intersection and collision detection techniques in graphics, the virtual and real objects in the augmented reality system can interact with each other to produce realistic physical effects.

[0098] In addition, the present invention can provide a more vivid sensory experience for the user by using multimedia forms such as text, sound, pictures, animation, and movies. Instruction manual

[0099] The present invention solves the problem that children are boring in the process of learning vocabulary, objects, phenomena, and the like, and provides a visually sensory experience with a whole new fun. Project virtual objects onto the screen and let illusory objects appear in the real world. At the same time, humans can observe objects, understand and learn various kinds of knowledge through the natural reaction of humans to 3D space.

2 is a schematic diagram showing the calculation principle of the spatial orientation information of the physical teaching aid in the embodiment of the present invention. The physical teaching aid 40 is a quadrilateral literacy card, and the camera device 20 is a camera. Taking the camera space coordinate system (origin 0 _ε , coordinate axis X _e , Y _c , Z _c ) as the reference coordinate system, the literacy card spatial orientation is extracted to calculate the literacy card space coordinate system (origin 0 „, coordinate axis is X „, Y _n , Z _a ) is the representation in the reference coordinate system. The method for extracting the spatial orientation of the literacy card specifically includes - [0101] 1) marking the connected region in the video image. First, the image needs to be binarized. Generally, a fixed threshold binarization is used. For example, a pixel whose luminance value is higher than 100 is set to 1, and the others are set to 0. Then, a progressive scan (from left to right, top to bottom) is performed on the binary image, and pixels having the same pixel value and adjacent pixels are marked with the same number as the connected region.

[0102] 2) For each connected region, extract the contour of the connected region, and verify whether the contour is quadrilateral by a line detection algorithm (because the quadrilateral literacy card should be convex quadrilateral in the video image according to the perspective projection principle) For the quadrilateral region, calculate the two-dimensional coordinates of its four vertices in the image space.

[0103] Contour extraction algorithm: As shown in FIG. 3, for each connected region, the uppermost and leftmost edge points (dark pixels in FIG. b) are first found as the starting point of the contour search. Then search in the possible 8 directions (Fig. a) and find the pixels with the same label as the next contour point. The first search starts from direction 2 (because the starting point is already at the top and left), and when it is turned counterclockwise to direction 4, the pixels with the same label are found as the new contour point. The starting direction of the next search is (this direction +5) modulo 8 , so the second search starts from direction 1 and turns to direction 3 to find a new contour point (Fig. c). Other contour points are derived by analogy until the algorithm returns to the starting point.

[0104] Quadrilateral detection algorithm: As shown in FIG. 4, first traverse the entire contour point sequence, find the point farthest from the starting point (number 0), mark it, and then pair (0, n _t ) and ( _ηι respectively) Two segments of , 0) are processed. At _(0, Πι), the connection from the search point and the farthest point 0 _ηι point, if the distance exceeds a given width value, it indicates that new vertices found, labeled η _2, then continue processing (0 , ) and (η ₂ , η,) segments. In the recursive process, if three vertices are found (including a total of four vertices at the starting point), the outline is proved to be a quadrilateral.

[0105] Quadrilateral coordinate algorithm: using camera calibration parameters, the two-dimensional coordinates of the image space are converted into three-dimensional coordinates of the camera space,

(Xi,y (formula i)

[0107] ( _Xl , _Yl ) is the image pixel coordinate (the origin is in the upper left corner of the image), and ( _X(:, y _c ) is the representation of the image pixel in the camera coordinate system; s^ Sy is the scaling factor, and the unit is Pixels per pixel (pixel/mm), ( _Cl , c _y ) is the pixel coordinate of the camera coordinate system z. The axis coordinates of the intersection of the axis and the imaging plane. In addition, since the distance between the imaging plane and the camera is equal to the focal length f of the camera, imaging The z coordinate of all points on the plane is f. From this, the representation of the four vertices P0~P3 of the quadrilateral in the image in the camera coordinate system can be calculated, as shown in Fig. 5.

[0108] According to the principle of parallel line projection, the representation of the anatomical card coordinate system axis vector (X _ra , Υ „, Z _m ) in the camera coordinate system is calculated. The two sides of the square literacy card parallel to the axis vector are respectively Q And C ₂ C ₃ , the normal of the plane 0 _e C< (the yellow plane A shown above) is 1^, plane 0. The normal of C ₂ C ₃ (purple plane B) is n ₂ , then the axial direction The quantity Χ „= _ηι * _η2 (x represents the vector cross multiplication operation). In addition, since the projections of (:< and C ₂ C ₃ on the imaging plane are respectively P ₂ P ₃ and the coordinates of P. to P ₃ are known,

N2=O _c P ₂ *O _c P ₃ . Similarly, the axis vector Y _{B is obtained} , and the axis vector Z„=X _ra *Y„; Instruction manual

[0109] Calculate the representation of the literacy card center 0 _摄像机 in the camera space coordinate system in the camera space. As shown in FIG. 5, since (X _m , Y _ffl , Ζ _π ) has been obtained, and the knowledge word card size is s, the genre card vertices can be expressed as C. =0 _ra + (-s/2) X„+ (s/2) Y„, will be C. And its projection P on the imaging plane. The coordinate values are substituted into Equation 2 (perspective projection equation) to get the equations. The same method is applied to C ₂ , C ₃ , calculated by solving the situation of the equations.

3⁄4. = i (G _Xf Cy) , (Formula 2)

[0110] The three-dimensional interactive learning system 100 prestores at least one template pattern, each template pattern corresponding to a three-dimensional model. The identification module 11 is configured to extract the two-dimensional code on the physical teaching aid 40, and identify whether the two-dimensional code matches the template pattern by using a predetermined image matching algorithm, and if yes, determine that the three-dimensional model corresponding to the matched template pattern is the three-dimensional corresponding to the physical teaching aid 40. model.

[01 1 1] The literacy card is taken as an example to describe the two-dimensional code identification in detail:

[0112] Each detected quadrilateral image area is compared with a template pattern in the system database to find the best match, and the number of the template pattern is recorded.

[01 13] The template pattern has a uniform resolution, such as 32*32, so before the comparison, the quadrilateral image region needs to be normalized, that is, the rectangular projection is transformed into a square region by using the perspective projection homography matrix transformation. As shown in Fig. 6, for any point (x _t , y _t ) on the plane of the template image, the coordinates ( _Xi , y , ) of the projection point on the imaging plane can be obtained by the homography matrix (Equation 3), and ( _Xl , The pixel value of y _; ) is given (x _t , y _t ). To solve the values of the homography matrix items, substitute the coordinates of the four vertices of the square template plane, such as (0,0), (32, 0), (32, 32), (32, 0), and the previously obtained The vertex coordinates of the quadrilateral on the imaging plane are solved by the column equations.

[01 14] w -

[0115] The principle of the image matching algorithm is as shown in Equation 4, where 1\ is the pixel value of the template pattern, and the pixel value of the two-dimensional code approaches n when approaching η. Since the literacy card can be flipped at will, it is necessary to compare the extracted pattern with the template in four different orientations, as shown in FIG.

^r - ΙνΜ-ΐ β2. I . W (Formula 4)

[0117] Furthermore, in order to mitigate the effect of varying lighting conditions on image matching, the average gray level of the entire image may be first counted and the average gray level subtracted from a single pixel value, as shown in Equation 5.

^[0118 ] W 一 ^ ^ ~ ^{∑i = 0} i ⁽ Equation ⁵⁾

[0119] where E _s is the average gray level of the two-dimensional code, and E _t is the average gray level of the template pattern.

[0120] When the moving speed of the two-dimensional code is too fast or the shooting angle of the camera is too large, the instantaneous failure of the two-dimensional code recognition is likely to occur, thereby causing the display of the virtual object associated with it to be stuck, which seriously affects the user. Experience. In order to solve the problem, the present invention adopts a delayed rendering technique, first backs up a plurality of two-dimensional code recognition results before the current frame, and delays the motion of the virtual object by a corresponding number of frames, thereby reserved for real-time processing of video input. A certain time margin; the instantaneous position of the virtual object does not directly correspond to the spatial position of the two-dimensional code, but is obtained by interpolating the position of the two-dimensional code of the previous frame, thereby greatly improving the smoothness of the movement of the virtual object. [0121] The idea of the deferred rendering technique of the present invention is that the three-dimensional rendering module 13 is based on the real-time rendering frame rate. Each time the M frame is rendered, the camera device 20 captures an image, and the image is performed by the recognition module 11 and the orientation calculation module 12. deal with. From the 0th frame to the first period of the N*M frame, the orientation calculation module 12 obtains the spatial orientation information of the physical teaching aid 40 N times. The three-dimensional rendering module 13 does not activate any virtual objects during the first period. During the second period from the Nth frame to the 2N*M frame, the 3D rendering module 13 constructs an N-time Bezier curve by using the results of the first N times of visual capture, thereby estimating the spatial orientation information of the physical teaching aid 40 of any frame, and Act on virtual objects.

8 is a schematic diagram of the principle of delay rendering in the embodiment of the present invention. The real-time rendering frame rate of the rendering engine is used as a reference. Each time n frames are rendered, the camera captures an image; the visual engine processes the image to extract the literacy card. Orientation. From frame 0 to frame 3n, the vision engine gets the orientation of the literacy card four times; during this time, the rendering engine does not activate any virtual objects, so-called delayed rendering. From the 3nth frame to the 6n frame, the rendering engine constructs a cubic Bezier curve by visually capturing the first 4 results, thereby estimating the orientation of the literacy card of any frame and acting on the virtual object.

Say

[0123] Delayed rendering causes the movement of virtual objects to lag behind real literacy cards. If the frame rate of real-time rendering is 30 frames per second, taking η=3, the delay of 3n frames is equal to the delay of 9/30=0. 3 seconds, so the impact on human-computer interaction is small. As the rendering frame rate increases, the actual latency is further reduced.

[0124] In order to prevent the image acquisition process from taking up too much CPU book time, which in turn causes the application to jam, the augmented reality application of the present invention employs multi-thread programming, including background threads for running video capture operations and for The main thread that runs other functional operations. As shown in Figure 9, the image capture operation is run in the background thread, while the main thread is responsible for running the main logic and rendering operations of the program.

10 is a flowchart of a three-dimensional interactive learning method based on augmented reality according to the present invention, which can be implemented by the three-dimensional interactive learning system 100 shown in FIG. 1, and includes the following steps:

[0126] Step S101: Set an identification information on each of the entity teaching tools 40, and each of the identification information corresponds to a three-dimensional model.

[0127] The physical teaching aid 40 is preferably a teaching card, such as a literacy card and a literacy card. Of course, the physical teaching aid 40 can also be a teaching book, a teaching stencil, or the like. The identification information is preferably a two-dimensional code, and the two-dimensional code can be composed of various characters, patterns, and the like. The physical teaching aid 40 corresponds to at least one vocabulary, object, phenomenon and the like.

[0128] Step S102: After the augmented reality application is started, video capture is performed on the real environment by the camera device 20.

[0129] The imaging device 20 preferably employs a camera. What is more, at the same time, the ordinary camera and the infrared camera are used as the camera device 20, which form complementary advantages and solve the fatal problem that the two-dimensional code input fails when the light in the real environment is weak.

[0130] The augmented reality application refers to various types of application categories of knowledge education, such as nature, ocean, universe, and the like. Each augmented reality application can have specific interactions and game logic. Preferably, after the augmented reality application is started, various multimedia resources are used according to specific application logic, including three-dimensional models, images, sound effects, animations, and other special effects (for example, when the literacy card appears in the video image, the corresponding three-dimensional model is displayed and Trigger smoke effects). Users can learn basic information about each augmented reality application through the application browser, and select and execute the augmented reality application of interest. Preferably, the augmented reality application employs multi-threaded programming, including background threads for running video capture operations and main threads for running other functional operations.

[0131] Step S103, the display device 30 displays the video image of the real environment.

[0132] The display device 30 may be a screen of a computer, a communication terminal, or a television.

[0133] Step S104, when the user moves the physical teaching aid 40 to the shooting range of the camera device 20, the information Description

The processing device 10 identifies the identification information on the physical teaching aid 40 and analyzes the three-dimensional model corresponding to the identification information.

[0134] The information processing device 10 may be a computer, a communication terminal, a television, or the like, and the communication terminal may be a mobile phone, a PDA, a tablet, or the like. The three-dimensional model is pre-existing in the three-dimensional interactive learning system 100.

[0135] Step S105, the information processing apparatus 10 calculates spatial orientation information of the physical teaching aid 40.

[0136] Preferably, in this step, when the user moves the physical teaching aid 40 in the real environment, the information processing apparatus 10 performs moving object tracking on the physical teaching aid 40, and calculates the current space of the physical teaching aid 40 in real time. Bearing information.

[0137] Step S106, the information processing apparatus 10 acquires the three-dimensional model corresponding to the identification information, and renders the three-dimensional model to generate a corresponding virtual object, and according to the spatial orientation of the physical teaching aid 40. The information is displayed by placing the virtual object at a corresponding position in the video image.

[0138] Preferably, in the step, the information processing apparatus 10 controls the virtual object to perform synchronous display on the corresponding position in the video image according to the current spatial orientation information.

[0139] Preferably, the step S104 includes -

[0140] 1) Pre-storing at least one template pattern, each of the template patterns corresponding to one of the three-dimensional models.

[0141] 2) The information processing apparatus 10 extracts the two-dimensional code on the entity teaching aid 40, and identifies whether the two-dimensional code matches the template pattern by a predetermined image matching algorithm.

[0142] 3) If yes, it is determined that the three-dimensional model corresponding to the matched template pattern is the three-dimensional model corresponding to the physical teaching aid 40.

Σΐί^ ':

[0143] The formula of the image matching algorithm is: r =

[0144] where 1 is the pixel value of the template pattern, and S _f is the pixel value of the two-dimensional code. When approaching 1\, r approaches 1 and the extracted two-dimensional code is compared with the template pattern in N different orientations, respectively. or

[0148] Preferably, the step S105 further includes: the information processing device 10 calculates two-dimensional spatial orientation information of the physical teaching aid 40 in the physical teaching aid space coordinate system; and according to the calibration of the imaging device 20 The parameter converts the two-dimensional spatial orientation information into three-dimensional spatial orientation information of the camera coordinate system.

[0149] Preferably, the step S106 further includes: the information processing apparatus 10, according to the three-dimensional spatial orientation information, placing the virtual object in a corresponding position in the video image for display.

[0150] Preferably, the step S106 further includes: the information processing device 10 is based on a real-time rendering frame rate, and each time the M frame is rendered, the camera device 20 collects an image, and the information is processed by the information. The device 10 processes the image. The information processing apparatus 10 obtains the spatial orientation information of the entity teaching aid 40 N times from the 0th frame to the first period of the N*M frame. The information processing apparatus 10 does not activate any of the virtual objects during the first period. From Description

During a second period from the NM frame to the 2N*M frame, the information processing apparatus 10 constructs an N-time Bezier curve using the result of the first N times of visual capture, thereby estimating the physical teaching aid 40 of any one frame. Spatial orientation information and acts on the virtual object.

[0151] The present invention utilizes a computer to generate a virtual environment with realistic perceptions of sight, sound, force, touch, and motion, and immerses the user in the environment through various sensing devices to realize natural interaction between the user and the environment. Through interaction, it is fun to learn, and language learning, image cognition and 3D digital models are organically linked, and the easier it is for users to remember what they have learned. Studies have shown that the present invention can improve the learning efficiency by 10 to 20%, and at the same time greatly improve the brain memory effect.

[0152] FIG. 11 is a flowchart of a three-dimensional interactive learning method based on the augmented reality of the present invention, which can be implemented by the three-dimensional interactive learning system 100 shown in FIG. 1. The information processing apparatus 10 in this embodiment uses a computer and a camera. The device 20 employs a camera, the display device 30 employs a computer screen, the physical teaching aid 40 employs a teaching card, and the teaching card is provided with a two-dimensional code. The method includes the steps of -

[0153] Step S111, the user obtains several AR (Augmented Reality) applications through the Internet. AR applications refer to a variety of application categories for knowledge education, such as nature, oceans, the universe, and so on.

[0154] Step S112, the user starts an AR application.

[0155] Step S113, after the AR application is started, the camera starts to work. That is, the camera is used as a video capture device to perform video capture on the real environment.

[0156] Step S114, displaying the real environment on the computer screen.

[0157] Step S115, the user moves the teaching card in front of the camera, and the teaching card is provided with a two-dimensional code. The user can now move the QR code to the shooting range of the camera and move the position of the QR code at will.

[0158] Step S116, the system automatically tracks and identifies the two-dimensional code of the teaching card.

[0159] Step S117, the system displays the 3D virtual item corresponding to the two-dimensional code on the computer screen.

[0160] Step S118, the user provides the three-dimensional information implied in the real background image through interaction, so that the 3D virtual object can directly interact with the three-dimensional information, thereby greatly improving the interactive entertainment of the system.

[0161] Step S119, the user quits the application.

[0162] Thereby, the user can activate the display of the three-dimensional model and the animation by recognizing the teaching card in front of the camera; and by interacting with the spatial movement of the teaching card and rotating, a rich interactive experience is generated.

In summary, the present invention provides a three-dimensional interactive learning system and method based on augmented reality in order to solve the problem that the existing multimedia teaching device has low interest and poor teaching effect, and when the user starts the augmented reality application. After that, the system performs video capture through the camera device, and displays the real environment on the display device. At this time, the user can move the physical teaching aid with the identification information to the shooting range of the camera device, and the system identifies the identification information and spatial orientation of the physical teaching aid. Information, the entity teaching aid is preferably a teaching card, and the identification information is preferably a two-dimensional code; then the system acquires the three-dimensional model corresponding to the identification information and renders it into a corresponding virtual object, and virtualizes according to the spatial orientation information of the physical teaching aid. Objects are placed at corresponding locations in the video image for display in a mixed environment with the real environment. Thereby, the invention introduces the augmented reality technology into the multimedia teaching device, and superimposes the real environment and the virtual object into the same scene in real time, and the two kinds of information complement and superimpose each other, thereby bringing a new experience of the sensory effect to the user, and simultaneously utilizing Human instinct for three-dimensional spatial cognition to improve the user's learning ability and memory ability, thereby improving the teaching effect. More preferably, the user can move the entity teaching aids at will, the system will track the position of the physical teaching aids, and control the virtual objects to be synchronously displayed according to the actions of the real space, so as to achieve the purpose of free movement of the virtual objects controlled by the user, and generate rich interactions. Experience and realize the effect of entertaining and entertaining. The invention is especially suitable for multimedia interactive teaching of children and adolescents. The invention may, of course, be embodied in a variety of other embodiments, and various modifications and changes can be made in accordance with the present invention without departing from the spirit and scope of the invention. These respective changes and modifications are intended to fall within the scope of the appended claims.

Say

Book

Claims

Claim

A three-dimensional interactive learning system based on augmented reality, comprising: an information processing device, a camera device, a display device, and at least one entity teaching aid;

Each of the entity teaching aids is provided with an identification information, and each of the identification information corresponds to a three-dimensional model; the camera device is configured to perform video collection on a real environment after the augmented reality application is started;

The display device is configured to display a video image of the real environment;

The information processing apparatus further includes:

An identification module, configured to: when the user moves the physical teaching aid to a shooting range of the camera device, identify the identification information on the physical teaching aid, and analyze the three-dimensional model corresponding to the identification information;

An orientation calculation module, configured to calculate spatial orientation information of the physical teaching aid;

a three-dimensional rendering module, configured to acquire the three-dimensional model corresponding to the identification information, and render the three-dimensional model to generate a corresponding virtual object, and place the virtual object according to the spatial orientation information of the physical teaching aid Displayed at a corresponding location in the video image.

The three-dimensional interactive learning system according to claim 1, wherein the orientation calculation module is configured to perform moving object tracking on the physical teaching aid when the user moves the physical teaching aid in a real environment, and calculate the real-time object in real time. Current spatial orientation information of the physical teaching aid;

The three-dimensional rendering module is configured to control, according to the current spatial orientation information, the virtual object to be synchronously displayed at a corresponding position in the video image.

3. The three-dimensional interactive learning system according to claim 2, wherein after the augmented reality application is started, various multimedia resources are used according to specific application logic.

4. The three-dimensional interactive learning system of claim 2, wherein the augmented reality application employs multi-thread programming, including a background thread for running a video capture operation and a main thread for running other functional operations.

The three-dimensional interactive learning system according to claim 2, wherein the orientation calculation module is configured to calculate two-dimensional spatial orientation information of the physical teaching aid in a physical teaching aid space coordinate system; a calibration parameter of the device, the two-dimensional spatial orientation information is converted into three-dimensional spatial orientation information of the camera coordinate system; the three-dimensional rendering module is configured to place the virtual object according to the three-dimensional spatial orientation information The corresponding position in the video image is displayed.

The three-dimensional interactive learning system according to claim 2, wherein the three-dimensional rendering module is based on a real-time rendering frame rate, and each time the M frame is rendered, the camera device collects an image by the identification. The module and the orientation calculation module process the image; the first orientation period from the 0th frame to the N*M frame, the orientation calculation module obtains the spatial orientation information of the physical teaching aid N times; The three-dimensional rendering module does not activate any of the virtual objects during the first period; from the NM frame to the second period of the 2N*M frame, the three-dimensional rendering module constructs N times Bessel by utilizing the results of the first N times of visual capture A curve, thereby estimating the spatial orientation information of the physical teaching aid of any one frame and acting on the virtual object.

7. The three-dimensional interactive learning system according to claim 2, wherein the physical teaching aids are teaching cards, teaching books, and teaching dies.

The three-dimensional interactive learning system according to any one of claims 1 to 7, wherein the identification information is a two-dimensional code.

The three-dimensional interactive learning system according to claim 8, wherein the three-dimensional interactive learning system prestores at least one template pattern, each of the template patterns corresponding to one of the three-dimensional models; Lift Claim

Extracting the two-dimensional code on the physical teaching aid, and identifying whether the two-dimensional code matches the template pattern by a predetermined image matching algorithm, and if yes, determining that the matched three-dimensional model of the template pattern is The three-dimensional model corresponding to the physical teaching aid.

10. The three-dimensional interactive learning system, characterized in that the formula of the image matching algorithm

Where T is the pixel value of the template pattern, Si is the pixel value of the two-dimensional code; when Si approaches Ti, r approaches 1 and the extracted two-dimensional code and the template are The patterns are compared in N different orientations; or

Its E _s is the average gray level of the two-dimensional code,

11. A three-dimensional interactive learning method based on augmented reality, comprising the steps of: setting an identification information on each of the physical teaching aids, each of the identification information corresponding to a three-dimensional model; after the augmented reality application is started, Video capture of the real environment by the camera device;

Display device displays a video image of the real environment;

When the user moves the physical teaching aid to the shooting range of the camera device, the information processing device identifies the identification information on the physical teaching aid, and analyzes the three-dimensional model corresponding to the identification information;

The information processing device calculates spatial orientation information of the physical teaching aid;

Obtaining, by the information processing device, the three-dimensional model corresponding to the identification information, and rendering the three-dimensional model to generate a corresponding virtual object, and placing the virtual object according to the spatial orientation information of the physical teaching aid The corresponding position in the video image is displayed.

12. The three-dimensional interactive learning method according to claim 11, wherein the step of calculating, by the information processing device, the spatial orientation information of the physical teaching aid further comprises: when the user moves the physical teaching aid in a real environment, The information processing device performs moving object tracking on the physical teaching aid, and calculates current spatial orientation information of the physical teaching aid in real time;

The step of placing the virtual object in a corresponding position in the video image according to the spatial orientation information of the physical teaching aid further includes:

The information processing device controls the virtual object to perform synchronous display at a corresponding position in the video image according to the current spatial orientation information.

The three-dimensional interactive learning method according to claim 12, wherein after the augmented reality application is started, various multimedia resources are used according to specific application logic.

14. The three-dimensional interactive learning method of claim 12, wherein the augmented reality application employs multi-thread programming, including a background thread for running a video capture operation and a main thread for running other functional operations.

The method of claim 12, wherein the step of calculating, by the information processing device, the spatial orientation information of the physical teaching aid further comprises:

The information processing device calculates a two-dimensional spatial orientation letter of the entity teaching aid in a physical teaching aid space coordinate system Claim

And converting the two-dimensional spatial orientation information into three-dimensional spatial orientation information of the space coordinate system of the imaging device according to the calibration parameter of the imaging device;

The information processing apparatus displays the virtual object at a corresponding position in the video image according to the three-dimensional spatial orientation information.

The three-dimensional interactive learning method according to claim 12, wherein the information processing device root is based on a real-time rendering frame rate, and each time the M frame is rendered, the camera device collects an image, The information processing apparatus processes the image: the first time period from the 0th frame to the N*Mth frame, the information processing apparatus obtains the spatial orientation information of the physical teaching aid N times; the information processing apparatus is The first period does not activate any of the virtual objects; from the NM frame to the second period of the 2N*M frame, the information processing apparatus constructs an N-time Bezier curve by using the results of the first N times of visual capture to estimate Obtaining the spatial orientation information of the physical teaching aid of any one frame and acting on the virtual object.

The three-dimensional interactive learning method according to claim 12, wherein the physical teaching aid is a teaching card, a teaching book, and a teaching mold.

The three-dimensional interactive learning method according to any one of claims 11 to 17, wherein the identification information is a two-dimensional code.

The three-dimensional interactive learning method according to claim 18, wherein the information processing device identifies the identification information on the entity teaching aid, and analyzes the three-dimensional model corresponding to the identification information. The steps include:

Pre-storing at least one template pattern, each of the template patterns corresponding to one of the three-dimensional models;

The information processing device extracts the two-dimensional code on the entity teaching aid, and identifies whether the two-dimensional code matches the template pattern by a predetermined image matching algorithm;

If yes, it is determined that the three-dimensional model corresponding to the matched template pattern is the three-dimensional model corresponding to the entity teaching aid.

The three-dimensional interactive learning method is characterized in that the image matching algorithm is public

^3⁄4 force

Where 1\ is the pixel value of the template pattern, S is the pixel value of the two-dimensional code; when approaching T _t , r approaches 1 and the extracted two-dimensional code is The template patterns are compared in N different orientations respectively; or