TWI413034B - The System of Mixed Reality Realization and Digital Learning - Google Patents

Abstract

A system of mixed augmented reality and digital learning comprises: a first processor for receiving acceleration values at three directions generated by an accelerometer and touch signals generated by a touch sensor and converting the acceleration values at three directions and the touch signals into packets for transmission; a computer for receiving the acceleration values at three directions and the touch signals transmitted as packets from the first processor, receiving the image data with a physical device generated by a webcam, calculating the acceleration values at three directions to determine the action performed by the physical device, and calculating the position of the physical device according to the image data, wherein the position and action of the physical device are combined, and the image data of a virtual object is added to the image data of the physical device, such that the image of the virtual object is combined with the image of the physical device.

Description

Hybrid augmented reality and digital learning system

The invention relates to a system for mixing and amplifying reality and digital learning, and applying the augmented reality and mixed reality technology to a natural science experiment course of interactive digital teaching.

In the natural sciences experimental course, children are required to take hands-on exercises to learn the natural sciences, but for chemical experiments or equipment that are dangerous and difficult to obtain, children are operationally hazardous or have no equipment to operate. In order to overcome the above problems, a virtual reality is designed to replace the actual operation. For example, a child sees a virtual picture on the screen, and the mouse is displayed on the virtual screen of the screen, and the experiment is displayed on the screen according to the operation of the mouse. The result, in turn, achieves the effect of learning. However, according to the operation on the virtual screen of the screen, only the visual feedback of the user of the child is given, and there is no tactile feedback of the physical object, and the actual feeling of the user during the operation cannot be made.

The present invention provides a system for hybrid augmented reality and digital learning, which is used by children to replace traditional natural science experimental courses, and utilizes Augmented Reality technology to utilize Webcam and Web. The physical device designed by the system uses the operation as a virtual experiment to present the actual experiment, and presents the traditional experimental results with the visual feedback of the augmented reality. It can enjoy the feeling of feedback and have a more real feeling.

One aspect of the present invention provides a system for hybrid augmented reality and digital learning, comprising: a physical device comprising: an accelerometer for sensing an acceleration of the physical device to generate acceleration in three directions a touch sensor for sensing whether the physical device is touched to generate a touch signal, and a first processor for receiving acceleration values of the three directions generated by the accelerometer and the touch The touch signal generated by the sensor, and combining the acceleration values in three directions with the touch signal to transmit in a packet form; a network camera for capturing images to generate image data having the physical device; and a main The control device includes: a computer, configured to receive acceleration values of the three directions transmitted by the first processor in a packet form, and the touch signal, and receive image data of the physical device generated by the network camera, Calculating the acceleration values in three directions to know what action the physical device performs, and calculating the location of the physical device based on the image data, combining the entity Positioning and moving, and adding image data of a virtual object to the image data of the physical device, so that the image of the virtual object is combined with the image of the physical device, and the virtual object performs the same action as the physical device; And a screen for receiving image data transmitted by the computer to display a knot The virtual object and the image of the physical device are combined.

According to the system of the present invention, the physical device further includes a first radio frequency module, and the first radio frequency module wirelessly transmits the acceleration values of the three directions transmitted by the first processor in a packet form and the touch The control unit further includes a second radio frequency module, and the second radio frequency module wirelessly receives the acceleration values of the three directions wirelessly transmitted by the first radio frequency module and the touch signal.

A system according to the aspect of the invention, wherein the accelerometer produces an acceleration value in three directions analogous; the touch sensor produces an analog signal to the touch signal; the first processor will be analogous to the three directions The acceleration value and the touch signal are converted into acceleration values of the three directions of the digit and the touch signal, and the acceleration value of the three directions of the digit and the touch signal are combined into a packet to transmit the computer of the main control device; And the computer receives the acceleration value of the digital direction and the touch signal transmitted by the first processor in a packet form.

The system according to the aspect of the present invention further includes a physical object, the main control device includes a third radio frequency module, and the computer will activate the signal for one of the digits and the acceleration values of the three directions in the form of a packet via the third The radio frequency module wirelessly transmits to the physical object, and the physical object includes: a plurality of servo motors for driving the physical object; a fourth radio frequency device for wirelessly receiving the activation signal and the acceleration values of the three directions wirelessly transmitted by the third RF module; and a processor And receiving the start signal and the acceleration values of the three directions transmitted by the fourth radio frequency device, and calculating acceleration values in three directions to obtain a moving direction value of the digital position, wherein the processor is based on the digital position The activation signal activates the operation of the servo motors, and the physical object is driven to drive the physical object according to the value of the movement direction of the digital position, so that the physical object has a directional movement.

The preferred embodiments of the present invention are described with reference to the following drawings.

1 is a system block diagram of a system for hybrid augmented reality and digital learning according to an embodiment of the present invention, and FIG. 2 is a schematic diagram of a system for operating a hybrid augmented reality and digital learning according to the present invention.

In Figures 1, 2, the system for hybrid augmented reality and digital learning 10 includes a physical device 12, a network camera 14, and a master device 16. The physical device 12 includes an accelerometer 20, a touch sensor 22, a processor 24, and a radio frequency (RF) module 26. The main control device 16 includes a computer 28, a radio frequency module 30, and a screen 32.

The touch sensor 22 is used to sense whether the physical device 12 is touched. When the operator touches the physical device 12, the touch sensor 22 is generated as an analogy. A touch signal. The accelerometer 20 is used to sense the acceleration of the physical device 12. When the operator moves, taps, flips or throws away the physical device 12, the accelerometer 20 generates three directions in the analogy according to the action of the physical device 12 (ie, three-dimensional space). The acceleration value.

The processor 24 receives the acceleration values in the three directions analogous to the accelerometer 20 and the analog signals generated by the touch sensor 22, and converts the acceleration values and touch signals in the three directions of the analogy into The acceleration values and the touch signals of the three directions of the digits are combined into the packet form of the acceleration signals in the three directions of the digits to be transmitted to the RF module 26.

The RF module 26 wirelessly transmits the acceleration values and the touch signals in three directions of the digital position, and the RF module 30 of the main control device 16 wirelessly receives the acceleration values and the touch signals in three directions of the digital position, and The acceleration values and the touch signals received in three directions of digits are transmitted to the computer 28.

The webcam 14 is used to capture images to produce digital or analog image data having physical devices 12, wherein the physical device 12 is provided with a pattern 42 (see FIG. 2) for identification.

The computer 28 is configured to receive the acceleration values and the touch signals transmitted by the RF module 30 in the form of packets in three directions, and the image data of the physical device 12 generated by the receiving network camera 14. The computer 28 calculates the acceleration values in three directions to know that the physical device 12 is moving, tapping, and flipping. The action of turning or throwing, and identifying the physical device 12 provided with the pattern based on the image data, and calculating the position of the physical device 12 in the image.

The computer 28 combines the position and action of the physical device 12 in the image to generate image data of the physical device 42 (see FIG. 2) according to the calculation results, and the computer 28 finds a corresponding virtual object according to the pattern 46 of the physical device 12 ( As shown in FIG. 2), the image data of the image of the virtual object 44 is added to the image data of the image having the physical device 42 so that the image of the virtual object 44 is combined with the image of the physical device 42 and the computer 28 calculates the image according to the image. The acceleration values in the directions cause the virtual object 44 to perform the same action as the physical device 42. The computer 28 transmits the image data combined with the position and motion of the physical device 42 and the virtual object 44 to the screen 32.

The screen 32 receives image data transmitted by the computer 28 in combination with the position and motion of the physical device 42 and the virtual object 44 to display an image in which the position and motion of the physical device 42 and the virtual object 44 are combined.

The embodiment is not limited to using only one physical device, but a plurality of physical devices can be used. FIG. 3A is a schematic diagram of the system of the hybrid augmented reality and digital learning system of the present invention having two physical devices, and FIG. 3B is a schematic diagram of FIG. Another schematic diagram of the hybrid augmented reality and digital learning system of the present invention having two physical devices is shown.

In FIG. 3A, the left side is the case where the operator operates two physical devices 50, 52, and the right side displays an image of the operator operating a chemical experiment on the screen. The two physical devices 50, 52 are respectively provided with different patterns 54, 56, and then according to After the operation of the components of the hybrid augmented reality and digital learning system 10, the computer 28 identifies the image of the virtual object corresponding to the pattern 54 after identifying the pattern 54 of the physical device 50, and the virtual object shown in FIG. 3A. For the beaker 55, after the computer recognizes the pattern 56 of the physical device 52, the image of the virtual object corresponding to the pattern 56 is found. The virtual object shown in FIG. 3A is the dropper 58. The computer 28 transmits the image data of the image combined with the image of the physical device 50 and the beaker 55 to the screen 32, and transmits the image data of the image combined with the image of the physical device 52 and the image of the dropper 57 to the screen 32, thereby being operated by the operator. The operation of the two physical devices 50, 52 (i.e., the movement, shaking, etc. of the physical devices 50, 52 by the webcam 14) allows the operator to display an image of the chemical experiment of the dropper 57 and the beaker 55 on the screen.

Similarly, in FIG. 3B, the left side operator operates the two physical devices 60, 62, and the right side displays an image of the operator playing with the pet on the screen. The two physical devices 60, 62 are respectively provided with different patterns 64, 66, and according to the operation of the components of the hybrid augmented reality and digital learning system 10, the computer 28 looks for the pattern 64 of the physical device 60. An image of the virtual object corresponding to the pattern 64 is displayed. The virtual object shown in FIG. 3B is the pet 65. After the computer recognizes the pattern 66 of the physical device 62, the image of the virtual object corresponding to the pattern 66 is found, as shown in FIG. 3B. The virtual object displayed is fruit 68. The computer 28 transmits the image data combined with the image of the physical device 60 and the image of the pet 65 to the screen 32, and transmits the image data combined with the image of the physical device 62 and the image of the fruit 67 to the screen 32, thereby borrowing The operator can operate the two physical devices 60, 62 (ie, move, touch, throw, etc. of the physical devices 60, 62 via the webcam 14) to display on the screen that the operator is holding the fruit 67 and playing with the pet 65. Image.

4 is a system block diagram of a system for hybrid augmented reality and digital learning in accordance with another embodiment of the present invention. The components in FIG. 4 are the same as those in FIG. 1, and the same reference numerals are used, and the description thereof is omitted.

The embodiment of the hybrid augmented reality and digital learning system 70 of FIG. 4 illustrates that an operator can operate a physical object such as the self-propelled vehicle 74 by the physical device 12.

In FIG. 4, the hybrid augmented reality and digital learning system 70 includes a physical device 12, a webcam 14, a master device 72, and a self-propelled vehicle 74. The main control device 72 includes a computer 28, a radio frequency module 30, and a radio frequency module 76. The self-propelled vehicle 74 includes a radio frequency module 78, a processor 80, a servo motor 82, and a servo motor 84.

The computer 28 transmits the acceleration signal in one of the digits (ie, the ON/OFF signal) and the acceleration values in three directions to the RF module 76 in a packet form, and the RF module 76 wirelessly transmits the RF to the self-propelled vehicle 74. Module 78.

The RF device 78 wirelessly receives the digital start signal and the three-direction acceleration values emitted by the RF module 76, and transmits the digital start signal and the three-direction acceleration values to the processor 80.

The processor 80 receives the digital start signal and the acceleration values of the three directions transmitted by the radio frequency device 78, and calculates the acceleration values in three directions to obtain a moving direction value of the digital position. The processor 80 is based on a digital start signal (ON/OFF signal) to start the operation of the servo motors 82, 84, and to drive the servo motor 82 to drive the right wheel (not shown) of the carriage 74 and control the servo motor 84 in accordance with the digital movement direction value. The left wheel (not shown) of the self-propelled vehicle 74, by the operator operating the directional movement of the physical device 12, causes the self-propelled vehicle 74 to also have a directional movement.

The advantages of the present invention provide a system for hybrid augmented reality and digital learning that is used by children to replace traditional natural science experimental courses, and utilizes augmented reality techniques to utilize entities of the network camera and the system. The device, with the operation as the virtual reality of the real experiment, and the visual feedback of the augmented reality presents the traditional experimental results, and can enjoy the feeling of feedback and have a more real feeling.

The present invention has been described above with reference to the preferred embodiments and the accompanying drawings, and should not be considered as limiting. Various modifications, omissions and changes may be made without departing from the scope of the invention.

10‧‧‧Combined augmented reality and digital learning systems

12‧‧‧ physical devices

14‧‧‧Webcam

16‧‧‧Master control unit

20‧‧‧Accelerometer

22‧‧‧Touch sensor

24‧‧‧ Processor

26‧‧‧RF Module

28‧‧‧ computer

30‧‧‧RF Module

32‧‧‧ screen

42‧‧‧ physical devices

44‧‧‧Virtual objects

46‧‧‧ style

50‧‧‧ physical devices

52‧‧‧ physical devices

54‧‧‧ style

55‧‧‧Beaker

56‧‧‧ style

57‧‧‧ Dropper

60‧‧‧ physical devices

62‧‧‧ physical devices

64‧‧‧ style

65‧‧‧ Pets

66‧‧‧ style

67‧‧‧ Fruit

70‧‧‧Combined augmented reality and digital learning systems

72‧‧‧Master control unit

74‧‧‧Self-driving car

76‧‧‧RF Module

78‧‧‧RF Module

80‧‧‧ processor

82‧‧‧Servo motor

84‧‧‧Servo motor

1 is a system block diagram of a system for hybrid augmented reality and digital learning according to an embodiment of the present invention; FIG. 2 is a schematic diagram of a system for operating a hybrid augmented reality and digital learning according to the present invention; A system for mixing augmented reality and digital learning with two physical devices; 3B is another schematic diagram of a system for hybrid augmented reality and digital learning of the present invention having two physical devices; and FIG. 4 is a system of a system for hybrid augmented reality and digital learning according to another embodiment of the present invention; Block diagram.

10‧‧‧Combined augmented reality and digital learning systems

12‧‧‧ physical devices

14‧‧‧Webcam

16‧‧‧Master control unit

20‧‧‧Accelerometer

22‧‧‧Touch sensor

24‧‧‧ Processor

26‧‧‧RF Module

28‧‧‧ computer

30‧‧‧RF Module

32‧‧‧ screen

Claims (5)

A system for hybrid augmented reality and digital learning, comprising: a physical device comprising: an accelerometer for sensing an acceleration of the physical device to generate acceleration values in three directions; a touch sensor Sensing whether the physical device is touched to generate a touch signal, and a first processor for receiving the acceleration values of the three directions generated by the accelerometer and the generated by the touch sensor Touching a signal, and combining the acceleration values in three directions with the touch signal to transmit in a packet form; a network camera for capturing images to generate image data having the physical device; and a master device comprising: a computer, configured to receive acceleration values of the three directions transmitted by the first processor in a packet form, and the touch signal, and receive image data of the physical device generated by the network camera, and calculate acceleration values in three directions Knowing the action of the physical device and calculating the location of the physical device based on the image data, combined with the location and action of the physical device, Add a virtual object image data having the image data of the physical device, so that the image of the virtual image of the object to be bonded of the physical device, the virtual object and the same action with the entity device; to And a screen for receiving image data transmitted by the computer to display an image combining the virtual object and the physical device. The system of claim 1, wherein the physical device further comprises a first radio frequency module, and the first radio frequency module wirelessly transmits the acceleration values of the three directions transmitted by the first processor in a packet form and the The touch signal; and the main control device further includes a second RF module, and the second RF module wirelessly receives the acceleration values of the three directions wirelessly transmitted by the first RF module and the touch signal. The system of claim 1, wherein the accelerometer generates an acceleration value in three directions; the touch sensor produces an analogy to the touch signal; the first processor will be analogous to the three The acceleration value of the direction and the acceleration signal of the three directions of the touch signal and the touch signal, and combining the acceleration value of the three directions of the digit with the touch signal to transmit the computer of the main control device in a packet form And the computer receives the acceleration value of the digital direction and the touch signal transmitted by the first processor in a packet form. For example, the system of claim 3 includes a physical object, and the main control device includes a third radio frequency module, and the computer will activate the signal for one of the digits and the acceleration values of the three directions in the form of a packet. The third RF module wirelessly transmits to the physical object, and the physical object includes: a plurality of servo motors for driving the physical object; and a fourth RF device for wirelessly receiving the wireless transmission of the third RF module The activation signal of the digital position and the acceleration values of the three directions; and a processor for receiving the activation signal of the digits transmitted by the fourth radio frequency device and the acceleration values of the three directions, and calculating the acceleration values of the three directions Obtaining a digital direction of movement value, wherein the processor starts the operation of the servo motors according to the start signal of the digits, and drives the physical object according to the value of the movement direction of the digits to control the servo motors, The physical object is moved in a directional manner. The system of claim 1 further includes a physical object, the main control device includes a third radio frequency module, and the computer transmits an activation signal and acceleration values in three directions through the third radio frequency mode in a packet form. The wireless device transmits to the physical object, and the physical object includes: a plurality of servo motors for driving the physical object; and a fourth radio frequency device for wirelessly receiving the activation signal wirelessly transmitted by the third RF module and An acceleration value in three directions; and a processor for receiving the activation transmitted by the fourth radio frequency device The signal and the acceleration values in three directions are calculated, and the acceleration values in three directions are calculated to obtain a movement direction value, wherein the processor starts the operation of the servo motors according to the activation signal, and controls the servos according to the movement direction value. The motor drives the physical object to cause the physical object to move in a directional manner.