TWI789083B - Method and system for controlling augmented reality content playback andcomputer readable medium thererfor - Google Patents

Abstract

The present invention is a method for controlling augmented reality content playback and system thereof. A paragraph description file for the content of the augmented reality video is configured in the augmented reality video to generate each paragraph of the film and its audio-visual characteristics. When the augmented reality video is played by an electronic device as the augmented reality content, whether the scene situation meets the recorded characteristics can be compared, and compare results can be used for controlling the playback progress of the content of the augmented reality video. In addition, recording changes in the playback progress due to control actions can also be used as a reference for video content adjustment. Therefore, the present invention allows the user to focus on practical operations without distracting control of the playback progress of the augmented reality video, so that better augmented reality learning operations and efficiency are provided. The present invention also provides a computer-readable medium for executing the method of the present invention.

Description

Control method, system and computer-readable medium for augmented reality content playback

The present invention relates to a technique for playing augmented reality content, in particular to a control method and system for playing augmented reality content and its computer-readable medium.

"Augmented Reality", as the name implies, means to augment some information in reality, and the augmented content will vary according to different needs. The content superimposed on the current live-action images is already very varied and rich. However, whether the provided augmented reality content is consistent with the current situation and needs of users is the goal of each manufacturer's efforts to improve. .

The research and development of the aforementioned technology includes the improvement of the accuracy of image overlay, the adjustment of image color or brightness, and the improvement of virtual and real image fusion between augmented reality content objects and on-site objects. There are also many literature records on the application of environmental technology in education and training, such as news reports "App creates augmented-reality tutorials from normal videos" (IOXPin Mannheim, Germany, 2017), various companies or schools (such as Qiwei Co., Ltd. Company, Taiwan Normal University, Taiwan Panasonic, Beijing Shanxueyun Technology, Yangzhi Technology, etc.) developed various augmented reality systems. i.e. when a video is used When working on augmented reality content, how to know which section of the video corresponds to the on-site situation, so that the augmented reality device can play the appropriate video section, so as to increase the system efficiency.

Therefore, how to overcome the various deficiencies of the aforementioned existing technologies, especially, enable the augmented reality device to play appropriate video segments according to the on-site conditions, thereby improving the application benefits and efficiency of the augmented reality, which will become the current technology. The goal that people want to pursue.

In view of the above problems, the present invention proposes a control method for playing augmented reality content, which includes: providing an augmented reality module on an electronic device, so that the augmented reality module can be downloaded from a cloud server As a video of augmented reality content; during the execution of the augmented reality module, make the electronic device receive a real-time live multimedia data, so as to send the live multimedia data back to the cloud server; make the cloud server According to the paragraph description data of the video, the server extracts the corresponding part of the video content from the video; When the data matches the features contained in the content of the part of the video, the video will automatically jump to the specified paragraph according to the paragraph description data, so that the augmented reality module starts or continues to play the video from the specified paragraph.

In the above method, the video includes at least one real-scene image and augmented reality information superimposed on the at least one real-scene image.

In the above method, the electronic device displays the video on the virtual display screen, and the video is superimposed on a real-time real-time image.

In the aforementioned method, the augmented reality module further includes a voice control interface for receiving a voice command to control the playing behavior of the video.

In the aforementioned method, the paragraph description data is used to record information about the paragraph, including the start and end time of the paragraph and the multimedia features used for subsequent comparison operations.

In the aforementioned method, the cloud server generates a playback progress recommendation point based on the comparison result of the on-site multimedia data and the features contained in the part of the video content, as the basis for the video to automatically jump to the specified paragraph.

In the aforementioned method, the cloud server automatically records each playback process of the video, including the information of pause, cancel playback and jump to the paragraph from other places due to the comparison result of each paragraph point, as an adjustment for the purpose of this video.

The present invention further proposes a computer-readable medium, which is applied to a computing device or a computer and stores instructions to execute the aforementioned control method.

The present invention further proposes a control system for playing augmented reality content, which includes: a cloud server, which is used to provide videos for augmented reality content; and an electronic device, which communicates with the cloud server and has An augmented reality module, so that during the execution of the augmented reality module, the augmented reality module downloads the video from the cloud server, and receives a real-time on-site multimedia data and sends it back to the The cloud server, wherein the cloud server extracts the corresponding part of the video content from the video according to the paragraph description data of the video, and compares the on-site multimedia data with the features contained in the part of the video content, so as to When the on-site multimedia data matches the characteristics contained in the part of the video content, the video will automatically jump to the specified paragraph according to the paragraph description data, so that the augmented reality module starts or continues to play the video from the specified paragraph .

In the above system, the cloud server further includes: a video comparison unit, which is used to perform a comparison between the on-site multimedia data and the features contained in the part of the video content; and an audio-visual data processing unit is used to generate a playback progress recommendation point based on the comparison result of the video comparison unit, as the basis for the video to automatically jump to the specified paragraph.

In the above-mentioned system, the audio-visual data processing unit automatically records each playback process of the video, including the information of pause, cancel playback and jump to the paragraph from other places due to the comparison result of each paragraph point, as a Used to adjust this video.

In the aforementioned system, the electronic device displays the video on a virtual display screen, and the video is superimposed on a real-time real-time image.

In the aforementioned system, the video includes at least one real-scene image and augmented reality information superimposed on the at least one real-scene image.

In the aforementioned system, the augmented reality application further includes a voice control interface for receiving a voice command to control the playback behavior of the video.

In the aforementioned system, the paragraph description data is used to record information about the paragraph, including the start and end time of the paragraph and the multimedia features used for subsequent comparison operations.

To sum up, the control method and system for playing augmented reality content and its computer-readable medium of the present invention are to store videos and paragraph description data of the videos on a cloud server, and the user can use the augmented reality of the electronic device The environment module downloads a video from the cloud server and uses the video as the augmented reality content. During the execution of the augmented reality module, the electronic device receives a real-time on-site multimedia data and returns the on-site multimedia data. At the same time, the cloud server extracts the corresponding part of the video content from the video according to the paragraph description data of the video, and then compares the on-site multimedia data with the part of the video content to confirm whether it is consistent with The features contained in the content of this part of the video match. If they match, the video will automatically jump to the specified paragraph according to the paragraph description data. In other words, the augmented reality module can start from the specified paragraph according to the comparison result or continue playing the video as a live Augmented reality content. Therefore, the present invention allows users to focus on their own practical operations without being distracted by controlling the playback progress of the augmented reality video content, and by recording the automatic control of video playback, it can also be used as a video producer to adjust the augmented reality. environment video, and increase system efficiency.

10: Cloud server

101. U: Video comparison unit

102: Audio-visual data processing unit

20: Electronic equipment

201: Augmented Reality Module

30: Editing system

S101~S104: steps

S201~S207: Process

FIG. 1 is a step diagram of the control method for playing augmented reality content in the present invention.

FIG. 2 is a flow chart of a specific embodiment of the control method for playing augmented reality content in the present invention.

FIG. 3 is a schematic diagram of recording augmented reality content in the present invention.

FIG. 4 is a schematic diagram of a user performing practical operations with an augmented reality video in the present invention.

FIG. 5 is a system architecture diagram of a control system for playing augmented reality content according to the present invention.

The following describes the technical content of the present invention through specific embodiments, and those skilled in the art can easily understand the advantages and effects of the present invention from the content disclosed in this specification. However, the present invention can also be implemented or applied in other different specific implementation forms.

FIG. 1 is a step diagram of the control method for playing augmented reality content in the present invention. The object of the present invention is to provide a control for playing augmented reality content. By configuring a paragraph description file for the video content of augmented reality and recording each paragraph of the video and its audio-visual features, the electronic device can play the video as an extension. When augmenting reality content, it can compare whether the on-site situation conforms to the recorded characteristics, and once it matches, it can use this information to control the playback progress of the video content; secondly, in the process of augmenting reality, through recording Record the changes in the playback progress due to control actions, and report the records to the video content provider, which will be used as a reference for video content adjustments.

As shown in FIG. 1 , in step S101 , an augmented reality module is provided on an electronic device, so that the augmented reality module downloads a video used as augmented reality content from a cloud server. This step is to watch the video of the augmented reality content through the electronic device. In one embodiment, the electronic device is an augmented reality electronic glasses, which contains the augmented reality module and can connect to the cloud server , the electronic device is capable of carrying back a video and displaying the video through its virtual display screen, and the video is superimposed on a real-time real-time image. In one embodiment, the augmented reality module can be software, hardware or firmware; if it is hardware, it can be a processing unit, processor, computer or server with data processing and computing capabilities; if If it is software or firmware, it may include instructions executable by a processing unit, a processor, a computer or a server, or an application program.

In one embodiment, the video includes at least one real-scene image and augmented reality information superimposed on the at least one real-scene image. In other words, the video can be recorded by the same electronic device, that is, the same electronic device is used to record the video and watch the video, so that the viewing angles or ranges of the two can be consistent, so as to facilitate the comparison of subsequent effects.

In step S102, during the execution of the augmented reality module, make the electronic device receive a real-time live image or audio data (also called multimedia data) and return the live image or the audio data (multimedia data) to the cloud server. This step is to illustrate that when the augmented reality module is running, the electronic device can receive live images or audio data (multimedia data) in real time and send them back to the cloud server. Specifically, the augmented reality module is intermittent Capture live real-time images or record live real-time audio continuously or continuously, and upload the above-mentioned live real-time video and audio data to the cloud server, so as to ask to confirm where the current progress of the on-site operation is in the content of the augmented reality video. a paragraph.

In addition, the above-mentioned live real-time image screen contains at least one object displayed in the video content, and the above-mentioned live real-time sound contains the sound emitted by the object displayed in the teaching material video content. In short, the live real-time video screen Objects displayed in the video content, such as buttons, switches, lights, etc., should be included in the video content, so as to facilitate subsequent image comparison to determine which section of the augmented reality video content the current progress of the on-site operation is in. Similarly, through the capture of on-site sounds, such as button sounds, reminder sounds, warning sounds, etc., it can be determined which part of the augmented reality video content the current progress of the on-site operation is located in after subsequent comparison.

In step S103, the cloud server is instructed to extract a corresponding part of the video content from the video according to the paragraph description data of the video. This step is based on the confirmation requirements mentioned in the previous step, first obtain the video and its paragraph description data from the cloud server that stores the video, and extract each An image frame at the end time point of a paragraph or a short piece of sound feature data including the end time point of the paragraph; in addition, if it is the first paragraph, additionally capture the image frame at the start time point or include the sound feature data at the start time point A short piece of sound signature data.

As can be seen from the above, the paragraph description data records information about the video paragraph, including the start and end time of the paragraph and the image or sound features (also called multimedia features) used for subsequent comparison operations.

In step S104, make the cloud server compare the on-site image or the audio data (multimedia data) with the features contained in the part of the video content, so as to compare the live image or the audio data (multimedia data) with the part of the video When the characteristics contained in the content match, the video will automatically jump to a specified paragraph according to the paragraph description data, so that the augmented reality module starts or continues to play the video from the specified paragraph. Book The step is to compare the on-site operation situation with the audio-visual features of each paragraph or subsequent paragraphs in the film, so as to confirm whether the progress of the two is consistent.

In one embodiment, the cloud server generates a playback progress recommendation point based on the comparison result of the live image or the audio data (multimedia data) and the features contained in the part of the video content, as the video automatically jumps to the The basis for the specified paragraph. That is to say, by comparing the real-time image or sound of the scene with the audio-visual feature data of the video provided by the video storage cloud, it can be judged which section of the video is suitable for playing the current situation of the scene, and then, the video can be targeted A playback progress recommendation point is set, and the playback progress recommendation point is put into a response data, so as to respond to the request for confirming which section of the video the current progress of the on-site operation is in.

If the response data does not include any playback progress recommendation point, the augmented reality module can maintain the current playback status of the teaching video according to the result. Further, the play progress recommendation point may include a play instruction, such as pausing at this point, playing from this point, and so on.

In addition, the cloud server automatically records and counts the playback process of the video every time, including the pause, cancellation of playback and jumping to the paragraph from other places due to the comparison result of each paragraph point, as an adjustment to the video. For video purposes. In other words, the cloud server will automatically record and count the points during each playback of the video, and each paragraph point is paused, canceled, or jumped to the paragraph point due to the result of the comparison. Information, and report the above records and statistical information.

With the aforementioned records and statistical information, an editing system can be provided in the cloud server, by allowing the editing system to obtain the above-mentioned records and statistical information from the video storage, and provide a user interface, which can be displayed on the user interface The complete playback progress timeline of the video is further marked with the record and statistical information on the complete playback progress timeline, which can help to adjust the playback progress of subsequent video content.

In other embodiments, the augmented reality module further includes a voice control interface for receiving voice commands from the user to control the playback behavior of the video. In short, in order to prevent the virtual display screen of the electronic device from covering the live scene image, a voice control interface can be designed in the augmented reality module, in addition to controlling the playback behavior of the video through voice commands, such as start, pause, restart In addition to playing, forward, and backward, you can also hide or show the virtual display screen with voice commands, so as to avoid the augmented reality image covering the live scene image.

In other embodiments, the present invention can edit a video and complete the paragraph description data of the video through an editing system, and then transmit the video and its paragraph description data through a communication link and store them on the cloud server, specifically That is, through the editing system, the image features of the object contained in the image frame at the beginning or end of the specified paragraph can be specified. Further, the image feature of the object can further include the key points of the image features, and the key points of the image features can be compared in subsequent images. In the operation, the relative value of the necessity of image comparison and matching is increased to enhance the accuracy of the paragraph judgment of the video, and the paragraph description data of the video, in addition to recording the start and end time of a paragraph, can also include the paragraph the visual characteristics of the passage or the sound characteristics of the passage.

In the present invention, the augmented reality module of the electronic device communicates with the cloud server in the cloud, and according to the recommendation point of the video playback progress provided by it, requests the corresponding paragraph content of the video from the video storage place, so that in the expanded In the augmented reality module, the content of the video segment is used as real-time augmented reality content, so that the learners who use the augmented reality to operate, so that the augmented reality video can match the progress of the on-site operation. This enhances learning outcomes and increases system efficiency.

FIG. 2 is a flow chart of a specific embodiment of the control method for playing augmented reality content in the present invention.

In the process S201, a video and paragraph description data of the video are stored in a cloud. At In one embodiment, an editing system for augmented reality teaching material content can be used to edit teaching material videos. The editing system can be a computer program product running on an Internet browser, and can have a graphical user interface, allowing After logging into the editing system, the user can obtain a teaching video through the teaching material management interface in the editing system, and the aforementioned process of obtaining the teaching video can further be connected to a video storage cloud through the editing system, for example Network storage device, cloud hard disk, etc., and retrieve the teaching video from the video storage cloud. In addition, before the video storage cloud, it first receives the teaching material video recorded by it from an electronic device (such as augmented reality electronic glasses, which will be described later as an example). Furthermore, the above augmented reality The teaching video recorded by the electronic glasses is a teaching video with augmented reality information, wherein the content of the teaching video with augmented reality information is captured by the camera on the augmented reality electronic glasses The ingested real-scene images are mainly superimposed on the augmented reality information generated by operating the augmented reality electronic glasses.

Next, please refer to Figure 3. In the editing system, a teaching material video player is provided, so that the user who acts as a teacher can browse and view the content of the teaching material video. When the teaching material video is loaded, the editing system According to the length of the video, in this example, a total of 60 minutes, the teaching video can be automatically defined as a paragraph, which starts at 00:00 and ends at 60:00. Then, the teacher can use the "set paragraph" function provided by the editing system to split the teaching video into four paragraphs, and simply represent a paragraph in the format of "start time ~ end time". Therefore, the teaching video The four paragraphs can be expressed as "00:00~03:20", "03:21~10:00", "10:01~48:30", "48:31~60:00", further, the teacher The editing system can be used to specify the image characteristics of the object contained in the video frame at the beginning or end of the paragraph. For example, the teacher can pause the teaching video at 03:20 and select an object on the video frame. In this example, it is the knob on the upper left of the machine tool, and the editing system will record the image feature of the knob as the key point of the image feature at the end of the first paragraph. Further, If there are multiple selected objects, the editing system will record the image features of each object and the relative image features between the objects as image feature key points, which can be improved in subsequent image comparison operations The necessity value of its relative image comparison and matching is used to strengthen the accuracy of video paragraph judgment, and the above-mentioned relative image features can include the relative position of objects; in addition, the editing system can set the beginning and ending images of paragraphs In addition to the characteristics, further, the audio track data contained in the teaching video can also be used to set the sound characteristics of the beginning and ending of the paragraph. As can be seen from the above, the paragraph description data of the teaching material film, in addition to recording the start and end time of a paragraph, also includes the image feature of the paragraph or the sound feature of the paragraph.

Please also refer to Figure 4. U in the figure is a video comparison unit in the cloud server, which is used to use the audio-visual features contained in the paragraph description data of the video to determine whether the real-time on-site situation is consistent with a certain video in the video. paragraphs match.

In the process S202, an augmented reality module is provided on an electronic device, and the module can download the video from the cloud and use the video as augmented reality content. For example, when a user who acts as a student wants to use the teaching material video to watch and do it, so as to practice the operation of the machine tool, another augmented reality electronic glasses can be provided here (also can be used with The same one used by the teacher to record teaching materials), and let the augmented reality electronic glasses be pre-installed or equipped with an augmented reality module, so that students can connect with the augmented reality module through the augmented reality module during class exercises. Take the video and store it in the cloud, download the teaching material video of the tool machine operation, and after the students put on the augmented reality electronic glasses and activate the augmented reality module, they can play in the virtual environment provided by the augmented reality electronic glasses. The video of the teaching material can be seen on the display screen. In order to avoid the virtual display screen from covering the real scene image of the tool machine, a voice control interface can be implemented and set up in the augmented reality module. In addition to allowing students to control the teaching material through voice commands In addition to video playback behavior (start/pause/replay/forward/backward), you can also hide or show the virtual display screen by voice command.

In the process S203, the augmented reality module can receive a real-time on-site image or audio data (multimedia data) during execution, and provide the on-site image or audio data (multimedia data) to a video comparison unit . This process means that when students wear the augmented reality electronic glasses and use the textbook video as the augmented reality content, the augmented reality module can intermittently or continuously capture live real-time video images Or record the live real-time sound, and upload the above-mentioned live real-time audio-visual data to a video comparison unit U on the cloud server, so as to ask to confirm the current progress of the students' on-site operation and which section of the teaching material video they are in. Among them, The aforesaid on-site real-time image screen contains at least one object shown in the video content of the teaching material. In this embodiment, it is the machine tool and its auxiliary components. sound.

In the process S204, the cloud extracts a corresponding portion of video content from the video according to the paragraph description data of the video, and provides it to the video comparison unit. This process refers to building an audio-visual data processing unit on the cloud server and communicating with the video storage cloud, so that the audio-visual data processing unit can first process the confirmation request mentioned in S203 from the The video storage cloud office obtains the teaching material video and its paragraph description data, and according to the record of the paragraph description data, extracts from the teaching material video an image frame at the end time point of each paragraph or an image containing the end time point of the paragraph. A small piece of sound feature data, the first paragraph (00:00~03:20), and then additionally capture the image at the start time point or a short piece of sound feature data including the start time point, further, if the teacher had In these image frames, if any one or more image feature points are set by frame selection (as described in the process S201), then the or these image feature points can be additionally extracted. After all the data are extracted, all the data are provided to the video comparison unit U.

In another implementation, the video comparison unit U can also be set on an electronic device (such as augmented reality electronic glasses), or on another server, and let the video comparison unit U communicate with the cloud server The audio-visual data processing units above can be connected by communication, so that they can cooperate with each other.

In process S205, through the video comparison unit, compare the on-site image or audio data (multimedia data), whether it is consistent with the features contained in the part of the video content, and in process S206, if they match, then according to the paragraph description data, the The movie automatically jumps to the specified paragraph. The above two processes refer to using the video comparison unit U to compare the real-time image or sound of the scene with the audio-visual feature data of the teaching material video provided by the video storage cloud, so as to judge the current situation of the scene, which is suitable for playing the teaching material Which section in the video, and then reply the judgment result to the audio-visual data processing unit, so that the audio-visual data processing unit can set a playback progress recommendation point for the teaching material video, and put the playback progress recommendation point into a response data, In response to the current progress of the learner's on-site operation, which section of the teaching material video is currently in the confirmation request, if the response data does not include any playback progress recommendation points, the augmented reality module can rely on this result, The current playing state of the teaching material video is maintained, and further, the recommended playing progress point may include a playing instruction, such as pausing at this point and playing from this point.

Generally speaking, whenever the audio-visual data processing unit receives a confirmation request, it will let the video comparison unit U traverse all the set audio-visual features of the teaching material video, and compare them one by one with the ones mentioned in the confirmation request. On-site real-time audio-visual data comparison, further, we can let the augmented reality module, for the teaching video, additionally upload a current playback progress location information to the video comparison unit U, to facilitate the video comparison The unit U refers to the position information of the current playback progress, and compares the currently playing segment or the next segment to be played first, so as to improve efficiency. As shown in Figure 4, the left area is the real-time real-time scene image captured by the AR glasses when the students wear the AR glasses in actual operation, and the right area is the four videos in the teaching material. The image frames (frames) of paragraph points, wherein 1) is the start frame of the first paragraph, and then, the following are examples of four situations.

In the first situation, when the video comparison unit U recognizes that the scene is at the beginning of the teaching video, it will notify the augmented reality module through a communication link to automatically start playing the teaching video.

In the second situation, when the upper two images in the left area have already been captured, and the teaching material video plays relatively quickly, it has already reached the third image position from top to bottom in the right area, because the first The three images represent the end of the second section, but the students operate slowly, so the video comparison unit U has not received and recognized the third image in the left area during the playback of this section, so in this situation , the augmented reality module will be notified to pause the teaching material video at the end of the second paragraph, so as to wait for the students to operate to this progress, and then continue to play the third paragraph.

The third situation is opposite to the above-mentioned second situation. Before the second image is played in the right area, the students in the left area have actually operated the third image, so the video comparison unit U will notify the The augmented reality module allows it to fast-forward or jump to the beginning of the third paragraph of the teaching video, and continue to play.

In the fourth situation, when the teaching material video in the right area is playing the third segment of video content, but before the image in 4), the video comparison unit U suddenly receives the first image in the left area ( In the initial state of the scene), it means that the student suddenly wants to interrupt and start the practice again. Therefore, in this situation, the augmented reality module will be notified to automatically reverse or jump to the beginning of the first paragraph, and then continue to play.

Furthermore, in the above-mentioned several situations, the video comparison unit U will automatically record and count the teaching material video during each playback process, and each paragraph point is paused, canceled and played due to the result of the comparison. , jump to the information of the paragraph point from other places, and combine the above records and statistical information report to the video storage cloud, and when the teacher uses the editing system of the augmented reality teaching material content again, the above-mentioned records and statistical information can be obtained from the video storage cloud. In addition, the editing system can provide a user interface on which a timeline of the complete playback progress of the teaching video is displayed, and the record and statistical information can be marked on the timeline of the complete playback progress.

In the process S207, the augmented reality module starts or continues to play the video from the specified paragraph as the current augmented reality content of the scene. This process means that through the augmented reality module, the audio-visual data processing unit on the cloud can be connected by communication, and according to the recommended point of the playback progress of the teaching video provided by it, request the corresponding paragraph content of the teaching video to the video storage cloud , so as to read the paragraph content of the teaching material video into the augmented reality module, and use it as the real-time augmented reality content.

In another embodiment, the augmented reality module can download the entire teaching video file from the video storage cloud at one time, and then control the player according to the recommended points of the teaching video playback progress to automatically adjust the teaching video. playback position.

In other embodiments, the teaching video can be played in the background, that is, hidden or not superimposed on the live scene. At this time, the teaching video and the live image can still be continuously compared and secretly synchronized.

In other embodiments, context-based image comparison confirmation can be provided. Specifically, when comparing images, it is not limited to feature matching of a single teaching material video frame (frame), but the features in the current few frames are also Satisfied, after the formation of consecutive features matching, it will be used as the basis for confirming the current paragraph. For example, if the second paragraph of the teaching video ends, there is an image feature that the first indicator light of the machine tool lights up red, and we know The first indicator light of the machine tool is red, indicating that the first button is pressed and the second knob is rotated 90 degrees forward, and a certain paragraph in the teaching video also sees these image features in sequence, hereby In this design, the continuous matching of image features such as "finger object detected -> the first button is pressed -> the pointer of the second knob is rotated clockwise -> the first indicator light is red" is compared in the video ,when Only when this example appears is the end of the second paragraph. Therefore, if there is this design, the video comparison unit U needs to be equipped with a memory for successful video and audio feature comparison in order to support sequential video and audio feature comparison. Yes, moreover, the above-mentioned finger object or such physical objects that may appear on the scene can use AI technology in advance to pre-train the corresponding image feature model, so as to facilitate the comparison of the video comparison unit U or the content of the augmented reality teaching materials. editing system to use.

In other implementations, it is possible to carry out statistics on the synchronization of the teaching material videos, which includes feedback information such as students restarting the practice from the beginning, and the playback rhythm of a certain section of the video is too fast, so as to provide teachers with reference for further adjustment of the teaching materials.

FIG. 5 is a system architecture diagram of a control system for playing augmented reality content according to the present invention. As shown in the figure, a control system for playing augmented reality content includes a cloud server 10 and an electronic device 20, wherein the cloud server 10 is used to provide a video for augmented reality content, and the video includes at least A real scene image and augmented reality information superimposed on the at least one real scene image, the electronic device 20 communicates with the cloud server 10 and has an augmented reality module 201, and the electronic device 20 uses its virtual display screen Displaying the video, during the execution of the augmented reality module 201, overlaying the video on a real-time real-time image, causing the augmented reality module 201 to download the video from the cloud server 10, and simultaneously receive A real-time on-site image or audio data (multimedia data) is sent back to the cloud server 10.

The cloud server 10 can extract the corresponding part of the video content from the video according to the paragraph description data of the video, and compare the live image or the audio data (multimedia data) with the features contained in the part of the video content Yes, when the on-site image or the audio data (multimedia data) matches the features contained in the part of the video content, the video will automatically jump to the specified paragraph according to the paragraph description data, so that the augmented reality module 201 starts or continues to play the movie from the specified paragraph. In short, the paragraph description data is used to record information about the paragraph, including the start and end time of the paragraph and the In the image feature or sound feature (multimedia feature) of the follow-up comparison operation, the present invention compares the feature data contained in the on-site image or sound data (multimedia data) with the paragraph description data corresponding to the video, so as to determine the quality of the on-site operation The progress matches which segment in the video, and then adjust the playback status of the video, such as skipping a certain section, pausing or continuing to play.

As can be seen from the above, in a control system for playing augmented reality content of the present invention, a video and paragraph description data of the video are stored in a cloud server 10, and an augmented reality module is provided on the electronic device 20 201, the augmented reality module 201 can download the video from the cloud server 10, and use the video as the augmented reality content, and can receive live images from the scene in real time during the execution of the augmented reality module 201 Or sound data (multimedia data), that is, the current teaching operation situation of the wearer is captured through the electronic device 20, and the augmented reality module 201 is continuously receiving real-time on-site images or sound data in the process of playing the video (multimedia data), and then return the on-site image or audio data (multimedia data) to the cloud server 10 for comparison to determine whether to adjust the video playback progress.

Further, the cloud server 10 further includes a video comparison unit 101 and an audio-visual data processing unit 102 . When the cloud server 10 receives the on-site image or audio data (multimedia data) returned by the electronic device 20, the cloud server 10 will extract the corresponding part of the video content from the video according to the paragraph description data of the video, and Provided to the video comparison unit 101, through which the video comparison unit 101 compares the on-site image or audio data (multimedia data), whether it is consistent with the features contained in the part of the video content, and if it is consistent, then according to the paragraph description data, the The video automatically jumps to a designated paragraph, that is, the audio-visual data processing unit 102 generates a playback progress recommendation point according to the comparison result of the video comparison unit 101, as the basis for automatically jumping to the designated paragraph for the video, so , when the wearer (student or learner) executes the augmented reality module 201 for operation learning, the augmented reality module 201 can start or continue to play the video from the specified paragraph according to the aforementioned comparison result, as the current augmented reality content of the scene.

In one embodiment, if there are multiple paragraphs, they have a sequence relationship in the video playing time sequence; in addition, any paragraph has a feature representing the end of the paragraph. Further, when the augmented reality module 201 is playing a certain section of the movie, if the feature of the end of the section does not match any live video or audio data during the playing time of the section, Then when playing to the end of the segment, the playback of the video will be automatically paused.

In addition, a video editor (such as a teacher) can edit the video through the editing system 30, and the video can be obtained by the electronic device 20. After editing, a video containing paragraph description data is generated and stored in the cloud server 10, such as video storage Cloud, follow-up, the learner (such as a student) wears the electronic device 20 for operation learning, obtains the on-site operation situation through the electronic device 20 and sends it back to the cloud server 10 for comparison, and then confirms whether the learner's operation progress is consistent with the video playback content match, and further adjust the video playback state, such as the present invention can provide a good learning mode in augmented reality learning, to prevent the learner from being unable to keep up with the progress in operation and unable to complete the study, or because the learner is already familiar with the video but the video has been played Slow and affect the learning effect and other issues.

The module, unit, device, equipment, server, etc. of the present invention include a microprocessor and a memory, and algorithms, data, programs, etc. are stored in the memory or chip, and the microprocessor can load data from the memory or Algorithms or programs perform data analysis or calculation processing, which will not be described in detail here. In other words, the cloud server and electronic equipment of the present invention belong to different independent structures. Among them, the cloud server is not limited to a single device, but can also be presented as a large-scale control center. The video comparison unit and The audio-visual data processing unit can be built in a single device or presented as separate devices (each with a server structure). In addition, the electronic device is a device used in augmented reality, such as wearable augmented Reality Glasses, but not limited to. Therefore, the components of the present invention can be designed as calculators, memories, storage devices, or firmware architectures with processing units, or in combination with software programs, hardware, or firmware architectures. presented.

In addition, the present invention also discloses a computer-readable medium, which is applied to a computing device or computer having a processor (for example, CPU, GPU, etc.) and/or memory, and stores instructions, and can be used by this computing device or The computer executes the computer-readable medium through the processor and/or memory, so as to execute the above-mentioned method and each step when executing the computer-readable medium.

To sum up, the control method and system for playing augmented reality content and its computer-readable medium of the present invention are to store videos and paragraph description data of the videos on the cloud server, and users can use the augmented reality content on electronic devices to The reality module downloads the video from the cloud server and uses the video as the augmented reality content. During the execution of the augmented reality module, the electronic device can receive a real-time on-site image or audio data (multimedia data) and Return the on-site image or the audio data (multimedia data) to the cloud server, and at the same time, the cloud server extracts the corresponding part of the video content from the video according to the paragraph description data of the video, and then the live image Or the audio data (multimedia data) is compared with the part of the video content, and the augmented reality module can start or continue to play the video from the specified paragraph according to the comparison result, as the current augmented reality on the spot environmental content. Therefore, the present invention can achieve the following technical effects.

The invention allows users not only to be hands-free, but also to be mind-free, so that they do not need to be distracted from the control of the playback progress of the AR video content, but instead focus on their own practical operations and increase system efficiency.

The invention allows the user to learn in paragraphs, so that the user has a chapter-like key impression of the overall situation, and it is easier to learn and memorize.

The present invention can further detect the situation of automatic control of AR video playback, collect them and feed them back to the video recorder as a reference for adjusting the content of the video.

The above-mentioned embodiments are only illustrative to illustrate the principles and effects of the present invention, and are not intended to limit the present invention. Anyone skilled in the art can modify and change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the rights of the present invention should be listed in the scope of the patent application described later.

S101~S104: steps

Claims (13)

A control method for playing augmented reality content, including: providing an augmented reality module on an electronic device, so that the augmented reality module downloads a video used as the augmented reality content from a cloud server , wherein, the video includes at least one real scene image and augmented reality information superimposed on the at least one real scene image; during the augmented reality module playing the video, the electronic device receives a real-time on-site multimedia Data, so as to return the on-site multimedia data to the cloud server; make the cloud server extract the corresponding part of the video content from the video according to the paragraph description data of the video; and make the cloud server The on-site multimedia data is compared with the features contained in the part of the video content, so that when the on-site multimedia data matches the features contained in the part of the video content, the video will automatically jump to the specified paragraph according to the paragraph description data, so that the extended The augmented reality module starts or continues to play the video from the specified paragraph. The control method for playing augmented reality content as described in Claim 1, wherein the electronic device displays the video on a virtual display screen, and the video is superimposed on a real-time real-time image. The method for controlling playback of augmented reality content as described in Claim 1, wherein the augmented reality module further includes a voice control interface for receiving a voice command to control the playback behavior of the video. The method for controlling the playback of augmented reality content as described in Claim 1, wherein the paragraph description data is used to record information about the paragraph, including the start and end time of the paragraph and the multimedia features used for subsequent comparison operations. The method for controlling the playback of augmented reality content as described in Claim 1, wherein the cloud server generates a playback progress recommendation point based on the comparison result of the on-site multimedia data and the features contained in the part of the video content, as the The basis for the video to automatically jump to the specified paragraph. The method for controlling the playback of augmented reality content as described in Claim 1, wherein the cloud server automatically records each playback process of the video, including the pause and cancellation of playback at each paragraph point due to the comparison result and information about jumping to that paragraph from elsewhere to adjust the video. A computer-readable medium, used in a computing device or a computer, stores instructions to execute the method for controlling playback of augmented reality content as described in any one of Claims 1 to 6. A control system for playing augmented reality content, comprising: a cloud server for providing a video used as augmented reality content, wherein the video includes at least one real-scene image and is superimposed on the at least one real-scene image The augmented reality information on the above; and the electronic device, which is in communication with the cloud server and has an augmented reality module, so that during the execution of the augmented reality module, the augmented reality module Download and play the video from the cloud server, receive a real-time on-site multimedia data and send it back to the cloud server, wherein the cloud server extracts the corresponding information from the video according to the paragraph description data of the video part of the video content, and compare the on-site multimedia data with the features contained in the part of the video content, so that when the on-site multimedia data matches the features contained in the part of the video content, the video will be automatically skipped according to the paragraph description data to a specified segment, so that the augmented reality module starts or continues to play the video from the specified segment. The control system for playing augmented reality content as described in Claim 8, wherein the cloud server further includes: The video comparison unit is used to compare the on-site multimedia data with the features contained in the part of the video content; and the audio-visual data processing unit is used to generate a playback progress recommendation point based on the comparison result of the video comparison unit , as the basis for the video to automatically jump to the specified paragraph. The control system for playing augmented reality content as described in Claim 9, wherein the audio-visual data processing unit automatically records each playback process of the video, including the pause and cancellation of each paragraph point due to the comparison result Play and jump from elsewhere to that segment of information for use in tweaking the video. The control system for playing augmented reality content according to Claim 8, wherein the electronic device displays the video on a virtual display screen, and the video is superimposed on a real-time real-time image. The control system for playing augmented reality content as described in Claim 8, wherein the paragraph description data is used to record information about the paragraph, including the start and end time of the paragraph and the image features or sounds used for subsequent comparison operations feature. The control system for playing augmented reality content as described in Claim 8, wherein the augmented reality module further includes a voice control interface for receiving a voice command to control the playback behavior of the video.

Images