KR20200053914A - Apparatus and Method for Searching Information using Augmented Reality and Mixed Reality - Google Patents

Abstract

The present invention relates to an apparatus for searching for information by using augmented reality (AR) and mixed reality (MR), which can learn various objects in the real world through a user terminal and provide AR/MR content and a web search function; and a method thereof. The apparatus for searching for information by using AR and MR comprises: an input unit for receiving information on images captured through a camera of a user terminal and objects selected from objects recognized on the screen of the user terminal; an image extraction unit for extracting the images input to the input unit; an image analysis unit for recognizing the objects for the contents of the images by analyzing the images extracted from the image extraction unit in chronological order; an object generation unit for generating one piece of class content including the characteristics and properties of the objects based on the objects defined in the image analysis unit by using Java script object notation (JSON); an object extraction unit for extracting characteristic and property information on class content selected among pieces of class content displayed on the screen of the user terminal; an AR/MR content detection unit for detecting AR/MR education content linked with the objects by searching for AR/MR education content previously stored in an AR/MR content database based on the object information extracted from the object extraction unit; and a search information generation unit for generating information content by using JSON for the AR/MR education content detected by the AR/MR content detection unit.

Description

Apparatus and Method for Searching Information using Augmented Reality and Mixed Reality

The present invention relates to an information retrieval apparatus and method for providing augmented reality and composite reality (AR / MR) content and a web search function capable of learning various objects in the real world through a user terminal.

In general, a well-known search engine technology is based on keyword search terms, and is a method of analyzing and indexing words of web data and providing information data matching the keyword search terms of information users. Here, search engines are software that helps you find materials on the Internet easily.

Various services are provided through such search engine technology, and among them, education services using the service are steadily increasing due to various demands for educational contents.

However, the current educational service shows the limitations of providing only one-dimensional information, such as listening practice, intensive speaking, and listening again, depending on the voice guidance broadcast and learning content broadcast displayed and heard by the system.

As a way to overcome this limitation, augmented reality (AR) has recently been applied to the educational field, creating a lot of contents that provide information through various senses. In particular, the American augmented reality company Magic Leap implements animals that are difficult to encounter through augmented reality so that students can vividly experience things that are difficult to experience in school.

In addition to this, there is a trend in which educational techniques for viewing a book's contents in three dimensions through a display are used in a variety of ways.

Also, according to the Ministry of Education of Korea, it is planning to disseminate immersive digital textbooks using AR technology soon in Korea, and has already started distributing digital textbooks to elementary and junior high schools since 2015.

As such, current field education is provided through educational contents incorporating augmented reality (AR), and the demand for various services is increasing accordingly.

In order to solve these needs, the present invention provides an information retrieval apparatus and method for providing augmented reality and complex reality (AR / MR) content and a web search function to learn about various objects in the real world through a user terminal. It is aimed at providing.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention not mentioned can be understood by the following description, and will be more clearly understood by embodiments of the present invention. In addition, it will be readily appreciated that the objects and advantages of the present invention can be realized by means of the appended claims and combinations thereof.

In order to achieve the above object, the information retrieval apparatus using the augmented reality and complex reality of the present invention is an input unit that receives information about a selected object among images captured through a camera of a user terminal and objects recognized on a screen of the user terminal. ; An image extraction unit for extracting an image input to the input unit; An image analysis unit that analyzes the images extracted from the image extraction unit in chronological order to recognize objects of image content; An object generating unit that generates one class content including characteristics and properties of an object based on an object defined in the image analysis unit by using JSON (Jave Script Object Notation); An object extracting unit for extracting property and attribute information on an object of a selected class content among class content displayed on a user terminal screen; An AR / MR content detection unit that detects AR / MR education contents linked with an object by searching for AR / MR education contents previously stored in an AR / MR content database based on the object information extracted from the object extraction unit; And a search information generation unit that generates information content by using AR / MR education content detected by the AR / MR content detection unit using JSON (Jave Script Object Notation).

In addition, the information retrieval apparatus using the augmented reality and composite reality includes a keyword extracting unit that extracts a keyword associated with an object based on the object information extracted from the object extraction unit; A web search processing unit searching for information related to keywords among information accessible through the web by performing a web search of the legacy system through keywords extracted from the keyword extraction unit; And a result list generator configured to generate a web search result list by sorting information searched by the web search processor by at least one of time, association, type, and type, and the search information generator generates the result list. Information content is generated by using the generated web search result list using JSON (Jave Script Object Notation).

In addition, the image analysis unit divides the real-time image at regular intervals in chronological order, or directly divides it into a specified distance value, and defines at least one object from the divided image.

In addition, the search information generation unit generates one class content including the properties and properties of each object defined by utilizing JSON (Jave Script Object Notation).

In addition, the search information generation unit generates AR / MR education contents in which the detected AR / MR education contents are combined with real-time images extracted from the image extraction unit.

In order to achieve this object, the information retrieval method using the augmented reality and complex reality of the present invention, (A) machine learning (Machine learning) technique by utilizing the off-source TensorFlow-based TensorFlow Object Detection API Storing the searched AR / MR education content in the AR / MR content database (DB) using the data provided by TensorFlow through; (B) When an image photographed through a camera of a user terminal is input, the input image is analyzed in chronological order through the image analysis unit, and one class content including characteristics and attributes of the object is generated through the object generation unit step; (C) when one of the class contents displayed on the user terminal screen is selected, extracting characteristic and attribute information on the object of the selected class content through an object extraction unit; (D) the AR / MR content detection unit detects AR / MR education content linked with the object by searching for AR / MR education content previously stored in the AR / MR content database based on the extracted object information; And (E) generating the information content by providing the detected AR / MR educational content to the user terminal by using the JSON (Jave Script Object Notation).

In addition, the analysis of the image input in the step (B) may include dividing the real-time image at regular intervals in sequential order or dividing it by a specified distance value, and defining at least one object in the segmented image. Includes.

In addition, in step (B), the generation of class content generates one class content including characteristics and properties of each object defined by using JSON (Jave Script Object Notation).

In addition, the information retrieval method using the augmented reality and the complex reality comprises: extracting keywords associated with an object based on the extracted object information in a keyword extractor; Performing a web search of the legacy system through the extracted keyword in the web search processing unit and searching for information related to the keyword among information accessible through the web; Generating a web search result list by sorting the information searched through the result list generation unit at least one of time, association, type, and type; And generating the information content using the JSON (Jave Script Object Notation), and providing the generated web search result list in the search information generation unit to the user terminal.

In addition, the step (E) includes the step of generating AR / MR educational content combining the detected AR / MR educational content and the real-time video extracted from the video extraction unit through the search information generation unit.

According to the present invention as described above, the information retrieval apparatus and method using augmented reality and complex reality of the present invention, augmented reality and complex reality (AR / MR) capable of learning about various objects in the real world through a user terminal By providing content and web search capabilities, it is possible to provide a new learning experience that is real and immersive.

In addition, it is possible to provide a search for information that a user wants more efficiently because it is possible to vividly experience difficult things that are difficult to access without spatial and temporal restrictions.

In addition to the above-described effects, the concrete effects of the present invention will be described together while describing the specific matters for carrying out the invention.

1 is a block diagram showing an information retrieval apparatus using augmented reality and complex reality according to an embodiment of the present invention.
FIG. 2 is a block diagram showing a detailed configuration of the information retrieval server of FIG. 1.
3 is a flowchart illustrating an information retrieval method using augmented reality and complex reality according to an embodiment of the present invention
Figure 4 is an embodiment showing that the information search results provided by the information search server is displayed on the screen of the user terminal.

The above-described objects, features, and advantages will be described in detail below with reference to the accompanying drawings, and accordingly, a person skilled in the art to which the present invention pertains can easily implement the technical spirit of the present invention. In the description of the present invention, when it is determined that detailed descriptions of known technologies related to the present invention may unnecessarily obscure the subject matter of the present invention, detailed descriptions will be omitted. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings are used to indicate the same or similar components.

Hereinafter, an information retrieval apparatus and method using augmented reality and complex reality according to a preferred embodiment of the present invention will be described in detail with reference to the drawings.

1 is a block diagram showing an information retrieval apparatus using augmented reality and complex reality according to an embodiment of the present invention.

As shown in FIG. 1, the information retrieval device includes a user terminal 100 that checks the real-time image transmission for information retrieval and AR / MR education content search results, a legacy system 200 in which a data source is stored, and a user And an information retrieval server 300 that processes a search of related AR / MR educational contents and web search of the legacy system 200 through related keywords by analyzing real-time images received from the terminal 100.

The user terminal 100 transmits a real-time image (streaming image) photographed through a camera to the information retrieval server 300, and receives an information retrieval result based on the analysis result of the real-time image from the information retrieval server 300 Refers to. For example, the user terminal 100 may be a portable terminal such as a smart phone, a tablet PC, and a PDA. However, the user terminal 100 is not limited thereto, and any device that can access the information retrieval device 200 through wired / wireless communication may be included.

The legacy system 200 refers to a data distribution storage environment that stores a data source to be searched for information, and includes, for example, a server that stores and manages various information accessible through the web as a data source.

And the information retrieval server 300 analyzes the real-time image input from the user, searches the related AR / MR education contents based on the analyzed real-time image results, and searches the web through related keywords in the legacy system 200. As a device for processing, for example, a web server may correspond to this.

FIG. 2 is a block diagram showing a detailed configuration of the information retrieval server of FIG. 1. At this time, the information retrieval server 300 shown in FIG. 2 is according to an embodiment, and its components are not limited to the embodiment shown in FIG. 2, and some components are added, changed, or deleted as necessary Can be. And Figure 4 is an embodiment showing that the information search results provided by the information search server is displayed on the screen of the user terminal 100.

As shown in FIG. 2, the information retrieval server 300 includes an input unit 301, an image extraction unit 302, an image analysis unit 303, an object generation unit 304, an object extraction unit 305, and an AR. / MR content detection unit 306, keyword extraction unit 307, web search processing unit 308, result list generation unit 309, search information generation unit 310, AR / MR content database (DB) 311 Includes.

The input unit 301 receives information on an object selected by a user from a real-time image (streaming image) captured through a camera of the user terminal 100 and an object recognized on the screen of the user terminal 100.

The image extraction unit 302 extracts a real-time image input to the input unit 301.

The image analysis unit 303 analyzes the real-time images extracted by the image extraction unit 302 in chronological order to recognize objects of image content. That is, the image analysis unit 303 divides the real-time image at regular intervals in chronological order or directly divides the distance value. And at least one object is defined in the segmented image.

The object generation unit 304 generates one class content including characteristics and properties of an object based on an object defined in the image analysis unit 303 by using JSON (Jave Script Object Notation). For example, the characteristics of the object called student include the coordinates of the object, the box size (x, y value), ID, etc., and the properties of the object called student include the student's number, name, grade, and place of residence. In addition, class content refers to content embodied in the text (name) of students including these characteristics.

The class content generated by the object generating unit 304 is delivered to the user terminal 100 as shown in FIG. 4 (b), and the class content 10 in which the object name is described by box processing the recognized object Is displayed on the user terminal 100 screen.

The object extraction unit 305 extracts characteristic and attribute information (hereinafter referred to as 'object information') of an object of the selected class content 10 among the class content 10 displayed on the screen of the user terminal 100.

The AR / MR content detection unit 306 searches for AR / MR educational content previously stored in the AR / MR content database (DB) 311 based on the object information extracted from the object extraction unit 305, and searches for objects and Detect interlocked AR / MR educational content. At this time, AR / MR educational content is augmented reality and complex reality (AR / MR) content made to learn about various objects in the real world, and utilizes the off-source TensorFlow Object Detection API as a machine. It uses the data provided by TensorFlow through a machine learning technique. The AR / MR content database (DB) 311 updates the stored AR / MR education content through continuous training.

The keyword extraction unit 307 extracts a keyword associated with an object based on the object information extracted by the object extraction unit 305. At this time, the extracted keyword may be extracted from the attribute information on the object, but is not limited thereto, and the keyword associated with the attribute information may also be extracted.

The web search processing unit 308 performs a web search of the legacy system 200 through the keywords extracted by the keyword extraction unit 307 to search for information related to keywords among various information accessible through the web.

The result list generation unit 309 sorts information searched by the web search processing unit 308 by time, degree of association, type, type, and the like to generate a web search result list.

The search information generation unit 310 utilizes a web search result list generated by the AR / MR education content and result list generation unit 309 detected by the AR / MR content detection unit 306 using JSON (Jave Script Object Notation). To create information content respectively. The search information generation unit 310 may generate AR / MR education content that combines the detected AR / MR education content with the real-time video extracted by the image extraction unit 302.

The information content generated by the search information generation unit 310 is delivered to the user terminal 100 as shown in FIG. 4 (c) and displayed on the screen. Accordingly, when the user selects the information content displayed on the user terminal 100, augmented reality and complex reality (AR / MR) content and a web search function capable of learning various objects in the real world can be provided. As a result, you can experience new learning that is real and immersive.

In addition, the user can select and confirm a required part from the list displayed on the user terminal 100.

The operation of the information retrieval apparatus using augmented reality and composite reality according to the present invention configured as described above will be described in detail with reference to the accompanying drawings. The same reference numerals as in FIGS. 1 and 2 refer to the same members performing the same functions.

3 is a flowchart illustrating an information retrieval method using augmented reality and complex reality according to an embodiment of the present invention.

Referring to FIG. 3, first, the information retrieval server 300 utilizes data provided by TensorFlow through a machine learning technique by utilizing an off-source TensorFlow Object Detection API. Then, the searched AR / MR educational content is stored in the AR / MR content database (DB) 311 (S10). However, AR / MR educational content stored in the AR / MR content database (DB) 311 is not limited thereto, and may be produced and stored in the information search server 300.

In addition, the AR / MR education content stored in the AR / MR content database (DB) 311 is updated through continuous training.

Subsequently, when a real-time image (streaming image) photographed through the camera of the user terminal 100 is input, the real-time image input through the image analysis unit 303 is analyzed in chronological order, and through the object generation unit 304 One class content including object characteristics and properties is generated (S20).

At this time, the method of analyzing the image is first, the real-time image is segmented at regular intervals in chronological order, or it is divided into a specified distance value. And at least one object is defined in the segmented image.

And the method of creating an object creates one class content that includes the properties and properties of the object based on each object defined by using JSON (Jave Script Object Notation). For example, the characteristics of the object called student include the coordinates of the object, the box size (x, y value), ID, etc., and the properties of the object called student include the student's number, name, grade, and place of residence. In addition, class content refers to content embodied in the text (name) of students including these characteristics.

Then, the generated class content is delivered to the user terminal 100 as shown in FIG. 4 (b), and the user terminal 100 is used as the class content 10 in which the object name is described by box processing on the recognized object. It is displayed on the screen (S30).

Thereafter, when one of the class contents 10 displayed on the screen of the user terminal 100 is selected, the object information (attribute and attribute information) for the object of the selected class content 10 is extracted through the object extraction unit 305, Recognize information about the object (S40).

Then, the AR / MR content detection unit 306 searches for AR / MR education content previously stored in the AR / MR content database (DB) 311 based on the extracted object information, and AR / MR education linked with the object. Content is detected.

In addition, the keyword extraction unit 307 extracts a keyword associated with an object based on the extracted object information, and the web search processing unit 308 performs a web search of the legacy system 200 through the extracted keyword to search the web. Among various information that can be accessed through, search for information related to keywords. Then, the information searched through the result list generator 309 is sorted by time, degree of association, type, type, and the like to generate a web search result list.

Subsequently, the search information generation unit 310 generates each of the detected AR / MR education content and the generated web search result list using JSON (Jave Script Object Notation), and provides the information content to the user terminal 100 (S50).

The generated information content is delivered to the user terminal 100 as shown in FIG. 4 (c) and displayed on the screen.

Accordingly, when the user selects the information content displayed on the user terminal 100, augmented reality and complex reality (AR / MR) content and a web search function capable of learning various objects in the real world can be provided. As a result, you can experience new learning that is real and immersive.

In addition, the user can select and confirm a required part from the list displayed on the user terminal 100.

The above-described present invention, the above-described embodiment and the accompanying drawings because it is possible for a person having ordinary knowledge in the technical field to which the present invention belongs, various substitutions, modifications and changes are possible without departing from the technical spirit of the present invention. It is not limited by.

100: user terminal 200: legacy system
300: information search server 301: input unit
302: image extraction unit 303: image analysis unit
304: object creation unit 305: object extraction unit
306: AR / MR content detection unit 307: Keyword extraction unit
308: Web search processing unit 309: Result list generation unit
310: search information generation unit 311: AR / MR content DB

Claims (10)

An input unit that receives information on a selected object from an image captured through a camera of the user terminal and an object recognized on the screen of the user terminal;
An image extraction unit for extracting an image input to the input unit;
An image analysis unit that analyzes the images extracted from the image extraction unit in chronological order to recognize objects of image content;
An object generating unit that generates one class content including characteristics and properties of an object based on an object defined in the image analysis unit by using JSON (Jave Script Object Notation);
An object extracting unit for extracting property and attribute information on an object of a selected class content among class content displayed on a user terminal screen;
An AR / MR content detection unit that detects AR / MR education contents linked with an object by searching for AR / MR education contents previously stored in an AR / MR content database based on the object information extracted from the object extraction unit; And
An information retrieval device using augmented reality and complex reality including a retrieval information generator that generates information content by using AR / MR education content detected by the AR / MR content detector using JSON (Jave Script Object Notation).
According to claim 1,
The information search device using the augmented reality and composite reality
A keyword extracting unit extracting keywords associated with an object based on the object information extracted by the object extracting unit;
A web search processing unit searching for information related to keywords among information accessible through the web by performing a web search of the legacy system through keywords extracted from the keyword extraction unit; And
Further comprising a result list generating unit for generating a web search result list by sorting the information searched by the web search processing unit by at least one of time, association, type, and type,
The search information generation unit is an information search device using augmented reality and complex reality to generate information content by using a web (Jave Script Object Notation) JSON web search result list generated by the result list generation unit.
According to claim 1,
The image analysis unit is a device for retrieving information using augmented reality and complex reality that divides a real-time image at regular intervals in chronological order or divides it by a specified distance value and defines at least one object from the segmented image.
The method of claim 3,
The search information generation unit
An information retrieval device using augmented reality and complex reality that generates one class content including the properties and properties of each object defined above by using JSON (Jave Script Object Notation).
According to claim 1,
The search information generation unit
An information retrieval apparatus using augmented reality and complex reality to generate AR / MR education content that combines the detected AR / MR education content with real-time video extracted from the image extraction unit.
(A) AR / MR training contents retrieved using the data provided by TensorFlow through machine learning technique by utilizing the TensorFlow Object Detection API, which is an off-source, AR / MR contents database Storing in (DB);
(B) When an image photographed through a camera of a user terminal is input, the input image is analyzed in chronological order through the image analysis unit, and one class content including characteristics and attributes of the object is generated through the object generation unit step;
(C) when one of the class contents displayed on the user terminal screen is selected, extracting characteristic and attribute information on the object of the selected class content through an object extraction unit;
(D) the AR / MR content detection unit detects AR / MR education content linked with the object by searching for AR / MR education content previously stored in the AR / MR content database based on the extracted object information; And
(E) information using the augmented reality and complex reality, including the step of generating and providing the information content to the user terminal using the detected AR / MR education content in the search information generation unit using JSON (Jave Script Object Notation). How to search.
The method of claim 6,
Analysis of the image input in step (B) is
Dividing the real-time video at regular intervals in chronological order, or dividing it into a distance value manually designated;
And defining at least one object in the segmented image.
The method of claim 7,
In step (B), the generation of the class content is an information retrieval method using augmented reality and complex reality that generates one class content including the properties and properties of each defined object by using JSON (Jave Script Object Notation). .
The method of claim 6,
The information search method using the augmented reality and composite reality
Extracting a keyword associated with the object based on the extracted object information from the keyword extraction unit;
Performing a web search of the legacy system through the extracted keyword in the web search processing unit and searching for information related to the keyword among information accessible through the web;
Generating a web search result list by sorting the information searched through the result list generation unit at least one of time, association, type, and type; And
The information search method using augmented reality and composite reality further comprising the step of generating information content by using the JSON (Jave Script Object Notation) to generate the web search result list in the search information generating unit to the user terminal.
The method of claim 6,
Step (E) is
And an AR / MR educational content combining the detected AR / MR educational content and a real-time video extracted from the image extraction unit through a search information generator.

Images