KR102183363B1 - Apparatus and Method for Searching Information using Augmented Reality and Mixed Reality - Google Patents

Abstract

The present invention relates to an information retrieval apparatus and method for providing augmented reality and mixed reality (AR/MR) content and a web search function capable of learning about various objects in the real world through a user terminal. An information retrieval apparatus using augmented reality and complex reality of the present invention includes: an input unit for receiving information on a selected object among images captured through a camera of a user terminal and objects recognized on a screen of the user terminal; An image extraction unit for extracting an image input to the input unit; An image analysis unit that analyzes the images extracted by the image extraction unit in chronological order and recognizes an object of image content; An object generator that generates one class content including characteristics and properties of an object based on the object defined by the image analysis unit using JSON (Jave Script Object Notation); An object extracting unit for extracting characteristic and attribute information of an object of the selected class content from among the class content displayed on the user terminal screen; An AR/MR content detector configured to search for AR/MR educational content previously stored in an AR/MR content database based on the object information extracted by the object extractor to detect AR/MR educational content linked to the object; And a search information generation unit for generating information content by using JSON (Jave Script Object Notation) in the AR/MR education content detected by the AR/MR content detection unit.

Description

Apparatus and Method for Searching Information using Augmented Reality and Mixed Reality}

The present invention relates to an information retrieval apparatus and method for providing augmented reality and mixed reality (AR/MR) content and a web search function capable of learning about various objects in the real world through a user terminal.

In general, a well-known search engine technology is based on keyword search terms, and is a method of analyzing and indexing words in web data, and providing information data that matches the keyword search word of information users. Here, a search engine refers to software that makes it easy to find data on the Internet.

Various services are provided through such search engine technology, and among them, educational services using the same are steadily increasing due to various demands for educational contents.

However, the current educational service shows a limitation in providing only one-dimensional information such as listening practice, intensive speaking, and listening to the desired part again according to the voice announcement and learning content broadcast displayed and heard by the system.

As a way to overcome this limitation, by applying Augmented Reality (AR) to the education field, a lot of contents that provide information through various senses have been created. In particular, Magic Leap, an augmented reality company in the United States, implements animals that are difficult to actually encounter through augmented reality, so that you can experience things that are difficult to experience in school.

In addition, technologies for viewing the contents of books in three dimensions through displays are being used in various educational ways.

And according to the domestic Ministry of Education, it is planned to distribute realistic digital textbooks using AR technology in Korea soon, and digital textbooks have already started to be distributed to elementary, middle and high schools in 2015.

As such, vivid field education is currently provided through educational contents incorporating augmented reality (AR), and accordingly, the demand for various services is further increasing.

The present invention provides an information retrieval apparatus and method for providing augmented reality and complex reality (AR/MR) content and a web search function capable of learning about various objects in the real world through a user terminal in order to solve such demands. It aims to provide.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention that are not mentioned can be understood by the following description, and will be more clearly understood by examples of the present invention. In addition, it will be easily understood that the objects and advantages of the present invention can be realized by the means shown in the claims and combinations thereof.

In order to achieve this object, the information retrieval apparatus using augmented reality and complex reality of the present invention is an input unit that receives information on a selected object among images captured through a camera of a user terminal and objects recognized on a screen of the user terminal. ; An image extraction unit for extracting an image input to the input unit; An image analysis unit that analyzes the images extracted by the image extraction unit in chronological order and recognizes an object of image content; An object generator that generates one class content including characteristics and properties of an object based on the object defined by the image analysis unit using JSON (Jave Script Object Notation); An object extracting unit for extracting characteristic and attribute information of an object of the selected class content from among the class content displayed on the user terminal screen; An AR/MR content detector configured to search for AR/MR educational content previously stored in an AR/MR content database based on the object information extracted by the object extractor to detect AR/MR educational content linked to the object; And a search information generation unit for generating information content by using JSON (Jave Script Object Notation) in the AR/MR education content detected by the AR/MR content detection unit.

In addition, the information retrieval apparatus using the augmented reality and the complex reality includes: a keyword extracting unit for extracting a keyword associated with an object based on object information extracted from the object extracting unit; A web search processing unit that searches for information related to a keyword among information accessible through the web by performing a web search of a legacy system through the keyword extracted by the keyword extraction unit; And a result list generating unit for generating a web search result list by sorting the information searched by the web search processing unit into at least one of time, association, type, and type, wherein the search information generation unit Information content is generated by using JSON (Jave Script Object Notation) in the generated web search result list.

In addition, the image analysis unit divides the real-time image at regular intervals in chronological order or divides it by a directly designated distance value, and defines at least one object in the divided image.

In addition, the search information generation unit generates one class content including characteristics and properties of each object defined by using JSON (Jave Script Object Notation).

In addition, the search information generation unit generates an AR/MR education content in which the detected AR/MR education content and a real-time image extracted from the image extraction unit are combined.

In order to achieve such an object, the information retrieval method using augmented reality and complex reality of the present invention includes (A) a machine learning technique using an open source TensorFlow Object Detection API. Storing the AR/MR education content searched using data provided by TensorFlow through the AR/MR content database (DB); (B) When an image captured through the camera of the user terminal is input, the input image is analyzed in chronological order through the image analysis unit, and one class content including the characteristics and properties of the object is generated through the object generation unit. step; (C) when one of the class contents displayed on the user terminal screen is selected, extracting characteristic and attribute information of the object of the selected class contents through an object extraction unit; (D) searching for AR/MR educational content previously stored in an AR/MR content database based on the extracted object information by an AR/MR content detector, and detecting AR/MR educational content linked to the object; And (E) generating information content using JSON (Jave Script Object Notation) in the detected AR/MR education content in the search information generating unit and providing the information content to the user terminal.

In addition, the analysis of the image input in step (B) includes the steps of dividing the real-time image at regular intervals in chronological order or by a directly designated distance value, and defining at least one object in the divided image. Include.

In addition, the generation of class content in step (B) uses JSON (Jave Script Object Notation) to generate one class content including the properties and properties of each object defined above.

In addition, the information retrieval method using the augmented reality and the complex reality includes: extracting a keyword associated with an object based on the extracted object information by a keyword extraction unit; Performing a web search of a legacy system through the extracted keyword in a web search processing unit, and searching for information related to the keyword among information accessible through the web; Generating a web search result list by sorting the information searched through the result list generation unit into at least one of time, association, format, and type; And generating information content by using JSON (Jave Script Object Notation) in the generated web search result list by the search information generating unit and providing the information to the user terminal.

In addition, the step (E) includes generating AR/MR education contents in which the detected AR/MR education contents and real-time images extracted from the image extraction unit are combined through a search information generation unit.

According to the present invention as described above, the information retrieval apparatus and method using augmented reality and complex reality of the present invention is an augmented reality and complex reality (AR/MR) capable of learning about various objects in the real world through a user terminal. By providing content and web search functions, it is possible to provide a new learning experience that is realistic and immersive.

In addition, it is possible to vividly experience difficult things that are difficult to actually encounter without spatial and temporal constraints, so that the search for information desired by the user can be provided more efficiently.

In addition to the above-described effects, specific effects of the present invention will be described together while describing specific details for carrying out the present invention.

1 is a block diagram showing an information retrieval apparatus using augmented reality and complex reality according to an embodiment of the present invention.
2 is a block diagram showing a detailed configuration of the information search server of FIG.
3 is a flow chart for explaining an information search method using augmented reality and complex reality according to an embodiment of the present invention
4 is an exemplary embodiment showing that an information search result provided from an information search server is displayed on a screen of a user terminal.

The above-described objects, features, and advantages will be described later in detail with reference to the accompanying drawings, and accordingly, one of ordinary skill in the art to which the present invention pertains will be able to easily implement the technical idea of the present invention. In describing the present invention, if it is determined that a detailed description of known technologies related to the present invention may unnecessarily obscure the subject matter of the present invention, a detailed description will be omitted. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to indicate the same or similar elements.

Hereinafter, an information retrieval apparatus and method using augmented reality and complex reality according to an exemplary embodiment of the present invention will be described in detail with reference to the drawings.

1 is a block diagram showing an information retrieval apparatus using augmented reality and complex reality according to an embodiment of the present invention.

As shown in FIG. 1, the information retrieval apparatus includes a user terminal 100 that checks real-time image transmission for information retrieval and AR/MR education content search results, a legacy system 200 in which a data source is stored, and It includes an information retrieval server 300 for processing a web search of the legacy system 200 through a search for related AR/MR education content and related keywords by analyzing a real-time image received from the terminal 100.

The user terminal 100 transmits a real-time image (streaming image) captured through a camera to the information search server 300, and receives an information search result based on the analysis result of the real-time image from the information search server 300. It is called. For example, the user terminal 100 may correspond to a portable terminal such as a smart phone, a tablet PC, and a PDA. However, the user terminal 100 is not limited thereto, and all devices capable of accessing the information retrieval apparatus 200 through wired/wireless communication may be included.

The legacy system 200 refers to a data distribution storage environment in which a data source to be searched for information is stored, and includes, for example, a server storing and managing various information accessible through the web as a data source.

In addition, the information search server 300 analyzes the real-time image input from the user, searches for related AR/MR education contents based on the result of the analyzed real-time image, and searches the web through related keywords in the legacy system 200. This refers to a processing device, for example, a web server.

FIG. 2 is a block diagram showing a detailed configuration of the information search server of FIG. 1. At this time, the information retrieval server 300 shown in FIG. 2 is according to an embodiment, and its components are not limited to the embodiment shown in FIG. 2, and some components are added, changed, or deleted as necessary. Can be. And Figure 4 is an embodiment showing that the information search result provided from the information search server is displayed on the screen of the user terminal 100.

As shown in FIG. 2, the information search server 300 includes an input unit 301, an image extraction unit 302, an image analysis unit 303, an object generation unit 304, an object extraction unit 305, and AR. /MR content detection unit 306, keyword extraction unit 307, web search processing unit 308, result list generation unit 309, search information generation unit 310, AR/MR content database (DB) 311 Include.

The input unit 301 receives a real-time image (streaming image) captured by the camera of the user terminal 100 and information on an object selected by the user among objects recognized on the screen of the user terminal 100.

The image extracting unit 302 extracts a real-time image input to the input unit 301.

The image analysis unit 303 analyzes the real-time image extracted by the image extraction unit 302 in chronological order, and recognizes an object of image content. That is, the image analysis unit 303 divides the real-time image at regular intervals in chronological order or divides it into a directly designated distance value. In addition, at least one object is defined in the divided image.

The object generation unit 304 generates one class content including characteristics and properties of an object based on the object defined by the image analysis unit 303 by using JSON (Jave Script Object Notation). For example, the characteristics of an object called a student include the coordinates of the object, the size of a box (x,y value), and ID, and the properties of an object called a student include a student number, name, grade, and place of residence. And the class content refers to the content implemented in the text (name) of a student including these characteristics.

The class content generated by the object creation unit 304 is delivered to the user terminal 100 as shown in FIG. 4(b), and the class content 10 in which the object name is described by being boxed on the recognized object. Is displayed on the screen of the user terminal 100.

The object extracting unit 305 extracts characteristic and attribute information (hereinafter referred to as “object information”) of an object of the selected class content 10 from among the class content 10 displayed on the screen of the user terminal 100.

The AR/MR content detection unit 306 searches for AR/MR education content previously stored in the AR/MR content database (DB) 311 based on the object information extracted from the object extraction unit 305 to detect the object and Detects linked AR/MR education content. At this time, AR/MR education contents are augmented reality and mixed reality (AR/MR) contents made to learn about various objects in the real world, and machines using the open source TensorFlow Object Detection API. The data provided by TensorFlow is used through machine learning techniques. The AR/MR content database (DB) 311 updates the stored AR/MR education content through continuous training.

The keyword extraction unit 307 extracts a keyword associated with an object based on the object information extracted by the object extraction unit 305. In this case, the extracted keyword may be extracted from the attribute information on the object, but is not limited thereto, and a keyword related to the attribute information may be extracted.

The web search processing unit 308 searches for information related to the keyword among various pieces of information accessible through the web by performing a web search of the legacy system 200 through the keyword extracted by the keyword extracting unit 307.

The result list generation unit 309 generates a web search result list by sorting the information searched by the web search processing unit 308 by time, relevance, type, and type.

The search information generation unit 310 utilizes JSON (Jave Script Object Notation) in the web search result list generated by the AR/MR education content and result list generation unit 309 detected by the AR/MR content detection unit 306 To create each information content. The search information generation unit 310 may generate AR/MR education content in which the detected AR/MR education content and the real-time image extracted by the image extraction unit 302 are combined.

The information content generated by the search information generation unit 310 is transmitted to the user terminal 100 and displayed on the screen, as shown in FIG. 4C. Accordingly, when the user selects the information content displayed on the user terminal 100, augmented reality and mixed reality (AR/MR) content and a web search function capable of learning about various objects in the real world can be provided. As a result, you can experience new learning that is realistic and immersive.

In addition, the user can select and check a necessary part from the list displayed on the user terminal 100.

The operation of the information retrieval apparatus using augmented reality and complex reality according to the present invention configured as described above will be described in detail with reference to the accompanying drawings. The same reference numerals as in FIGS. 1 and 2 refer to the same members performing the same functions.

3 is a flowchart illustrating an information search method using augmented reality and complex reality according to an embodiment of the present invention.

Referring to FIG. 3, first, the information retrieval server 300 uses data provided by TensorFlow through a machine learning technique using an open source TensorFlow-based object detection API. The searched AR/MR education content is stored in the AR/MR content database (DB) 311 (S10). However, the AR/MR education content stored in the AR/MR content database (DB) 311 is not limited thereto, and may be produced and stored in the information search server 300.

In addition, AR/MR education contents stored in the AR/MR contents database (DB) 311 are updated through continuous training.

Subsequently, when a real-time image (streaming image) photographed through the camera of the user terminal 100 is input, the real-time image input through the image analysis unit 303 is analyzed in chronological order, and through the object creation unit 304 One class content including the properties and properties of the object is generated (S20).

In this case, the method of analyzing the image is to first divide the real-time image at regular intervals in chronological order, or divide it by a directly designated distance value. In addition, at least one object is defined in the divided image.

And the method of creating an object is to use JSON (Jave Script Object Notation) to create one class content including the properties and properties of the object based on each defined object. For example, the characteristics of an object called a student include the coordinates of the object, the size of a box (x,y value), and ID, and the properties of an object called a student include a student number, name, grade, and place of residence. And the class content refers to the content implemented in the text (name) of a student including these characteristics.

In addition, the generated class content is delivered to the user terminal 100 as shown in FIG. 4(b), and is boxed on the recognized object, and the user terminal 100 is used as the class content 10 in which the object name is written. It is displayed on the screen (S30).

Thereafter, when one of the class contents 10 displayed on the screen of the user terminal 100 is selected, object information (characteristic and attribute information) on the object of the selected class contents 10 is extracted through the object extraction unit 305 Information on the object is recognized (S40).

And based on the object information extracted by the AR/MR content detection unit 306, AR/MR education linked to the object is searched for AR/MR education content previously stored in the AR/MR content database (DB) 311 Detect content.

In addition, the keyword extraction unit 307 extracts a keyword related to the object based on the extracted object information, and the web search processing unit 308 performs a web search of the legacy system 200 through the extracted keyword to search the web. Searches for information related to keywords among a variety of information that can be accessed through. Then, the information searched through the result list generation unit 309 is sorted by time, association, type, type, etc. to generate a web search result list.

Subsequently, the search information generation unit 310 generates each information content using JSON (Jave Script Object Notation) in the detected AR/MR education content and the generated web search result list and provides it to the user terminal 100 Do (S50).

The generated information content is transmitted to the user terminal 100 and displayed on the screen as shown in FIG. 4(c).

Accordingly, when the user selects the information content displayed on the user terminal 100, augmented reality and mixed reality (AR/MR) content and a web search function capable of learning about various objects in the real world can be provided. As a result, you can experience new learning that is realistic and immersive.

In addition, the user can select and check a necessary part from the list displayed on the user terminal 100.

The above-described present invention is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those of ordinary skill in the technical field to which the present invention belongs. Is not limited by

100: user terminal 200: legacy system
300: information search server 301: input unit
302: image extraction unit 303: image analysis unit
304: object creation unit 305: object extraction unit
306: AR/MR content detection unit 307: keyword extraction unit
308: web search processing unit 309: result list generation unit
310: search information generation unit 311: AR/MR content DB

Claims (10)

An input unit for receiving information on a selected object among images captured through a camera of the user terminal and objects recognized on a screen of the user terminal;
An image extraction unit for extracting an image input to the input unit;
An image analysis unit that analyzes the images extracted by the image extraction unit in chronological order and recognizes an object of image content;
An object generator that generates one class content including characteristics and properties of an object based on the object defined by the image analysis unit using JSON (Jave Script Object Notation);
An object extracting unit for extracting characteristic and attribute information of an object of the selected class content from among the class content displayed on the user terminal screen;
An AR/MR content detector configured to search for AR/MR educational content previously stored in an AR/MR content database based on the object information extracted by the object extractor to detect AR/MR educational content linked to the object; And
An information retrieval device using augmented reality and complex reality comprising a search information generating unit that generates information content by using JSON (Jave Script Object Notation) in the AR/MR education content detected by the AR/MR content detection unit.
The method of claim 1,
The information search device using the augmented reality and the complex reality
A keyword extraction unit extracting a keyword associated with an object based on the object information extracted by the object extraction unit;
A web search processing unit that searches for information related to a keyword among information accessible through the web by performing a web search of a legacy system through the keyword extracted by the keyword extraction unit; And
Further comprising a result list generator for generating a web search result list by sorting the information searched by the web search processing unit into at least one of time, association, format, and type,
The search information generation unit is an information search device using augmented reality and complex reality for generating information content by using JSON (Jave Script Object Notation) in a web search result list generated by the result list generation unit.
The method of claim 1,
The image analysis unit divides the real-time image at regular intervals in chronological order or by a directly designated distance value, and defines at least one object from the divided image.
The method of claim 3,
The search information generation unit
An information retrieval device using augmented reality and complex reality that generates one class content including characteristics and properties of each object defined by using JSON (Jave Script Object Notation).
The method of claim 1,
The search information generation unit
An information retrieval apparatus using augmented reality and complex reality to generate AR/MR education contents in which the detected AR/MR education contents and real-time images extracted from the image extracting unit are combined.
(A) Using the open source TensorFlow-based object detection API, the AR/MR education contents retrieved using data provided by TensorFlow through machine learning techniques are converted into an AR/MR contents database. Storing in (DB);
(B) When an image captured through the camera of the user terminal is input, the input image is analyzed in chronological order through the image analysis unit, and one class content including the characteristics and properties of the object is generated through the object generation unit. step;
(C) when one of the class contents displayed on the user terminal screen is selected, extracting characteristic and attribute information of the object of the selected class contents through an object extraction unit;
(D) searching for AR/MR educational content previously stored in an AR/MR content database based on the extracted object information by an AR/MR content detector, and detecting AR/MR educational content linked to the object; And
(E) Search information using augmented reality and complex reality including the step of generating information content using JSON (Jave Script Object Notation) in the detected AR/MR education content in the search information generation unit and providing it to a user terminal Way.
The method of claim 6,
Analysis of the image input in step (B)
Dividing the real-time video at regular intervals in chronological order or by a directly designated distance value, and
Information retrieval method using augmented reality and complex reality, comprising the step of defining at least one object in the divided image.
The method of claim 7,
The generation of class content in step (B) is an information retrieval method using augmented reality and complex reality in which one class content including characteristics and properties of each defined object is generated using JSON (Jave Script Object Notation). .
The method of claim 6,
The information search method using the augmented reality and the complex reality
Extracting a keyword associated with an object based on the extracted object information by a keyword extraction unit;
Performing a web search of a legacy system through the extracted keyword in a web search processing unit, and searching for information related to the keyword among information accessible through the web;
Generating a web search result list by sorting the information searched through the result list generator into at least one of time, association, format, and type; And
An information search method using augmented reality and complex reality, further comprising the step of generating information content using JSON (Jave Script Object Notation) in the generated web search result list by the search information generating unit and providing the information to a user terminal.
The method of claim 6,
Step (E)
Information retrieval method using augmented reality and complex reality, comprising the step of generating AR/MR educational content in which the detected AR/MR educational content and the real-time image extracted from the image extracting unit are combined through a search information generator.

Images