KR102416338B1 - Method for searching data using visual presenter - Google Patents

Abstract

The present invention is an information retrieval method using a real imager. In particular, the real imager that converts a real photographed through a camera into an image is equipped with an AR function to select a picture with a selection gesture of a recognized object in the real world, and based on the selected picture It relates to an information retrieval method using a real imager capable of retrieving information.

Description

Information search method using a real imager {METHOD FOR SEARCHING DATA USING VISUAL PRESENTER}

The present invention relates to an information retrieval method using a real imager that includes a real imager that converts a real object photographed by a camera into an image, and can search information using the real imager.

A real imager, widely used for academic activities, education, and conferences, takes a picture of the real thing with a camera, converts the captured real thing into an image, and displays it on a display device such as a monitor.

As such a real imager, those described in Korean Patent Application Laid-Open No. 10-2011-0040255 (hereinafter referred to as 'Patent Document 1') are known.

In the case of Patent Document 1, it is composed of a base, a photographing unit for photographing a target object, a support unit for supporting the photographing unit with respect to the base, and a rotation unit provided between the base and the support unit, and provided with a first joint portion and a second joint portion.

In Patent Document 1, the first joint part rotates about the first axis with respect to the base, and the second joint part rotates about the second axis perpendicular to the first axis in a state where the second joint part rotates about the first axis with respect to the second joint part. Thus, the position and angle for photographing the subject can be determined.

Recently, a real presenter may display an image captured by a camera provided in the real imager on a display in a classroom or a seminar room, etc., and may perform a function of showing an enlarged image. As described above, the recent real presenter displays a photographed image on a display, thereby enabling multiple people to share information displayed on the display, and is used as an information delivery medium in education and conferences.

However, the conventional real imager can display a photographed image on a display or enlarge the image to show it in detail. If additional information needs to be supplemented through a search for a video to be explained during a meeting, it is inconvenient to move to a PC that can be searched through a search engine and search for additional information.

In addition, in order to display and share the searched additional information on the display, the user must output it separately and shoot it with a real presenter, disconnect the display connected to the real presenter, and reconnect to the PC on which the search information is displayed. There is a problem in that training or meeting time delay is inevitable due to the sharing of search results for additional information on video.

In addition, if a user is inexperienced in searching through a search engine and there is a lot of noise in searching for additional information about an image, training or meeting time may be wasted unnecessarily, and this may disrupt the flow of training or meeting, resulting in audience participation There is a problem that the degree may be lowered.

Korean Patent Publication No. 10-2011-0040255

The present invention has been devised to solve the above problem, and it is an object of the present invention to provide an information retrieval method using a real presenter that allows a user to conveniently search for necessary information using the real presenter without having to move to a separate searchable device. The purpose.

An information retrieval method using a real imager according to one aspect of the present invention includes: a capturing step of capturing a real image through a camera provided in the real imager; a picture selection step of selecting only a picture without selecting a text from the actual image when the recognition object is performing a selection gesture on the physical image; and a retrieval step of retrieving information based on the picture in the retrieval DB.

In addition, the searching step is characterized in that it includes a comparison step of comparing the characteristic points of the picture in the image recognition DB.

In addition, in the comparison step, if the characteristic points of the picture of the picture are not in the image recognition DB, a registration step of registering only the characteristic points of the picture in the image recognition DB; and, in the comparison step, the characteristic points of the picture of the picture are performed. When the feature point is in the image recognition DB, it is characterized in that the search step is performed.

In addition, after the registration step, the search step is performed, and the search step performed after the registration step is performed by a search engine.

In addition, the image recognition DB and the search DB are characterized in that it is possible to automatically upload the DB of the user image recognition DB and the user search DB.

According to the information retrieval method using the real presenter according to the preferred embodiments of the present invention as described above, the following effects are obtained.

According to the present invention, when a user needs to search for additional information on an object to be explained while giving a lecture or presentation in an education or conference, information can be immediately retrieved through a gesture of selecting a search object in real life with a recognition object such as a user's finger. can

In addition, the searched information can be displayed in real time through the display.

Due to this, the user can easily search for additional information on the spot without having to move to a place to search for additional information, and can provide information to the audience by directly displaying the search results.

In addition, since the searched information can be displayed and shared on the display in real time, it is possible to omit cumbersome tasks such as separately outputting the search results and shooting with a real imager, or reconnecting the display with the PC displaying the search results. there is

In addition, by performing a gesture of selecting a picture through a recognized object in real time in real time to select a picture, and the selected picture is registered in the image recognition DB and the search DB, database construction can be made easily.

In addition, since pictures registered in the image recognition DB and the search DB can be automatically updated with the user image recognition DB and the user search DB, various information can be built without separately updating the user image recognition DB and the user search DB. , due to this, when the system performs the information retrieval method of the present invention, it is possible to provide more accurate and various information to the user.

1 is a flowchart of an information retrieval method using a real presenter according to a first preferred embodiment of the present invention.
2 is a flowchart of an information retrieval method using a real presenter according to a second preferred embodiment of the present invention.
3 is a flowchart of a picture selection step of an information retrieval method using a real imager according to the first and second preferred embodiments of the present invention;
4 is a diagram schematically illustrating a server system and a user system for performing an information retrieval method using a real presenter according to the first and second preferred embodiments of the present invention.
5 is a diagram exemplarily implementing an information retrieval method using a real presenter according to the first and second preferred embodiments of the present invention.

Prior to explaining the information retrieval method using the real presenter (hereinafter, 'information retrieval method of the present invention') according to the first and second preferred embodiments of the present invention, the present invention provides a method that is connected to the real presenter by wire or wirelessly. It may be executed by the system, and may be implemented as a program installed on a PC or the like. In addition, the system and program described above may be configured to include an image recognition DB and a search DB.

In addition, the 'real imager' mentioned below may be a real imager equipped with an AR function.

In addition, the 'system' mentioned below may collectively refer to a server system that enables the sharing of registered information and a user's individual user system, and may be a server system and a user system, respectively. When the system for performing the information retrieval method of the present invention is a server system, the image recognition DB and the search DB included in the system may be a server image recognition DB and a server search DB. In addition, when the system for carrying out the present invention is a user system, the image recognition DB and the search DB included in the system may be a user image recognition DB and a user search DB.

In addition, the 'image recognition DB' mentioned below may collectively refer to the user image recognition DB and the server image recognition DB, and may be each.

In addition, the 'search DB' mentioned below may collectively refer to a user search DB and a server search DB, and may be respectively.

In addition, the 'picture' mentioned below may collectively refer to the shape of an object or the face of a person, and may be each.

In addition, 'information' referred to below may include at least one or more of the picture selected in the picture selection step, the result image searched for by keyword, a page linked with a search URL, and an image linked with an image URL, but is not limited thereto. does not In addition, the information may collectively refer to a selected picture, a keyword search image, a search URL-linked page, an image URL-linked video, and the like, and may be each.

Hereinafter, information retrieval methods 10' and 10 using the real imager 3 according to the first and second preferred embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a flowchart of an information retrieval method 10' using a real imager 3 according to a first preferred embodiment of the present invention, and FIG. 2 is a real imager 3 according to a second preferred embodiment of the present invention. It is a flowchart of the information retrieval method 10' used, and FIG. 3 is a flowchart of the picture selection step of the information retrieval method 10', 10 using the real imager 3 according to the first and second preferred embodiments of the present invention. , FIG. 4 schematically shows a server system 2 and a user system 1 for performing information retrieval methods 10' and 10 using the real presenter 3 according to the first and second preferred embodiments of the present invention. FIG. 5 is a diagram exemplarily implementing information retrieval methods 10' and 10 using the real imager 3 according to the first and second preferred embodiments of the present invention.

First, an information retrieval method 10' using the real imager 3 according to a first preferred embodiment of the present invention will be described.

The information retrieval method 10' using the real presenter 3 according to the first preferred embodiment of the present invention may be selectively performed in the user system 1 or the server system 2, but the following preferred methods of the present invention The information retrieval method 10 ′ using the real presenter 3 according to the first embodiment will be described as being performed in the user system 1 .

When the information retrieval method 10' using the real presenter 3 according to the first preferred embodiment of the present invention is performed in the user system 1, the image recognition DB 100 and the retrieval DB 200, which will be described later, are It may be an image recognition DB (100a) and a user search DB (200a). In addition, when the information retrieval method 10' using the real imager 3 according to the first preferred embodiment of the present invention is performed in the server system 2, the image recognition DB 100 and the retrieval DB 200 are It may be an image recognition DB (100b) and a server search DB (200b).

As shown in FIG. 4 , the user system 1 may be configured to include a user image recognition DB 100a and a user search DB 200a.

The image recognition DB 100 may be a database pre-built as an image in the system. For example, when the real imager 3 is used in a school class, if the educational textbook is a real image 5, the image recognition DB 100 is the same as a database built in advance with images included in textbooks distributed for education nationwide. can

The image stored in the image recognition DB 100 may include metadata.

The metadata may include at least one of an image identification ID, a search keyword, a search URL, and an image URL, but is not limited thereto. In addition, in the case of the image identification ID, it may be automatically assigned by the system.

For example, one of the images stored in the image recognition DB 100 may be an image of General 'Sun-Shin Yi'. In this case, the image of Admiral Yi Sun-sin may include metadata such as Yi Sun-sin 001 (image identification ID), Yi Sun-sin, turtle ship (search keyword), search URL, and image URL, and may be stored.

Also, the image recognition DB 100 may include image feature data in which feature points and keywords of the image are paired. Due to this, the system can find the keyword by matching the feature points of the image to be searched. If there is a keyword, the system may provide the user with a link to Internet information of the keyword in the metadata.

The search DB 200 may be software installed in the system, or a search engine that helps to easily find data on the Internet.

In addition, the search DB 200 may provide information in connection with at least one of an image and metadata stored in the image recognition DB 100 , and may store at least one of the provided metadata.

As shown in Fig. 1, the information retrieval method 10' using the real imager 3 according to the first preferred embodiment of the present invention includes a capturing step (S10'), a picture selection step (S20'), and a searching step. (S30') may be configured.

In the information retrieval method 10 using the real presenter 3 according to the first preferred embodiment of the present invention, the capturing step (S10') may be performed by the real presenter 3 . In addition, since the real imager 3 is equipped with an AR function, the picture selection step (S20') can be performed in both the real imager 3 and the system. In addition, the search step (S30') may be performed in the system. As such, the information retrieval method 10' of the present invention may be performed in the real presenter 3 or the system or may be performed in the real presenter 3 and the system.

The real imager 3 shown in FIG. 5 is equipped with augmented reality, that is, an AR function, so that a three-dimensional virtual object may be superimposed on a real world image. AR is a technology that enhances understanding of the real world by synthesizing the real world environment and graphic virtual reality in real time based on reality.

The information retrieval method 10' using the real presenter 3 according to the first preferred embodiment of the present invention uses the real presenter 3 equipped with an AR function to search for information using the real presenter 3 through a system. By carrying out the method 10', it is possible to select a picture of the real image 5 in real time and retrieve information based thereon.

In the capturing step ( S10 ′), a process of capturing the real image 5 through the camera provided in the real imager 3 may be performed.

In the capturing step (S10'), a plurality of photos may be captured in order to clarify the outer border of the picture of the real image 5 . In this case, 4 or more to 5 or less pictures may be captured.

The system may perform a picture selection step (S20') of selecting only a picture without selecting text from the real image 5 when the recognition object 6 is performing a selection gesture on the real thing.

The picture selection step (S20') can be described in detail with reference to FIG. 3 .

As shown in FIG. 3 , in the picture selection step ( S20 ′), first, a process of extracting a picture area from a captured image may be performed.

In the process of extracting the picture area, the system may first remove the background on the captured image. The background on the captured image may be text (T).

Specifically, the captured image may be grayed out and binarized by the system. The system can extract outlines other than letters by applying a labeling algorithm that recognizes and detects only connected parts in a binary image as a single object.

The system can remove the text (T) and display the image extraction area window in the picture area. The picture extraction area window may be the same as a rectangular window, and may be displayed along the outer edge of the picture area to be extracted by recognizing the outer edge of the picture area.

The system can clearly recognize the boundary of the outer edge of the picture area using histogram-based edge enhancement technology. If the boundary of the outer border of the picture area is ambiguous, picture extraction may fail. The system of the information retrieval method 10' using the real imager 3 according to the first preferred embodiment of the present invention is a histogram-based edge in order to prevent a picture extraction failure and to more clearly recognize the outer edge of the picture area. Brightness changes can be emphasized using enhancement techniques.

The picture area extracted by the system may be all picture areas on the real image 5 captured and recognized by the camera of the real presenter 3 .

The system performs adaptive binarization on the picture region with enhanced outer borders to convert pixels with large brightness changes to white pixels, and uses the CCL algorithm to extract and label foreground pixels to identify objects with size and location information. have.

The system may filter the picture area in the captured image using the size and position information of the identified objects, and filter the objects to be extracted through the minimum and maximum size information of the picture area. The system can merge multiple filtered objects expressed in one picture area into one object.

On the other hand, the system can re-perform the capture step when the picture area extraction fails.

After performing the process of extracting the picture area, the system may perform the process of recognizing all the extracted picture areas as one picture area based on the picture extraction area window. For example, in the system, if a plurality of pictures exist in the real image 5 and a plurality of picture extraction area windows are displayed and overlap, if the pixel size of the top, bottom, left, and right of the overlapping picture area is 10 pixels or more, one picture area can be recognized.

The system may perform a process of correcting the rotation, sharpness and/or brightness of the extracted picture area. The system can use an adaptive histogram smoothing method to correct the image brightness to an optimal state. In addition, the system can secure the sharpness by a special kernel designed to secure the sharpness. In addition, the system can extract and correct the rotation amount by extracting the inclination of the outer edge of the extracted picture area.

When the recognition object 6 is superimposed on the corrected picture area, the system may perform a process of recognizing the recognition object 6 . Specifically, in the mode using the augmented reality function of the real presenter 3 , the system may perform a process of recognizing the recognition object 6 .

In this case, the recognition object 6 may be a user's finger or various objects such as a stylus or a wand, but in order to conveniently select a picture while viewing the real image 5 of the real presenter 3 It may be most desirable for the user's finger to be used as the recognition object 6 .

The process of recognizing the recognition object 6 may be performed when the background of the captured image is not changed. When a motion in which the background of the captured image is changed occurs, the system may return to the capturing step (S10') and perform the picture selection step (S20') again in order from the capturing step (S10').

The process of determining whether to change the background shown in FIG. 3 may be the same as the process of returning to the capturing step S10' due to the occurrence of movement of the real image shown in FIG. 1 .

3, the system does not change the background of the captured image, but the recognition object 6 superimposed on the picture area of the captured image is not recognized, and the recognition object 6 is not recognized. If a motion such as the selection gesture in (6) is not detected, the picture area can be extracted automatically.

In this case, the selection gesture may be a gesture in which the recognition object 6 stops for a certain period of time, a clicking action such as clicking a kind of mouse, or a closed-type selection action drawn along the border based on the outer border of the picture, etc. and is not limited thereto.

The automatically extracted picture area may be a corrected picture area, and the system may recognize the corrected picture area as a picture and perform a process of inputting metadata.

The system may perform a picture correction process for correcting the horizontality or brightness of the picture shown in FIG. 1, and may perform a search step (S30') for the picture.

Meanwhile, the system may recognize the recognition object 6 superimposed on the picture area of the captured image, and perform a process of detecting a movement such as a selection gesture of the recognition object 6 .

As shown in FIG. 3 , the system recognizes the recognition object 6 superimposed on the picture area, and when a movement such as a selection gesture of the recognition object 6 is detected, the system extracts the movement area and the skin color area and recognizes it The object area can be detected.

Since the recognition object area detection can be performed when both the motion area and the skin color area are extracted, it is preferable that the recognition object 6 is a finger.

The movement area may include a movement in which a finger, which is the recognition object 6, selects a picture, and a movement area of a finger shadow generated on an image captured by light.

When the system detects a motion region, it may detect all motion regions in the captured image. In this case, the movement of the actual finger selecting the picture and the shadow area may be detected.

The system may extract both the motion region and the skin color region in order to accurately recognize the gesture selected by the finger and accurately extract a picture of the location where the finger selects the gesture is performed. In this case, the system may classify pixels corresponding to the finger area and extract the area estimated as the finger using the skin color information based on the threshold value, Hue, Saturation, and Intensity. . The system can identify the extracted area as an object using the CCL algorithm. The system can find the center point of the object with the largest area and simplify the outlines using the Douglas-Peucker algorithm. The system can find the candidate positions of the fingertips in the simplified outline area using the Convexhull algorithm. Then, the system detects the position of the convex space and the empty space (specifically, the colored blobs and outlines divided in the picture), investigates the angle, and recognizes it as a finger.

If the skin color region is not extracted from the movement region, the system may not recognize the gesture of the recognition object 6 by determining that it is a shadow region rather than the recognition object 6 .

In this way, the system can perform accurate gesture recognition of the recognition object 6 by extracting the motion region and the skin color region.

When the recognition object 6 is not a finger, the system can recognize the gesture of the recognition object 6 by detecting the recognition object area, not the shadow area, with an appropriate process.

Through the picture selection step (S20') in which the above-described process is performed, the system can select only the picture without selecting the text from the actual image when the recognition object 6 is performing the selection gesture in the real world.

The system may perform a picture correction process of correcting the horizontality and brightness of the picture selected in the picture selection step (S20').

The horizontal correction of the picture correction process may be a rotation amount correction of the selected picture. The amount of rotation correction can be performed based on the outer edge of the picture, and when the picture is tilted in any one direction, it can be corrected so that the outside edge of the extracted picture is straight and parallel by correcting the rotation at a predetermined angle in the tilted direction. .

In addition, the system can perform brightness correction and sharpness correction.

The system may perform a search step (S30') of searching for information in the search DB 200 based on the picture selected in the picture selection step (S20').

The search step (S30') may include a comparison step of comparing the characteristic points of the picture selected in the picture selection step (S20') in the image recognition DB 100 .

Characteristic points unique to a picture are picture elements that can uniquely determine the picture, and include the outline shape, features, appearance, and color of the selected picture. For example, when the selected picture is a plant, the outer shape, color, etc. of the plant may be a characteristic characteristic of the picture. In addition, when the selected picture is a person, the features of the person's features, appearance, clothing color, etc. may be characteristic points of the picture.

In addition, the characteristic point unique to the picture may be an immutable characteristic including the size, position, and rotation of the selected picture.

The system may perform a comparison step of comparing the image stored in the image recognition DB 100 with the characteristic points of the selected picture.

The system can compare the picture-specific feature points of the picture selected in the comparison step in the image recognition DB.

As shown in Fig. 1, when the system does not have the characteristic points of the picture of the selected picture in the image recognition DB 100, the registration step of registering only the characteristic points of the picture of the selected picture in the image recognition DB 100 (S40') ) can be done.

Specifically, as shown in FIG. 1 , the system may perform a comparison step of comparing the picture-specific feature points of the picture selected in the image recognition DB 100 after performing the picture correction process of the selected picture. If the system does not have a picture-specific feature point of the picture selected in the image recognition DB 100 in the comparison step, the registration step (S40') of registering only the picture-specific characteristic point of the selected picture in the image recognition DB 100 can be performed. have. In this case, the selected picture may be registered by including a unique keyword and a related keyword. For this reason, the system may provide a unique keyword and a related keyword highly related to the picture selected to the user in the search step (S30') performed after the registration step (S40').

After performing the registration step (S40'), the system may perform a search step (S30').

The search step (S30') performed by the system after the registration step (S40') may be performed in a search DB such as a search engine.

The system may first execute the search engine in the search step (S30') performed after the registration step (S40'). The system may perform a process of searching for an image with a picture selected in the executed search engine. The system may display the retrieved image result with the selected picture.

Although not shown in FIG. 1 , when one of the displayed image results is selected, the system may display a window for selecting whether to store the selected page URL as a result display value of the selected picture in the search DB.

The system may perform the process of storing the result display value in the search DB when the selection to save the result display value of the selected picture is input in the search DB.

As such, in the information retrieval method 10' using the real imager 3 according to the first preferred embodiment of the present invention, when a picture not stored in the image recognition DB 100 is selected, the system automatically recognizes the image. A registration step (S40') of registering the selected picture in the DB 100 may be performed.

In addition, after the registration step (S40'), the system may perform a search through a search engine, and perform a process of storing the correct result display value so that it can be connected to a picture newly registered in the image recognition DB 100 . For this reason, even information not stored in the image recognition DB 100 can be conveniently searched.

In addition, the system automatically performs the process of registering the characteristic points of the picture in the image recognition DB 100, and can store the exact result display value of the selected picture in the search DB 200, so that the image recognition DB 100 automatically ) and the effect of updating the search DB 200 can be obtained.

On the other hand, the system may perform the search step (S30') in the search DB (200) when the characteristic point of the picture of the picture is in the image recognition DB (100) in the comparison step.

As shown in Figure 1, the system performs a comparison step of comparing the picture-specific characteristic points of the picture selected in the picture selection step (S20') in the image recognition DB 100. If there is a picture-specific characteristic point, A search step may be performed in the search DB 200 .

The system can match with the search DB 200 based on a suitable image compared with the picture-specific characteristic points of the picture selected in the picture selection step (S20') in the image recognition DB 100 and the above-described image-specific characteristic points.

The system can connect to the URL returned by the search DB 200 by matching with the search DB 200 with the above-described image and unique feature points of the image, and display the result through this.

When the information retrieval method 10' using the real presenter 3 according to the first preferred embodiment of the present invention is performed in the server system 2, the image recognition DB 100b and the retrieval DB of the server system 2 200b may automatically upload the DB of the user image recognition DB 100a and the user search DB 200a of the user system 1 .

Also, the image recognition DB 100a and the search DB 200a of the user system 1 may automatically upload the DBs of the image recognition DB 100b and the search DB 200b of the server system 2 .

Through this, the image recognition DB (100a, 100b) and the search DB (200a, 200b) of each system can be automatically updated.

As described above, the information retrieval method 10' using the real presenter 3 according to the first preferred embodiment of the present invention provides information in the retrieval step S30' based on the picture selected in the picture selection step S20'. However, by performing the comparison step, it is possible to provide accurate information from which noise has been removed through the characteristic points of the figure in the comparison step.

In the information retrieval method 10' using the real presenter 3 according to the first preferred embodiment of the present invention, when a user needs to search for additional information on a subject to be explained while conducting a lecture or presentation in an education or conference, etc. Information can be immediately retrieved through a gesture of selecting a search target in the real world with a recognition object such as a user's finger. This can be implemented through feature point-based image recognition technology that extracts and matches invariant features and descriptors of a search target.

For this reason, it is possible to easily search for additional information in the real presenter 3 without having to move a seat for searching for additional information, and to display the search results immediately and provide them to the audience.

In addition, since the image recognition DB (100a, 100b) and the search DB (200a, 200b) of each system can be updated automatically, the DB of each system can be built in various ways without performing a separate update, and this When performing the information retrieval method 10' using the physical imager 3 according to the first preferred embodiment of the present invention, it is possible to provide more accurate and various information.

Hereinafter, an information retrieval method 10 using the real imager 3 according to a second preferred embodiment of the present invention will be described with reference to FIGS. 2 to 5 .

The information retrieval method 10 using the real presenter 3 according to the second preferred embodiment of the present invention includes the information retrieval method 10 ′ and the capture using the real presenter 3 according to the first preferred embodiment of the present invention. The steps (S10), picture selection step (S20), and search step (S30) are the same, and the information search method 10 is performed using the same real imager 3, except for the capturing step (S10). The detailed process and steps of the picture selection step (S20) and the search step (S30) may be performed differently.

The information retrieval method 10 using the real presenter 3 according to the second preferred embodiment of the present invention is the same as the information retrieval method 10' using the real presenter 3 according to the first preferred embodiment of the present invention. A description of the configuration will be omitted. In addition, the following description of the same configuration as the real imager 3 and the real image 5 of the information retrieval method 10 using the real imager 3 according to the second preferred embodiment of the present invention will be described above. Although not described in the information retrieval method 10' using the real presenter 3 according to the first preferred embodiment of the present invention, information using the real presenter 3 according to the first and second preferred embodiments of the present invention Since it is the same configuration for performing the search methods 10' and 10, the same may be applied.

The information retrieval method 10 using the real imager 3 according to the second preferred embodiment of the present invention includes a capturing step (S10), a picture selection step (S20), a search step (S30), and a registration step (S40). , the search or registration selection step (S35) may be selectively performed. This may be performed by selecting and inputting a step to be performed into the system.

Hereinafter, the capture step (S10) and the search step (S30) after the picture selection step (S20) will be first described, and the capture step (S10) and the picture selection step (S20) followed by the registration step (S40) After explaining what is performed, it will be described that the capture step (S10) and the search or registration selection step (S35) are performed after the picture selection step (S20).

As shown in Fig. 2, the information retrieval method 10 using the real imager 3 according to the second preferred embodiment of the present invention includes a capture step (S10), a picture selection step (S20), and a search step ( S30), a registration step (S40), a search or registration selection step (S35) may be configured.

In the information retrieval method 10 using the real presenter 3 according to the second preferred embodiment of the present invention, the capturing step S10 may be performed in the real presenter 3 . In addition, since the real imager 3 is equipped with an AR function, the picture selection step S20 can be performed both in the real imager 3 and in the system. In addition, the search step (S30) may be performed in the system. As described above, in the information retrieval method 10 using the real presenter 3 according to the second preferred embodiment of the present invention, any one specific step can be performed only by the real presenter 3 or the system, and any other specific steps can be performed in both the real presenter 3 and the system.

In the capturing step (S10), a process of capturing the real image 5 through the camera provided in the real imager 3 is performed.

Thereafter, when the recognition object 6 is performing a selection gesture on the physical image, a picture selection step S20 of selecting only a picture without selecting text in the physical image 5 is performed.

In the picture selection step (S20), an extraction step (S20a) of extracting a keyword related to a picture from the image recognition DB 100 may be performed. The extracted keyword may be utilized to search for an image in a search step (S30) to be described later.

In this case, the image recognition DB 100 may be a database built in advance as an image in the system. For example, when the real imager 3 is used in a school class, if the educational textbook is a real image 5, the image recognition DB 100 is the same as a database built in advance with images included in textbooks distributed for education nationwide. can

The image stored in the image recognition DB 100 may include metadata.

The metadata may include at least one of an image identification ID, a search keyword, a search URL, and an image URL, but is not limited thereto. In addition, in the case of the image identification ID, it may be automatically assigned by the system.

For example, one of the images stored in the image recognition DB 100 is the image of General 'Sun-Shin Yi'. In this case, the image of Admiral Yi Sun-sin may include metadata such as Yi Sun-sin 001 (image identification ID), Yi Sun-sin, turtle ship (search keyword), search URL, and image URL, and may be stored.

In addition, in the picture selection step ( S20 ), a search URL matching the picture may be provided from the image recognition DB 100 . The provided search URL may be used to search for information in a search step (S30) to be described later.

In this way, the system extracts only keywords related to the picture selected in the picture selection step (S20) through the image recognition DB 100, and receives only the search URL related to the selected picture, so that the picture selected from the search DB 200 to be described later is provided. When searching for information based on this, accurate information with noise removed can be retrieved.

For example, in a class for students, a method of searching for information using the real presenter 3 equipped with an AR function may be performed. In this case, if a search result including noise is displayed on the display 4 , the class concentration of elementary, middle, and high school students in the period with a somewhat distracted concentration may drop.

However, in the information retrieval method 10 using the real imager 3 according to the second preferred embodiment of the present invention, only keywords related to the selected picture are extracted through the image recognition DB 100, and only the search URL related to the selected picture is extracted. can be provided

For this reason, noise is removed through the image recognition DB 100 in the search step (S30) to be described later, and the search process can be performed using only keywords or search URLs related to the selected picture, so that only accurate search results can be searched. there will be

In addition, since the system can display only accurate search results on the display 4, it is possible to maintain the flow and concentration of the class steadily in the information search process, and to further improve the quality of the class by providing various and accurate information. have.

After the picture selection step (S20) is performed, a search step (S30) of searching for information based on the picture selected in the search DB 200 may be performed.

The information searched based on the selected picture is a keyword extracted in the extraction step (S20a) of the picture selection step (S20), and may be a result image searched through the search DB 200 in the search step (S30), and a picture in the image recognition DB It may be a page linked with a search URL that matches the , and may be a video linked with an image URL.

As shown in FIG. 2 , the information retrieval method 10 using the real presenter 3 according to the second preferred embodiment of the present invention may be performed as follows. Since the information retrieval method 10 using the real presenter 3 according to the second preferred embodiment of the present invention uses the real presenter 3 equipped with an AR function, the system follows the search process of the flowchart of FIG. The information retrieval method 10 using the imager 3 can be performed.

When the operating state of the real imager 3 is turned on, the real imager 3 performs a capturing step S10 of capturing the real image 5 .

In the capturing step ( S10 ), the real image 5 may be captured through a camera provided in the real imager 3 . In this case, the capture function may provide a function to the camera and may be built in an appropriate position of the real imager 3 .

After the capture step S10 is performed, the picture selection step S20 in which the recognition object 6 performs a selection gesture on the real object to select a picture is performed. In this case, performing the selection gesture on the real object may be the same as performing the selection gesture on the real image 5 .

In the picture selection step (S20), when the recognition object 6 is performing the selection gesture on the physical image, only the picture may be selected without selecting the text from the actual image 5, and the selected picture may be extracted.

In the picture selection step S20, only the picture is selected by the recognition object 6 performing a selection gesture in the real world, and the process of extracting the selected picture can be described with reference to FIG. 3 .

As shown in FIG. 3 , a captured image may be generated while the real image 5 is captured. The captured image may be captured in the form of a single page, like the real image 5 shown in FIGS. 4 and 5 .

The system may remove the background of the captured image, and detect whether the background of the captured image is changed. The detection function of whether the background of the captured image is changed may have a detection function suitable for the real presenter 3 may be built-in, and a function may be provided to the camera.

The system detects whether the background of the captured image is changed, and when the background of the captured image is changed, it is determined that the arrangement of the real image 5 on the real imager 3 is not completed, and the real image 5 again You can return to the process of creating a captured image by capturing

For example, like the real image 5 shown in FIGS. 4 and 5 , the form of one page of a textbook including the image of Admiral Yi Sun-sin is captured. Then the system can remove the text background except for Admiral Yi Sun-sin's image.

When the text background is continuously changed due to the arrangement preparation of the real image 5 , the system may determine that the arrangement of the real image 5 is not completed, and return to the process of capturing the textbook.

Meanwhile, when the background of the captured image does not change, the system may perform a process of detecting motion in the captured image.

The system can automatically extract the picture area when no movement of the object is detected.

When there are multiple picture areas in the captured image, the system can automatically extract a picture area close to the center of the captured image based on the captured image. Therefore, when there are several picture areas in the real image 5 to be captured, it may be preferable to arrange the drawing area to be automatically extracted from the real image 5 so that it can come to the center of the captured image.

In addition, when there is no border in the picture area, the system can extract the picture area by recognizing a picture at a close distance as one picture.

In addition, when there are multiple picture areas including a plurality of people in the captured image, the system can extract only the picture area including the person located in the center or close to the center of the captured image based on the captured image. .

The system can correct the rotation amount of the extracted picture. Rotation amount correction can be performed based on the outer edge of the picture. In the rotation amount correction, when the extracted picture is tilted in any one direction, it is possible to correct the rotation of the extracted picture by a predetermined angle in the tilted direction so that the outer edge of the extracted picture is straight and parallel. In addition, the system can also correct the brightness and sharpness of the extracted picture along with the rotation correction.

Also, when a face is included in the captured image, the system may extract a face by selecting a face detection function. Such a function may be appropriately provided to the real imager 3 and the system.

For example, like the real image 5 shown in FIGS. 4 and 5 , the form of one page of a textbook including the image of Admiral Yi Sun-sin is captured. In this case, a face may be extracted by selecting a face detection function.

The system may extract a picture through a preset size parameter.

The system may recognize the picture of the extracted picture area. Then, the system can perform the process of inputting metadata into the recognized picture, and perform the extraction step (S20a) of extracting the recognized picture and extracting the keyword from the image recognition DB 100 (S20a), which will be described later as a keyword A search step (S30) to be performed may be performed.

When the system detects motion in the captured image, it can detect the recognition object region by extracting the motion region and the skin color region.

Since the detection of the recognition object area can be detected when both the movement area and the skin color area are extracted, it may be a process that can be preferably performed when the recognition object 6 is a finger.

The movement area shown in FIG. 3 may include a movement in which the finger, which is the recognition object 6, selects a picture, and a movement area of a finger shadow generated on an image captured by light.

When the system detects a motion region, it may detect all motion regions in the captured image. In this case, the movement of the actual finger selecting the picture and the shadow area may be detected.

The system may extract both the motion region and the skin color region in order to accurately recognize the gesture selected by the finger and accurately extract a picture of the location where the finger selects the gesture is performed. If the skin color area is not extracted from the movement area, the system may not recognize the recognition object 6 as a gesture by determining that it is a shadow area rather than the recognition object 6 .

In this way, the system can perform accurate gesture recognition of the recognition object 6 by extracting the motion region and the skin color region.

If the recognition object 6 is not a finger, the system can recognize the gesture of the recognition object 6 by detecting the recognition object area, not the shadow area, by a suitable process.

Thereafter, the system may select a picture by recognizing the gesture of the recognition object 6 , extract the selected picture, and perform an extraction step ( S20a ) and a search step ( S30 ) to be described later.

After the selected picture is extracted, the system may perform a process of asking whether to proceed with the search with the extracted picture, or may automatically perform sequential steps and procedures.

Although not shown in FIG. 2 , after the selected picture is extracted, it is transmitted to the image recognition DB 100 to perform an image matching process.

The image recognition DB 100 may perform an image matching process with the extracted picture, and perform an extraction step (S20a) of extracting a keyword related to the picture. In this case, the keyword may be one of metadata included in the image stored in the image recognition DB 100 .

The system may transmit the keyword extracted from the image recognition DB 100 to the search DB 200 and search information in the search DB 200 with the keyword.

In other words, the system may extract a keyword based on a picture selected in the real world, and search for information using the extracted keyword.

The system may display the search results on a display 4 such as a monitor.

In addition, although not shown in FIG. 2 , after the selected picture is extracted, the image recognition DB 100 may match the image with the extracted picture and provide a search URL matching the picture. The system may search for information with the provided search URL.

As shown in FIG. 5 , when information is searched for by the search URL, the system may display a page linked to the search URL on the display 4 .

Also, after performing the information retrieval method 10 based on one selected picture, the system may perform the information retrieval method 10 based on another picture in the same real image.

Therefore, after displaying the result, if the operating state of the real imager 3 is not in the OFF state, the system determines that additional information search will be performed, and the recognition object 6 performs a selection gesture on the real image to select a picture It can return to the picture selection step (S20).

When the operating state of the real presenter 3 is not turned OFF, the system may repeatedly perform this process.

The repetitively performed process is performed after the capture step (S10), the picture selection step (S20), the search step (S30), the result display step, and then returns to the picture selection step (S20), and the picture selection step (S20). ), the search step (S30), and the result display process may be performed.

As shown in Fig. 4, when the information retrieval method 10 using the real imager 3 according to the second preferred embodiment of the present invention is performed in the user system 1, the picture selection step (S20) and The image recognition DB and the search DB in the search step (S30) may be a user image recognition DB (100a) and a user search DB (200a). In addition, when the information retrieval method 10 using the real presenter 3 according to the second preferred embodiment of the present invention is performed in the server system 2, the image in the picture selection step S20 and the retrieval step S30 The recognition DB 100 and the search DB 200 may be a server image recognition DB 100b and a server search DB 200b.

As such, the information retrieval method 10 using the real imager 3 according to the second preferred embodiment of the present invention recognizes the real image 5 when the recognition object 6 performs a selection gesture on the real world. By synthesizing the selection gesture of the object 6 into virtual reality in the form of a graphic in real time, a picture can be selected and information can be searched based on this. For this reason, when it is necessary to search for additional information on the real image 5 explained in an education or a meeting, it is possible to easily search for information through a gesture of selecting a target to be searched with the recognition object 6 in the real world.

In addition, since the searched information can be displayed and shared on the display 4 in real time, the search results can be output separately and photographed with the real presenter 3, or the PC on which the search results are displayed and the display 4 can be reconnected, etc. It has the effect of omitting the cumbersome work of

In the information retrieval method 10 using the real imager 3 according to the second preferred embodiment of the present invention, when the information retrieval method 10 is performed in the system, the information retrieval method 10 goes through the steps of the information retrieval method 10 . A keyword extracted from the image recognition DB 100 or a search URL may be input as related metadata in the picture that is the basis of the search, and may be registered in the image recognition DB 100 and the search DB 200 .

In the information retrieval method 10 using the real imager 3 according to the second preferred embodiment of the present invention, the capturing step (S10) and the registering step (S40) after the picture selection step (S20) can be performed.

In the capturing step (S10), a process of capturing the real image 5 through the camera provided in the real imager 3 is performed.

Thereafter, when the recognition object 6 is performing a selection gesture on the physical image, a picture selection step S20 of selecting only a picture without selecting text in the physical image 5 is performed.

Thereafter, a registration step (S40) of registering the selected picture in the image recognition DB 100 and the search DB 200 may be performed. Although not shown in FIG. 2 , the system may perform a process of searching based on the registered picture after registering the picture.

As shown in FIG. 2 , when the registration step S40 ′ is performed, the system may perform the registration process according to the flowchart of FIG. 2 .

When the operating state of the real imager 3 is turned on, the real imager 3 performs a capturing step S10 of capturing the real image 5 .

After the capture step S10 is performed, the picture selection step S20 in which the recognition object 6 performs a selection gesture on the real object to select a picture is performed. In this case, performing the selection gesture on the real object may be the same as performing the selection gesture on the real image 5 .

In the picture selection step (S20), when the recognition object 6 is performing the selection gesture on the physical image, only the picture may be selected without selecting the text from the actual image 5, and the selected picture may be extracted.

In the picture selection step (S20), only the picture is selected by the recognition object 6 performing a selection gesture in the real world, and the process of extracting the selected picture may be referred to FIG. You can refer to the description in the information retrieval method using (3).

The system may perform the process of inputting metadata into the selected picture.

A picture into which metadata is input may be registered in the image recognition DB 100 and the search DB 200 .

The pictures registered in the image recognition DB 100 and the search DB 200 may be updated with the user image recognition DB 100a and the user search DB 200a of the user system 1 .

In this case, when the information retrieval method 10 is performed in the server system 2, the user image recognition DB 100a and the user search DB 200a may be updated, and the information retrieval method ( When 10) is performed, the server image recognition DB 100b and the server search DB 200b may be updated.

Such an update can be made automatically, and because of this, information through the server system 2 is automatically updated in addition to the information registered by the user in the user system 1, so that the system can provide more accurate and diverse information. there will be

In addition, after performing the registration step (S40') based on one selected picture, the system may perform the registration step based on another picture in the same real image.

Therefore, after the system performs the process of updating the registered picture to the user image recognition DB 100a and the user search DB 200a, if the operating state of the real presenter 3 is not OFF, the registration of additional information is performed. It is determined to be performed, and the recognition object 6 may return to the picture selection step (S20) of selecting a picture by performing a selection gesture on the real object.

The system may repeatedly perform this process until the operating state of the real presenter 3 is turned OFF.

The process performed repeatedly may be a process in which the registered picture is updated in the user image recognition DB 100a and the user search DB 200a after the capture step (S10), the picture selection step (S20), and the registration step (S40). have.

The information retrieval method 10 using the real imager 3 according to the second preferred embodiment of the present invention includes a capturing step (S10) and a picture selecting step (S20), followed by a search or registration selection step (S35). can

As shown in FIG. 1 , in the capturing step S10 , a process of capturing the real image 5 through the camera provided in the real imager 3 is performed.

Thereafter, when the recognition object 6 is performing a selection gesture on the physical image, a picture selection step S20 of selecting only a picture without selecting text in the physical image 5 is performed.

In this case, the recognition object 6 may most preferably be the user's finger.

Thereafter, a search or registration selection step (S35) of selecting whether to search or register a picture may be performed.

When the search is selected in the search or registration selection step (S35), the search step (S30) of searching for information based on the picture in the search DB 200 may be performed.

The system searches for the information retrieval method 10 using the real presenter 3 according to the second preferred embodiment of the present invention when the retrieval is selected in the retrieval or registration selection step S35 and the retrieval step S30 is performed. The search step (S30) may be performed in the same manner as in the step (S30).

In addition, when registration is selected in the search or registration selection step (S35), the registration step (S40) of registering the picture in the image recognition DB 100 and the search DB 200 may be performed.

When registration is selected in the search or registration selection step (S35) and the registration step (S40) is performed, the system registers the information search method 10 using the real presenter 3 according to the second preferred embodiment of the present invention. The registration step (S40) may be performed in the same manner as in the step (S40) process.

The picture registered in the image recognition DB 100 and the search DB 200 in the registration step (S40) may be updated with the user image recognition DB 100a and the user search DB 200a, and such an update is performed automatically can be

As shown in FIG. 2 , the search or registration selection step S35 of the information search method 10 using the real presenter 3 according to the second preferred embodiment of the present invention may be performed as follows.

When the operating state of the real imager 3 is turned on, the real imager 3 performs a capturing step S10 of capturing the real image 5 .

After the capture step S10 is performed, the picture selection step S20 in which the recognition object 6 performs a selection gesture on the real object to select a picture is performed.

After the picture selected in the picture selection step S20 is extracted, the system may perform a search or registration selection step S35.

In this case, when a search is selected, a search step (S30) of searching for information based on a picture in the search DB 200 is performed, and when registration is selected, a registration step of registering a picture in the image recognition DB 100 (S40) ) can be performed.

The information retrieval methods 10' and 10 using the real presenter 3 according to the first and second preferred embodiments of the present invention allow the user to search for additional information on the subject to be explained while conducting a lecture or presentation in an education or conference. If necessary, information can be immediately retrieved through a gesture of selecting a search target in the real world with a recognition object 6 such as a user's finger.

In addition, the retrieved information may be displayed in real time through the display 4 .

Due to this, the user can easily search for additional information on the spot without having to move to a place to search for additional information, and can provide information to the audience by directly displaying the search results.

In addition, the information retrieval method (10', 10) of the present invention selects a picture by performing a gesture of selecting a picture through the recognition object 6 in real time in real time, and the selected picture is retrieved from the image recognition DB 100 and search By being registered in the DB 200, database construction can be made easily.

In addition, since pictures registered in the image recognition DB 100 and the search DB 200 can be automatically updated to the user image recognition DB 100a and the user search DB 200a, the user image recognition DB and the user search DB 200a ), a variety of information can be built without performing a separate update, so that when the system performs the information retrieval method (10', 10) of the present invention, more accurate and various information can be provided to the user It has the effect of being

As described above, although described with reference to preferred embodiments of the present invention, those skilled in the art can variously modify the present invention within the scope without departing from the spirit and scope of the present invention described in the claims below. Or it can be carried out by modification.

1: User system 2: Server system
3: Real presenter 4: Display
5: Physical image 6: Recognized object
T: text
100: image recognition DB 200: search DB
100a: User image recognition DB 100b: Server image recognition DB
200a: User search DB 200b: Server search DB
S10', S10: Capture step S20', S20: Picture selection step
S20a: extraction step S30', S30: search step
S40', S40: Registration step S35: Search or registration selection step
10', 10: information retrieval method of the present invention

Claims (5)

A capturing step of capturing a real image through a camera provided in the real imager;
a picture selection step of selecting only a picture without selecting text from the real image when the recognition object is performing a selection gesture on the real image; and
Including; a search step of searching for information based on the picture in the search DB;
The picture selection step is
Recognizing the recognition object superimposed on the picture area of the real image, and selecting the picture by recognizing the selection gesture when the same motion as the selection gesture of the recognition object is detected. Way. According to claim 1,
The search step is
An information retrieval method using a real imager, comprising a comparison step of comparing the characteristic points of the picture in the image recognition DB. 3. The method of claim 2,
In the comparison step, if there is no characteristic point of the picture in the image recognition DB, a registration step of registering only the characteristic point of the picture in the image recognition DB;
The information retrieval method using a real imager, characterized in that the retrieval step is performed when the characteristic point of the picture of the picture is in the image recognition DB in the comparison step. 4. The method of claim 3,
After the registration step, the search step is performed,
The information search method using a real imager, characterized in that the search step performed after the registration step is performed by a search engine. 3. The method of claim 2,
The image recognition DB and the search DB are information retrieval method using a real imager, characterized in that the DB of the user image recognition DB and the user search DB can be automatically uploaded.

Images