KR20150108575A - Apparatus identifying the object based on observation scope and method therefor, computer readable medium having computer program recorded therefor - Google Patents

Abstract

In the present invention, disclosed are an apparatus and a method for identifying an object based on an observation scope, and a recording medium recorded with a computer program. Accordingly, the present invention collects position information of eyes which look at content displayed on a display, sets a region of interest according to the detected position of the eyes, and can check an object in the region of interest. A quick and accurate search can be carried out by using the method, compared to an existing image search.

Description

[0001] The present invention relates to an object-based object identification apparatus and method, and a recording medium on which a computer program is recorded.

The present invention relates to an apparatus and method for identifying an object based on a viewing area and a recording medium on which a computer program is recorded. More particularly, the present invention relates to a method and apparatus for identifying and identifying objects by setting a region of interest And a recording medium on which a computer program is recorded.

As the demand for digital contents grows, a huge amount of multimedia contents (video, music, image, image, etc.) are generated, produced, distributed and serviced around us. Among them, digital images and images are explosively increasing in number with the spread of high-performance, portable digital cameras and falling prices of mass storage devices and portable storage media. Accordingly, there is a growing demand for a service apparatus, a user apparatus, and a method for efficiently searching for information included in such images and images, and for accurately identifying and classifying desired objects as others.

Accordingly, when objects are observed with a specific purpose, the degree of focus on the region-of-interest (ROI), which is an area of interest in the image or image, Techniques for detecting regions of interest are being utilized.

The eye tracking technique detects eye movements (eye line) through the front camera and the eye tracking sensor, and then performs a specific operation desired by the user according to the detected movement of the eyeball. For example, when the user looks at the screen to use the terminal, the user can automatically enter the sleep mode when the power is turned on or when the user eyes the screen. In addition, the user may blink the eyes or perform the scroll or page turn according to the left and right movements of the eyes.

However, the detection of the ROI through eye tracking requires identification of the object existing in the ROI and does not present the result of retrieving the image having high similarity. Therefore, a separate object identification request is required by the user.

An object of the present invention is to provide a viewing area-based object identifying apparatus and method for detecting an object quickly and accurately as compared with object detection for an entire image or image, by selecting an area of interest through eye tracking and detecting an object only for a selected area of interest , And a recording medium on which a computer program is recorded.

Another object of the present invention is to provide a method and apparatus for detecting an object for a region of interest, storing information about the object in the region of interest database, storing the corresponding web page information when the corresponding web page information exists, Based object identification apparatus and method, and a recording medium on which a computer program is recorded.

A viewing area-based object identification apparatus according to an exemplary embodiment of the present invention includes a line-of-sight corresponding area extractor for extracting a line-of-sight corresponding area by sensing a line of sight on a display screen on which content is displayed; A region of interest selector for selecting a region of interest based on the line of sight corresponding region; And a search unit for identifying an object in the ROI.

As an example related to the present invention, the present invention further includes a database for searching an image having high similarity based on the information of the identified object and providing the result.

As one example related to the present invention, the content includes text, photographs, moving pictures, and web pages.

A method of identifying an object based on a viewing area according to an exemplary embodiment of the present invention includes: a sight line checking step of accumulating position information of an observation image according to gaze angle information of a user gazing at contents displayed on a display; A ROI setting step of setting an ROI based on the accumulated position information; An object identification step of identifying the object identification information through a search unit for object identification after extracting features of a region of interest; And a retrieving step of retrieving an image of similarity from a pre-configured ROI database based on the identified object information and obtaining the result.

As an example related to the present invention, the object identification step stores information on the extracted region and the identified object for the region of interest and the region of interest in the region of interest database, and when the corresponding web page information exists, And storing the information in the interest area database in association with each other.

As an example related to the present invention, the search result of the ROI obtained in the search step has web page information in which the similarity corresponds to the image and the ROI, and provides the corresponding web page information.

As an example related to the present invention, the gaze confirmation step is executed in such a manner that index points representing gaze position information according to the user's gaze are accumulated over time.

As one example related to the present invention, the content includes text, photographs, moving pictures, and web pages.

A computer program for carrying out the method according to the above-described embodiment can be stored in the recording medium on which the computer program is recorded.

According to the present invention, an object region is selected through eye tracking, and an object is detected only for a selected region of interest. Therefore, the object can be detected more quickly and accurately than an object detection for an entire image or an image.

In addition, the present invention detects an object for a region of interest, stores information about the object in the region of interest database, stores the corresponding web page information when the corresponding web page information exists, .

1 is a diagram showing an eye tracker for extracting a line-of-sight corresponding area according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating a step of accumulating positional information of a user's observation region with respect to an image according to an embodiment of the present invention.
3 is a diagram illustrating a step of setting a user's area of interest for an image according to an embodiment of the present invention.
4 is a diagram illustrating an object identification apparatus according to an embodiment of the present invention.
5 is a diagram illustrating identification and retrieval of objects according to an embodiment of the present invention.
6 is a flowchart illustrating an object identification and search method according to an embodiment of the present invention.

It is noted that the technical terms used in the present invention are used only to describe specific embodiments and are not intended to limit the present invention. In addition, the technical terms used in the present invention should be construed in a sense generally understood by a person having ordinary skill in the art to which the present invention belongs, unless otherwise defined in the present invention, and an overly comprehensive It should not be construed as meaning or overly reduced. In addition, when a technical term used in the present invention is an erroneous technical term that does not accurately express the concept of the present invention, it should be understood that technical terms that can be understood by a person skilled in the art can be properly understood. In addition, the general terms used in the present invention should be interpreted according to a predefined or context, and should not be construed as being excessively reduced.

Furthermore, the singular expressions used in the present invention include plural expressions unless the context clearly dictates otherwise. The term "comprising" or "comprising" or the like in the present invention should not be construed as necessarily including the various elements or steps described in the invention, Or may include additional components or steps.

In addition, terms including ordinals such as first, second, etc. used in the present invention can be used to describe elements, but the elements should not be limited by terms. Terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to like or similar elements throughout the several views, and redundant description thereof will be omitted.

In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. It is to be noted that the accompanying drawings are only for the purpose of facilitating understanding of the present invention, and should not be construed as limiting the scope of the present invention with reference to the accompanying drawings.

1 is a diagram showing an eye tracker for extracting a line-of-sight corresponding area according to an embodiment of the present invention. The eye tracker 100 is used to extract an area corresponding to the user's line of sight.

The eye tracker 100 can perform automatic focusing on the pupil of the face while performing face recognition of the user. Upon completion of the automatic focusing, the eye tracker 100 can collect a face image including a pupil.

The eye tracker 100 may refer to an eye tracking algorithm stored therein. First, the eye tracker 100 performs facial analysis of the image to extract the pupil region of the face. Check the eye angle of the pupil in the pupil area.

The eye tracker 100 can initially perform positioning before collecting the eye position information of the pupil for reliable position measurement. The eye tracker 100 can confirm and correct which part of the area of the display 200 the user's gaze position information looks at.

FIG. 2 is a diagram illustrating a step of accumulating positional information of a user's observation region with respect to an image according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating a process of setting a user's region of interest Fig.

The eye tracking function of the present invention can be operated in a constant period or in real time. Accordingly, the eye tracker 100 can extract the corresponding region of the display 200 corresponding to the eye position information collected in a predetermined period.

Specifically, when the eye tracker 100 extracts visual line position information transmitted according to a predetermined period (for example, 0.2 seconds), it extracts line-of-sight position information for each predetermined period, The corresponding regions can be displayed and accumulated.

For example, it is assumed that the moment when the eye tracking mode is executed is assumed to be 0 second, the eye point information after 0.2 seconds is extracted to select the eye point 400, the eye point information after 0.2 seconds again, . As time elapses, index points (400) indicating eye position information according to the user's gaze are accumulated on the display (200).

Since the line-of-sight position information extraction is repeated at regular intervals, when the line-of-sight position information is extracted in an area in which the user's line of sight is concentrated, for example, a predetermined number of times or more, It can be judged to be a region of interest.

In FIG. 2, it can be seen that the user's attention is focused on the face, the tie, the bag, the airplane, the clouds, the building and the carrier of the user. In the case where the focused eye position information of the user is more than a predetermined reference, the face, the tie, the bag and the carrier can be judged as the region of interest in the figure.

A specific criterion for whether or not the region of interest is a density is, for example, density. That is, the user's line of sight position information recognized on the display 200 may appear as an index point 400 on the display 200 and the density of the plurality of index points 400 on the entire display 200 may fall within a predetermined range Or more can be judged as the region of interest.

Also, the density of the landmark 400, which is a criterion of the region of interest, may be set to be relatively high. In other words, it is possible to determine that some areas are in the descending order of density of the landmark points 400 among the user's sight line position information recognized on the display 200 as an area of interest.

By performing eye tracking in this way, it is possible to grasp the area where the user's gaze is concentrated without performing any other touch operation or input button control operation, and it is possible to set the area of interest with high reliability.

In FIG. 3, the setting reference of the corner of the ROI formed in the rectangle may be set to a case where the density of the surface point 400 formed by the line-of-sight position information is less than or equal to a predetermined range. That is, the larger the area of the area having the high density of the landmark points 400, the larger the area of the area set as the area of interest correspondingly.

3, the face 501, the tie 502, the bag 503, and the carrier 504 are set as the region of interest, but according to the setting criterion of the region of interest, more objects, or a smaller number of objects, .

3, the location of the region of interest is selected according to the criterion based on the density of the landmark 400. At this stage, the object identification apparatus 600 identifies an object in each ROI , That is, it does not identify a person's face, necktie, bag, and carrier.

The process of identifying an object in the region of interest will be described with reference to FIG. 4 is a diagram illustrating an object identification apparatus according to an embodiment of the present invention.

The object identifying apparatus 600 according to the embodiment of the present invention may include a line-of-sight corresponding region extracting unit 610, a region of interest selecting unit 620, a searching unit 630, and a database 640.

The line-of-sight correspondence region extracting unit 610 may collect the line-of-sight position information of the user extracted by the eye-tracker 100 and extract the region where the line of sight is concentrated. That is, candidate regions of interest can be selected.

The ROI selection unit 620 may select a ROI that is focused more than a predetermined reference among the ROI candidates extracted by the ROI extraction unit 610 as the ROI.

The search unit 630 performs a function of extracting features of objects in the ROI.

To do this, one or more test sets (not shown) are connected to learn each object.

For example, a plurality of test sets may be associated with the search unit 630 so that the first search unit searches for a face of a person, and a first test set includes a plurality of test sets, To the first search unit so that the first search unit can search for a face.

The second search unit searches the bag, and for this purpose, the second test set provides data for the various bags to the second search unit so that the second search unit can learn about the bag so that the second search unit searches for the shape of the bag .

The third search unit searches for the carrier, and for this, the third test set provides the third search unit with data for several carriers so that the third search unit can learn about the carrier, so that the third search unit searches for the shape of the carrier .

The fourth search unit searches for the slip, and for this purpose, the fourth test set provides the data for various slip to the fourth search unit so that the fourth search unit can learn about the slip, so that the fourth search unit searches for the shape of the slip .

Since the sizes of the objects and images may be different each time and the size of the image 300 may be different depending on the size of the image 300 even in the case of the same object, As shown in FIG.

The learned search unit 630 can search for an object in the region of interest 300 within the image 300. The search unit 630 can automatically identify and search the object even in other images so that the person does not have to search for objects in the region of interest for each image.

Since the search unit 630 searches only the region selected as the ROI, the reliability can be improved and the search speed can be improved. By searching for these objects, it is possible to identify what the object is.

The database 640 searches an image having high similarity based on the identified object information and the characteristics of the extracted object, and provides the result.

The database 640 stores the extracted features and the information about the identified object. If the web page information corresponding to the object exists, the database 640 also stores the information of the corresponding web page in the database 640 .

5 is a diagram illustrating identification and retrieval of objects according to an embodiment of the present invention.

In Fig. 5, the carrier 504 will be described as an example. When the carrier 504 is selected as the ROI, the search unit 630 extracts the feature information of the carrier 504, which is an object in the ROI. The feature information may include information such as color, ratio, shape, and size. The search unit 630 can identify the object based on the extracted feature information. This confirms that the object in the region of interest is a carrier.

In this way, the feature of the interest area that the user views is extracted and stored in the database 640 in association with the corresponding web page link including the object information, and then the feature having high similarity with the feature of the ROI is retrieved, (510) and related web page (520) as search results.

6 is a flowchart illustrating an object identification and search method according to an embodiment of the present invention. First, the eye tracking mode is executed (S100), and the position information of the image according to the gaze angle information is accumulated (S200). Accordingly, the eye tracker 100 can extract the eye part from the recognized face, and collect information on the eye angle of the pupil in the eye part.

Specifically, a pupil is a central portion of the eye and a rounded portion surrounded by an iris. The pupil is darker than other parts of the eye, and is usually black for Asians.

The user's gaze may be closely related to the pupil. For example, the point of interest with which the user is interested can be substantially coincident with the direction of the pupil center point.

The center point of the pupil can be extracted based on the relatively dark shape and / or color feature located in the center of the acquired eye image. For example, the center point of the image of the pupil in the round shape can be determined as the center point of the pupil.

The movement of the face can be detected through the face image of the user photographed through the camera 110. That is, the direction of the face can be extracted based on characteristic elements such as forehead, eyes, nose, mouth, ear, and the like in the face image.

The gaze direction can be calculated based on the motion of the face and the calculated center point of the pupil. For example, it may determine which point of the current display 200 the user is staring at. Thus, it is possible to calculate the gaze direction and position of the user gazing at the content displayed on the display 200. [

The content may include text, pictures, images (movies, music, and web pages.) The eye tracking mode may be selected after content is displayed on the screen.

The position information of the image according to the user's gaze angle information can be accumulated based on the gaze position information collected at regular intervals. For example, an index point 400 is generated in the display 200 in response to a user's gaze at a predetermined period, and the plurality of index points 400 are accumulated in the display 200.

Next, a region of interest is set based on the accumulated position information (S300). The criteria for setting the region of interest may be the density of the plurality of index points 400 of the display. That is, an area having a density greater than or equal to a predetermined range can be set as a region of interest. Or a region with a high density may be set as an interest region.

Next, the object identification information of the region of interest is confirmed (S400). This can be confirmed by searching through the search unit 630 for the region of interest. That is, the object region is identified and retrieved by using the region of interest selected by the eye tracking based on the eye tracking as the object candidate region. This identifies the object as a carrier.

The step of verifying the object identification information of the ROI includes storing the ROI in the ROI database 640 and the information about the extracted object and the identified ROI in the ROI database 640, And storing the corresponding web page information in association with each other.

Next, an image having high similarity to the object identified from the database 640 is retrieved and the result is obtained (S500). The acquired result may include an image having high similarity to the object identified in the previous step and corresponding web page information.

By this process, an attention area is selected by focusing the user's attention, and the object is identified and retrieved by using the selected area of interest as an object candidate area. Thus, compared with the object detection for the whole image or image, Can be detected.

In addition, the present invention detects an object for a region of interest, stores information about the object in the region of interest database, and when there is web page information corresponding to the object, the web page information is also linked and stored.

The object identification apparatus and method according to an embodiment of the present invention can be written in a computer program, and the codes and code segments constituting the computer program can be easily deduced by a computer programmer in the field. In addition, the computer program may be stored in a computer-readable medium and readable and executed by a computer or a user device according to an embodiment of the present invention.

The information storage medium includes a magnetic recording medium, an optical recording medium, and a carrier wave medium. A computer program implementing the ROI estimation method according to an embodiment of the present invention may be stored and installed in a built-in memory such as a user equipment. Alternatively, an external memory such as a smart card storing and installing a computer program implementing the ROI service method according to an embodiment of the present invention may be mounted on a terminal or the like through an interface.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or essential characteristics thereof. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

100: Eye Tracker 110: Camera
200: Display 300: Image
400: surface point 501: face
502: Tie 503: Bag
504: Carrier 600: Object Identification Device
610: line-of-sight correspondence region extracting unit 620:
630 search unit 640 database

Claims (10)

A line-of-sight corresponding region extracting unit for extracting a line-of-sight corresponding region by sensing a line of sight on the display on which the content is displayed;
A region of interest selector for selecting a region of interest based on the line of sight corresponding region; And
And a search unit for identifying an object in the region of interest. The method according to claim 1,
And a database for searching an image having a high degree of similarity based on the information of the identified object and providing the result. The method according to claim 1,
Wherein the contents include text, pictures, moving pictures, and web pages. A gaze confirmation step of accumulating position information of an observation image according to gaze angle information of a user gazing at contents displayed on a display;
A ROI setting step of setting ROIs based on the accumulated position information;
An object identifying step of identifying the object identification information through a search unit for object identification after extracting features of the ROI; And
And a retrieval step of retrieving the similarity image from the pre-configured ROI database based on the identified object information and obtaining the result. 5. The method of claim 4,
Wherein the object identifying step stores information on the extracted region and the identified region of interest in the region of interest and the region of interest, and if the corresponding web page information exists in the region of interest, And storing the ROI in the RO area database. 6. The method of claim 5,
Wherein the search result of the ROI obtained in the searching step includes web page information in which the similarity corresponds to the image and the ROI, and the corresponding web page information is provided. 5. The method of claim 4,
Wherein the gaze confirmation step is performed in such a manner that index points representing gaze position information according to the gaze of the user are accumulated as time elapses. 5. The method of claim 4,
Wherein the content includes text, pictures, moving pictures, and a web page. 5. The method of claim 4,
The visual line confirmation step may include recognizing the face of the user; Automatically focusing the user's pupils from the recognized face; And a line of sight is identified based on a center point of the pupil. 10. A recording medium on which a computer program for performing the method according to any one of claims 4 to 9 is recorded.

Images