KR102543306B1 - Apparatus for detecting objects of interest based on 3d gaze point information and providing metadata reflecting user's perspective and perception - Google Patents

Abstract

The present invention relates to an apparatus and method for detecting an object of interest based on 3D gaze point information and providing user visual perception metadata. The apparatus receives and processes an image of an external environment including a plurality of objects in the direction of the user's line of sight and information on the location of the user's pupil from a camera, and processes the image of the external environment and the pupils of both eyes of the user. a gaze point information estimator for estimating gaze point information according to an external environment and a gaze of a user through image information on the subject; Among the images of the external environment, a region of interest corresponding to the user's gaze point is created according to the user's gaze point information and a predetermined criterion, and the size of the region of interest matches the size of the object corresponding to the user's gaze point. a region of interest generating unit for adjusting the; Object information for detecting object information in the ROI using an object detection deep learning model having a structure of a plurality of feature map layers selected according to the ROI and the characteristics of the image of the external environment corresponding to the ROI. detection unit; and a metadata generator configured to generate metadata including visual/perceptual response information of the user based on the user gaze point information and the object of interest information. and a metadata providing unit that simultaneously provides the object of interest information of a surrounding environment where the user is located and visual/perceptual response information of the user to the user based on the metadata.

Description

Apparatus and method for detecting objects of interest based on 3D gaze point information and providing user's visual/perceptual metadata

The present invention relates to an apparatus and method for detecting an object of interest based on 3D gaze point information using a wearable device and providing user visual perception metadata.

Recently, with the development of IT technology, various wearable devices are being used as tools to perform various functions in daily life in connection with various applications.

A commercially available wearable device can store information such as a user or an object through a user's input, and can load desired information when the user needs it.

As an example of such a wearable device, Korean Patent Laid-Open Publication No. 10-2016-0126309 discloses a wearable electronic device that receives a user's input and displays an object of interest separately.

The wearable electronic device of the prior art displays a specific part of an image including a plurality of objects on the first area and the second area, and inputs moving a specific object among the plurality of objects from the first area to the second area. It includes a touch screen that receives and a processor that controls the touch screen so that a first object displayed on the first area and a second object displayed on the second area are distinguished and displayed in response to the received input.

However, such a conventional wearable device requires the user to directly touch the touch screen to move the area of the object of interest, and provides summary information about the object through the user's location information or POI information previously set by the user. It is difficult to analyze the user's behavior and intention and preemptively provide the user with an object that the user is interested in without the user's intervention.

In recent years, electronic devices considering living environment intelligence for interaction between users, devices, and object media are gradually increasing. Living environment intelligence refers to artificial intelligence that can proactively and preemptively respond to user needs without user intervention by analyzing the user's behavior and intentions and recognizing and recognizing surrounding environment information. Unlike artificial intelligence that is specialized for a given specific task, living environment intelligence can be classified as high-level artificial intelligence in that it can provide convenient functions optimized for each user through interaction with the user.

Unlike conventional wearable devices, which simply provide simple information about an object through user intervention, the user's behavior and intention are analyzed to determine which object the user is interested in, and data the user wants. There is a demand for a device capable of preemptively providing information without user intervention.

Korean Patent Publication No. 10-2016-0126309

An object of the present invention is to provide an apparatus and method for detecting an object of interest based on 3D gazing point information and providing metadata for a user's visual perception by detecting a user-centered object of interest and providing metadata for living environment intelligence therethrough.

Another object of the present invention is to optimize object detection performance and operation speed and selectively provide object information suitable for a user's situation by using metadata reflecting the user's living environment intelligence in the object information.

A first aspect of the present invention for solving the above problems is a 3D gaze point that receives and processes an image of an external environment including a plurality of objects in the direction of the user's gaze from a camera and information about the position of the user's pupil. An apparatus for detecting an object of interest based on information and providing user visual perception metadata. The apparatus may include: a gaze point information estimator for estimating gaze point information according to the gaze of the user and the external environment through the image of the external environment and image information of the pupils of both eyes of the user; Among the images of the external environment, a region of interest corresponding to the user's gaze point is created according to the user's gaze point information and a predetermined criterion, and the size of the region of interest matches the size of the object corresponding to the user's gaze point. a region of interest generating unit for adjusting the; Object information for detecting object information in the ROI using an object detection deep learning model having a structure of a plurality of feature map layers selected according to the ROI and the characteristics of the image of the external environment corresponding to the ROI. detection unit; and a metadata generator configured to generate metadata including visual/perceptual response information of the user based on the user gaze point information and the object of interest information. and a metadata providing unit that simultaneously provides the object of interest information of a surrounding environment where the user is located and visual/perceptual response information of the user to the user based on the metadata.

According to an embodiment of the present invention, the user's gaze point information may include a location of the gaze point in the space where the user is located and a gaze distance from the user to the gaze target.

According to an embodiment of the present invention, the ROI generating unit defines a reference location of the ROI using the location of the gaze point as a parameter, and determines the gaze distance, the width and length of the object corresponding to the gaze point, and the external environment. The size of the region of interest can be defined using the width of the image and the horizontal angle of view of a camera that captures the external environment as parameters.

According to an embodiment of the present invention, the ROI generating unit may adjust the width of the ROI by including a color of the image of the external environment, an edge of the image, and depth map information of the image as parameters.

According to an embodiment of the present invention, the object information detector sets a comparison feature between the region of interest generated by the region of interest generator and the image of the external environment as input information, learns the comparison feature, and then learns the plurality of features. An object of interest may be extracted by constructing a variable neural network that selectively combines map layers.

According to an embodiment of the present invention, the comparison feature may include the ratio of the area of the ROI to the width of the image of the external environment, the complexity of the image of the external environment corresponding to the ROI, or the image area corresponding to the ROI. Color contrast of the image with respect to the external environment may be included.

According to an embodiment of the present invention, the object information detection unit extracts the object of interest by activating at least one object candidate estimation layer determined for the input information among the plurality of feature map layers, and through the activated object candidate layer. The type and location information of the object of interest may be estimated.

According to an embodiment of the present invention, the user's gaze point information estimation unit acquires gaze frequency and gaze point formation time information for the object of interest, the metadata generator classifies the interest object by object type, and the interest Gaze frequency and gaze point formation time information on an object may be recorded for each object of interest classified by object type.

According to an embodiment of the present invention, the metadata providing unit represents the size and location of each object of interest in the space where the user is located in the form of a bounding box, and reflects the priority according to the frequency of staring at the object of interest over time. An object map may be created to provide the object of interest information of the surrounding environment where the user is located.

According to an embodiment of the present invention, the metadata providing unit receives the change in the object of interest of the user over time and the gaze frequency information for each object of interest updated by the metadata generating unit, and the objects of interest are selected in order of the user's interest. can be classified and provided.

According to an embodiment of the present invention, the metadata providing unit displays the location of the user's current gaze point, the object of interest corresponding to the gaze point, and the change of the gaze point for the change of the object of interest over time to correspond to each object of interest. It may be provided in the form of a bounding box and a movement path indicating changes in each of the objects of interest.

According to an embodiment of the present invention, the display of the moving path is a change in a gaze point for a change in an object of interest over time, and a corresponding user when the user perceives the nth object of interest from the user's gaze point for the n-1th object of interest. It can be provided to indicate the change in gaze point location to the gaze point of .

A second aspect of the present invention is to detect an object of interest based on 3D gaze point information by receiving and processing an image of an external environment including a plurality of objects in the direction of a user's gaze from a camera and information on the location of a user's pupil, and user It relates to a method for providing visual perception metadata. The method may include: (a) estimating gaze point information according to the external environment and the gaze of the user through the image of the external environment and image information of the pupils of both eyes of the user; (b) Among the images of the external environment, a region of interest corresponding to the user's gaze point is generated according to the user's gaze point information and a predetermined criterion, and the interest is matched to the size of the object corresponding to the user's gaze point. resizing the area; (c) detecting object information in the region of interest using an object detection deep learning model having a structure of a plurality of feature map layers selected according to the region of interest and characteristics of the image of the external environment corresponding to the region of interest step; (d) generating metadata including visual/perceptual response information of the user based on the user gaze point information and the object of interest information; and (e) simultaneously providing the object of interest information of the surrounding environment where the user is located and visual/perceptual response information of the user to the user based on the metadata.

According to the present invention, even if the user does not give a separate operation instruction through a wearable device, etc., the user's visual perception pattern is analyzed to determine and record significantly recognized object or scene (event, etc.) information, and the space where the user is located Detects a user-centered object of interest by using the user's 3D gaze point information, generates and utilizes meta data for living environment intelligence by using it, and actively provides it to the user at the right time at the right time. Make it convenient for you to remember only the information you need.

In addition, by selectively detecting objects within the user's region of interest and storing them, it is possible to extract and provide only necessary information to the user and maximize efficiency in data processing and storage.

1 is a diagram schematically showing each configuration of an apparatus for detecting an object of interest based on 3D gaze point information and providing user visual perception metadata according to an embodiment of the present invention.
FIG. 2 is a schematic diagram of an exemplary wearable device applied to the apparatus of FIG. 1 .
FIG. 3 is a flowchart schematically illustrating a method of detecting an object of interest based on 3D gazing point information and providing user visual perception metadata performed by the apparatus of FIG. 1 .
4 is a flowchart schematically illustrating a process of estimating gaze point information in the gaze point information estimation unit of FIG. 1 .
FIG. 5 is a diagram illustrating an example of generating a region of interest according to information on a user's gaze point by the region of interest generator of FIG. 1 .
FIG. 6 is a diagram illustrating a relationship between a real object and a region of interest for defining an initial region of interest according to a predetermined criterion by the region of interest generator of FIG. 1 by way of example.
7(a) and (b) are diagrams schematically illustrating a process of correcting each region of interest by reflecting object information in the region of interest generator of FIG. 1 .
FIG. 8 is a diagram schematically illustrating a process of selecting different object inference feature maps according to characteristics of a 3D region of interest according to an object by the object information detector of FIG. 1 .
FIG. 9 is a table showing results of improving object detection performance by combining some of the plurality of layers of FIG. 8 .
FIG. 10 is a diagram illustrating features of the selective activation layer of FIG. 8 .
FIG. 11 is a diagram for explaining the object candidate estimation layer selection step and the image feature analysis and object information estimation steps in the object information estimation step of FIG. 8 .
12 is a diagram illustrating an example of metadata in which object information generated by the metadata generator of FIG. 1 is classified by time.
FIG. 13 is a diagram showing an example in which the metadata of FIG. 12 is classified through an object-of-interest information log by time.
14 and 15 are diagrams illustrating examples of using metadata by the metadata providing unit of FIG. 1 .

Hereinafter, specific details for the practice of the present invention will be described with reference to the accompanying drawings. And, in the description of the present invention, if it is determined that the related known function may unnecessarily obscure the subject matter of the present invention as an obvious matter to those skilled in the art, the detailed description thereof will be omitted.

In this specification, a wearable device includes, for example, any device in the form of glasses that can be worn by a user, but is not limited thereto and is used to mean any device that can be worn by a user. In addition, this may include not only a case where the user wears it directly, but also a case where the user indirectly wears it, and may also include a case where the user wears it while intervening with any other configuration.

In addition, an apparatus for detecting an object of interest based on 3D gaze point information and providing user visual perception metadata according to an embodiment of the present invention may include a device such as a server configured separately from a wearable device, but is not limited thereto, and the wearable device It may be attached or integrally constructed, or it may be a processing device within a wearable device.

In addition, in this specification, a relationship between a user and an object may be 1:1, 1:n, or n:n, and when the term user or object is used, it should be interpreted as implying the above meaning.

1 is a diagram schematically showing each configuration of an apparatus for detecting an object of interest based on 3D gaze point information and providing user visual perception metadata according to an embodiment of the present invention. FIG. 2 is a schematic diagram of an exemplary wearable device applied to the apparatus of FIG. 1 . FIG. 3 is a flowchart schematically illustrating a method of detecting an object of interest based on 3D gazing point information and providing user visual perception metadata performed by the apparatus of FIG. 1 .

1 and 3, an apparatus for detecting an object of interest based on 3D gaze point information and providing user visual perception metadata according to the present invention provides an image of an external environment including a plurality of objects in the direction of a user's gaze from a camera and a user It is possible to receive and process information about the pupil's position.

The apparatus includes a gaze point information estimating unit 10, a region of interest generating unit 20, an object information detecting unit 30, a metadata generating unit 40, and a metadata providing unit 50.

The gaze point information estimator 10 estimates gaze point information according to the external environment and the gaze of the user through the image of the external environment and the image information of the pupils of both eyes of the user (s10).

The region of interest generation unit 20 generates a region of interest corresponding to the user's gaze point according to the user's gaze point information and a predetermined criterion in the image of the external environment, and determines the size of the object corresponding to the user's gaze point. Adjust the size of the region of interest accordingly (s20).

The object information detector 30 detects object information in the region of interest by using an object detection deep learning model having a layer structure of a plurality of feature maps selected according to the region of interest and the characteristics of the image of the external environment corresponding to the region of interest. Do (s30).

The metadata generation unit 40 generates metadata including user's visual/perceptual response information based on the user's gaze point information and the object of interest information (S40).

The metadata providing unit 50 simultaneously provides object-of-interest information of the surrounding environment where the user is located and visual/perceptual response information of the user to the user based on the metadata (S50).

A detailed description of each of the above components will be described later.

Each unit of the apparatus for detecting an object of interest and providing user visual perception metadata according to an embodiment of the present invention receives an image of an external environment and information about the location of a user's pupil by means of the wearable device exemplarily shown in FIG. 2 can receive However, the wearable device shown in FIG. 2 is only shown as an example, and is not limited thereto, and various types of known wearable devices may be applied.

Referring to (a) of FIG. 2 , the wearable device used in the present invention may be configured as, for example, a glasses-type device having an augmented reality function. In addition, the wearable device may include a front camera 1 configured to capture an image that includes at least a part of the user's field of view. The front camera 1 may be directly or indirectly mounted on the frame in a direction in which the user's gaze is directed.

Referring to (b) of FIG. 2 , the wearable device used in the present invention may include left and right pupil cameras 2 and 3 . The left and right pupil cameras 2 and 3 can detect an image of the pupil when worn by a user and generate a pupil image signal. The left and right pupil cameras 2 and 3 may be directly or indirectly mounted on the frame in a direction toward the user's pupil.

In addition, the wearable device may include a transparent or translucent display 4 displaying information necessary for a user. Furthermore, it may further include a microphone (not shown) configured to detect a user's voice and generate a sound signal.

The wearable device can obtain information about the user's behavior and state from the front camera 1, the left and right pupil cameras 2 and 3, or a microphone, and converts the information into the 3D gaze point information of the present invention. Based object of interest detection and user visual perception metadata can be transmitted to the providing device.

For example, the apparatus of the present invention includes information on an image of an external environment including a plurality of objects in the direction of the user's gaze obtained from the front camera 1 of the wearable device, and left and right pupil cameras 2, Information on the location of the pupil of the user obtained from 3) may be received.

Hereinafter, a configuration for processing information provided from a wearable device by the apparatus for detecting an object of interest and providing user visual perception metadata according to the present invention will be described in turn.

4 is a flowchart schematically illustrating a process of estimating gaze point information in the gaze point information estimation unit of FIG. 1 .

The gaze point information estimator 10 uses the image of the external environment acquired from the front camera 1 and the image information of the pupils of both eyes of the user obtained from the left and right pupil cameras 2 and 3 to determine the external environment and the user. Gaze point information according to the line of sight is estimated (s10). The gaze point information estimator 10 estimates the user's 3D gaze point information formed on the 3D space.

The user's gaze point information includes the location of the gaze point in the space where the user is located and the gaze distance from the user to the gaze target. The user's gazing point information may be obtained using not only pupil position information but also Purkinje images and gaze convergence information. The information on the user's gaze point may include not only the location of the gaze point and the gaze distance, but also information about the user's gaze frequency, pupil saccade, gaze fixation, gaze fixation time, and gaze tracking.

Referring to FIG. 4 , gaze point information of a user may be estimated through the following process.

First, the gaze point information estimator 10 may detect the Purkinje image and the pupil from the pupil capture images acquired from the left or right pupil cameras 2 and 3 (s2 and s3) (s11). In addition, image calibration may be performed between the pupil capture image and the front camera (s12), and a convergence point may be estimated using the detected pupil information and the calibrated gaze image of the front camera (s13).

Next, gaze information based on monocular disturbance may be estimated using the detected Purkinje image, pupil information, and gaze convergence information (s14). At this time, the information utilized is not limited to the information presented above.

Alternatively, the gaze point information estimation unit 10 performs calibration between the pupil capture image acquired from the left pupil camera 2, the pupil capture image acquired from the right pupil camera 3, and the gaze image captured from the front camera (s1) is performed (s12), and a binocular convergence point may be estimated using the gaze images obtained from the left and right pupil cameras (s13).

Next, gaze information may be estimated using the detected joint information, gaze convergence information, and the like (s14).

The gaze information may include a 2D gaze position, a gaze distance, information on whether gaze is fixed, information on a tracking gaze point, and information about whether or not a saccade is performed.

FIG. 5 is a diagram illustrating an example of generating a region of interest according to information on a user's gaze point by the region of interest generator of FIG. 1 .

Referring to FIG. 5 , the region of interest generation unit 20 generates a region of interest corresponding to the user's gaze point according to information about the user's gaze point and a predetermined criterion among the images of the external environment obtained from the front camera 1. Create (s20). The ROI generating unit 20 receives the user's 3D gaze point information from the gaze point information estimation unit 10 . In addition, a user gaze image is provided from the front camera 1 . The region of interest may be defined as a rectangle having a certain area or may be defined along the boundary of an object. The region of interest is a region including an object in which the user shows interest by gazing at the user.

The ROI generating unit 20 may create an adaptive 3D ROI by referring to the user's 3D gaze point information and gaze object information. The adaptive 3D region of interest refers to a region of interest generated to correspond to the structural characteristics (position, size, etc.) of the target object by referring to the user's 3D gaze point information.

The ROI generating unit 20 receives the user gaze image obtained from the front camera 1 . In addition, the ROI generator 20 receives the user's 3D gaze point information from the gaze point information estimation unit 10 . The region of interest generator 20 creates a region of interest based on the user gaze image and information on the gaze point of the user. The set region of interest may be a 3D region of interest in which the distance between the user and the object is reflected.

Referring to (a) of FIG. 5, the region of interest generator 20 defines the reference position of the region of interest using the position of the gaze point as a parameter, and determines the gaze distance, the width and length of the object corresponding to the gaze point, and the external environment. The size of the region of interest can be defined by using the width of the image and the horizontal angle of view of a camera that captures the external environment as parameters.

The region of interest generator 20 adjusts the size of the region of interest according to the gaze distance of the user. When the user's gazing distance is relatively long, the ROI generating unit 20 sets the size of the ROI to be small in order to focus the user on the corresponding object. This is because if the gaze distance is long, the user is looking at the object carefully. On the other hand, when the user's gazing distance is relatively short, the ROI generating unit 20 sets the size of the ROI large in order to focus the user on the corresponding object. That is, the gaze distance of the user may be a parameter defining the region of interest of the corresponding object.

Since the area of interest of the object is relatively different depending on the gazing distance, set the width of the area of interest with a long gazing distance to be relatively small and the width of the area of interest with a short gazing distance to be relatively large. can be reflected in the width of

FIG. 6 is a diagram illustrating a relationship between a real object and a region of interest for defining an initial region of interest according to a predetermined criterion by the region of interest generator of FIG. 1 by way of example.

Referring to FIG. 6 , the ROI generating unit 20 comprehensively utilizes the user's 3D gazing point information, object geometry information (size, location, etc.), and camera view angle and image information to determine the size of the initial ROI. , position, and ratio can be defined. The region of interest may be formed in three dimensions. The initial ROI refers to an ROI generated before the ROI generator 20 adjusts the size of the ROI, as will be described later.

The initial region of interest not only presents a reference point necessary for image characteristic analysis, but also limits the search range, and contributes to determining the correction direction (expansion/reduction) of the region of interest to be adjusted later.

For example, the initial region of interest may be defined as follows based on gaze point information such as the user's gaze position and gaze distance.

This is derived from the following process.

Here, 'ROIw(d)' means the initial region-of-interest width (pix) when the user looks at the distance 'd' through the user's gazing distance. Also, 'Ow' means the maximum width length (m) of the object. Also, 'R ^w cam' means the width (pix) of the image of the external environment. Also, 'HFOV' means the horizontal field of view (H-FOV) of the front camera 1. Also, 'd' means the gaze distance of the user. 'ROIw(d)' derived through the above parameters means the width (pix) of the initial region of interest when looking at the distance 'd'.

According to this, the width 'ROIw(d)' of the region of interest when looking at the distance 'd' is the maximum width length 'Ow' of the object, the image width 'R ^w cam', and the horizontal angle of view of the front camera 1 It is proportional to 'HFOV' and inversely proportional to the user's gaze distance 'd'.

For example, a relatively large initial ROI may be set for a relatively wide object, and a relatively small initial ROI may be set for a relatively small width object. In this case, a relatively small initial region of interest may be set for an object at which the user's gaze distance is relatively long, and a relatively large initial region of interest may be set for an object at which the user's gaze distance is relatively short.

Referring to (b) of FIG. 5 , the ROI generator 20 may create a ROI to exclude an object not belonging to the ROI from a detection target. That is, according to the configuration, the width of the region of interest having a long gazing distance is set to be relatively small, and other non-interest regions can be excluded relatively wide. On the other hand, the width of an area with a relatively short gaze distance is set to be relatively large, so that other non-interest areas can be excluded as relatively small. Accordingly, it is possible for the user to more selectively observe a region of interest with a long gazing distance.

Conventionally, there is a problem in that the loss of data representing the target object is large because the entire image including the target object is reduced to the input size of the network and used as input data. In the present invention, data loss can be minimized because an image obtained by adjusting only the region of interest in the entire image to the size of the network input image is used as network input data. That is, according to the present invention, object detection performance can be improved by providing high-quality image information around the object of interest to the user. In addition, unnecessary candidate objects may be excluded in advance by dividing the user's region of interest and region of non-interest, and an unnecessary search region may be reduced and the amount of computation of a processor may be reduced in an object detection and recognition step.

7(a) and (b) are diagrams schematically illustrating a process of correcting each region of interest by reflecting object information in the region of interest generator of FIG. 1 .

Referring to (a) and (b) of FIG. 7 , the ROI generator 20 may adjust the size of the initial ROI according to the size of the object corresponding to the user's gazing point (S20). The region of interest generator 20 may adjust the width of the region of interest by including the color of the image of the external environment, the edge of the image, and the depth map information of the image as parameters. The edge information of the object is obtained by, for example, capturing an image in front of the user through a front camera 1 mounted on a wearable device, and then analyzing depth information of each part of the image using a known image processing and detection technology can do.

Even if the initial region of interest is set for the object of interest corresponding to the user's gaze point, each object may have a different shape and size regardless of the user's gaze distance. For example, when a user sees a chair with short legs and a chair with long legs, the initial regions of interest may be set to be the same, but the shapes of the two are actually different and need to be reflected in the regions of interest.

In consideration of this, the ROI generating unit 20 refers to the image characteristics (color, edge, etc. of the object) and precisely corrects the initially generated ROI to match the shape and size of the object of interest corresponding to the gaze point. Post-processing is performed. In detail, image information such as the edge of an object, color, brightness, etc. is grouped to form the overall shape of the object, and the edge of the object is the preferred element that can express the shape of the object. It is considered, and the color and brightness information of the object can be additionally utilized.

Referring to (a) of FIG. 7 , the region of interest generator 20 detects the shape of the object, and if the object is smaller than the region of interest initially determined by the region of interest generator 20, the region of interest is set as the object's shape. It can be shrunk to the edge.

In detail, first, the ROI generating unit 20 generates an initial ROI according to the user's gazing point information and a predetermined criterion. Next, the edge of the object is detected by analyzing the depth information of each part of the image acquired through the camera or the like. The ROI generator 20 searches for the edge boundary of the object closest to the generated initial ROI, and if the edge of the nearest object is located inside the ROI, the initial ROI is contracted to the nearest edge boundary and updated. A region of interest can be created.

On the other hand, referring to (b) of FIG. 7 , the ROI generating unit 20 detects the geometric information of the object and, if the object is larger than the initially generated ROI, may extend the ROI to the edge of the object. there is.

In detail, first, the ROI generating unit 20 generates an initial ROI according to the user's gazing point information and a predetermined criterion. Next, the edge of the object is detected by analyzing the depth information of each part of the image acquired through the camera or the like. The ROI generator 20 searches for the edge boundary of the object closest to the generated initial ROI, and if the edge of the nearest object is located outside the ROI, extends the initial ROI to the nearest edge boundary and updates the update. A region of interest can be created.

FIG. 8 is a diagram schematically illustrating how the object information detector of FIG. 1 selects different object inference feature maps according to characteristics of a 3D region of interest according to an object. FIG. 9 is a table showing results of improving object detection performance by combining some of the plurality of layers of FIG. 8 . FIG. 10 is a diagram illustrating features of the selective activation layer of FIG. 8 .

Referring to FIG. 8 , the object information detector 30 is an object detection deep learning model having a structure of a plurality of feature-map layers selected according to a region of interest and characteristics of an image of an external environment corresponding to the region of interest. Object information within the region of interest may be detected using (s30). The object information detector 30 receives the adaptive 3D ROI information generated by the ROI generator 20 . In addition, a user gaze image is provided from the front camera 1 .

The object detection deep learning model can detect an object of interest by constructing a variable neural network that selectively combines a plurality of feature map layers by learning comparison features between an ROI and images of an external environment.

The object detection step (s30) includes an input step (s31), an object information estimation step (s32), and an output step (s33). Each of the above steps is performed by the object information detection unit 30 .

The input step (s31) is a step of inputting an image for object detection and other image features (selective-activator features). In the input step (s31), comparison characteristics between the region of interest generated by the region of interest generator 20 and the image of the external environment are set as input information.

The object information estimation step (s32) is a step of analyzing the image and other image characteristics of the input step (s31), adaptively selecting a layer to be used for object inference, and estimating object type and location information. In the object information estimation step (s32), an object of interest is extracted by constructing a variable neural network that selectively combines a plurality of feature map layers by learning the comparison features.

The output step (s33) is a step of selecting and outputting an object having high reliability among the objects extracted in the object information estimation step (s32). In the output step s33, an object with high reliability among the objects of interest estimated in the object information estimation step s32 is output.

The confidence of the detected object means the probability that the object information detection unit 30 will accurately determine the object as a corresponding class (type, category). For example, when detecting a “chair”, if the object information detection unit 30 infers a 90% probability that the corresponding object is a “chair” and a 35% probability that it is a “table”, the object is finally identified as “a chair”. This means that it is identified as “a chair” and outputted. Here, the object information detection unit 30 may gradually increase the reliability of the detected object as the detected object information accumulates by learning the detected object class using the object detection deep learning model.

Since each region of interest generated and resized by the region of interest generator 20 has a different size according to the size of an object, a plurality of feature maps are also determined according to the size of each region of interest. The object information detection unit 30 uses a plurality of feature map layer structures having different sizes to detect object information. Depending on which feature maps are selectively used among the plurality of feature maps, object detection performance and operation speed vary. . The object information detection unit 30 may exclude some of the plurality of feature maps by learning the type of the resized region of interest.

The object information detector 30 constructs various derived neural network combinations by changing the configuration of the feature map layers of the given reference neural network, and analyzes the characteristics of the input image provided from the region of interest generator 20 to determine the optimal derived neural network among them. Object recognition can be performed by selecting as a trial unit.

Conventionally, object recognition is performed using a predefined single object detection network for an input image, which corresponds to a general-purpose network independent of the size and characteristics of a target object. On the other hand, according to the present invention, a plurality of derived neural networks having different performance characteristics, that is, different network complexity, can be generated from a given single neural network according to target object properties, and by adaptively selecting and using the derived neural networks, optimal performance can be shown.

At this time, the object information detection unit 30 is an optional activation layer that selects different object inference feature maps according to the characteristics of the region of interest according to the detected object using information such as color, material, size, etc. of the object in the 3D region of interest. (Selective-Activator Layer) can be defined and applied to construct an artificial neural network optimized for object detection based on gaze point information.

The selective activation layer is designed to activate or deactivate each feature map layer to construct an optimized artificial neural network combination by analyzing the image characteristics of the adaptive 3D region of interest. Accordingly, since unnecessary feature map layers are excluded, there is an advantage in that accuracy and speed of calculation are improved. On the other hand, the selective activation layer is connected in a way of crossing the selective activation layer feature vector (Selective-Activator Feature Vector) according to the input image.

Referring to FIG. 9 , the object information detection unit 30 may optimize object detection performance by applying a plurality of derived neural networks having various feature map layer configurations. That is, it is possible to automatically design and select the most efficient combination of feature map layers required for object inference in any target environment, and the object detection performance and operation speed of the device can be improved through such customized selection of the object inference layer. .

There is a trade-off relationship between detection accuracy and speed, and the performance of derived neural networks varies greatly depending on the size of a target object. By selecting and applying a derived neural network for each trial unit, both high detection accuracy and fast detection speed can be satisfied. According to the embodiment, when both accuracy and speed are considered, the 'T2' model of FIG. 9 is advantageous for detecting large objects, and the 'T4' model is advantageous for detecting small objects.

Referring to FIG. 9 in detail, the 'SSD (Single-Shot Multi-Box Detector)' model corresponds to a previous study as a baseline, which is an algorithm to be compared. In the SSD model, 'Conv7', 'Conv8', 'Ext-1', 'Ext-2', 'Ext-3', and 'Ext-4' of FIG. 8 are all selected. This represents an example of a learned image feature analysis filter. The 'Ada-3D-RoI + SSD' model corresponds to an object detection model when the baseline is applied without changing the network structure by applying the 'adaptive 3D region of interest' of the present invention. In the 'Ada-3D-RoI + SSD' model, 'Conv7', 'Conv8', 'Ext-1', 'Ext-2', 'Ext-3', and 'Ext-4' in FIG. 8 are all selected. . The 'Ada-3D-RoI + Mod-SSD (Modified SSD)' model applies the 'adaptive 3D region of interest' of the present invention, but changes the network structure of the baseline (O/X in the 'Bounding Box Prediction Layer' column) indicates whether the corresponding layer is selected or excluded) and corresponds to one model.

The 'Object Size' column in FIG. 9 means object detection performance according to the size of the object, and FPS means the operating speed in terms of frames per second. 'Avg. 'Gain(%)' represents the average value of how many percent of each detection model's performance improved compared to the existing performance of 'SSD'. For example, in the case of the 'Ada-3D-RoI + SSD' model, it is about '23%' and '13%' depending on the size of each object ('small', 'medium', 'large') compared to the 'SSD' model. ', '7%' indicates an improvement in the detection rate, and the average value is 'Avg. It is displayed as 'Gain(%)' and '14.1%'.

According to the present invention, when 'adaptive 3D region of interest (Ada-3D-RoI)' is used, higher performance (' Avg. Gain (%)'). In addition, comparing the number of layers selected in 'Bounding Box Prediction Layer', T3 uses a total of 4 layers, while T4 uses a total of 3 layers, which is the smallest among T1 to T4. The action speed appears faster.

Combination data of the selected most efficient layers may be stored in a general storage device such as a hard disk or a server, and may be selectively included in a wearable device and/or an external computing device.

Referring to FIG. 10, the comparison characteristics between the region of interest and the image of the external environment, which is learned by the object detection deep learning model, are the ratio of the width of the region of interest to the image width of the external environment, and the image of the external environment corresponding to the region of interest. It may include the complexity of , or the color contrast of the image with respect to the external environment corresponding to the region of interest.

The image scale information is obtained using the original image and the region of interest generated by the region of interest generation unit as a size ratio of the original image to the input image by the following equation. The image scale information is expressed as a one-dimensional vector and corresponds to a geometric feature representing how much the size of the object of interest is relative to the entire input image.

Image scale ratio =

(

(

The complexity of the input image can be obtained using a frequency band. When reduced to a frequency band, it is possible to simplify and express information such as noise and edges existing in an image (encoding role). This is expressed as a total of 275-dimensional vectors, and out of the total 300-dimensional vectors, the remaining 275 dimensions, excluding the 25 dimensions representing image scale information and image contrast information, are used as vectors representing the input image complexity.

The color contrast information of the image may be obtained through contrast information for each color channel of the image. The color contrast information of the image can be expressed as a total of 24-dimensional vectors by generating a histogram of brightness information for each color channel (R, G, B) of the input image and dividing each histogram into 8 cells.

In the object detection step (s30), in the input step (s31), the input image defined through the adaptive 3D region of interest and the image characteristics of the region of interest are used as input information. For image characteristics, image scale information, input image complexity, and image contrast information are used, and encoded and input as vectors in 1-dimensional, 275-dimensional, and 24-dimensional forms, respectively, to construct an end-to-end network. It is input to the network in a form combined with the image. FIG. 11 is a diagram for explaining the object candidate estimation layer selection step, image feature analysis, and object information estimation step in the object information estimation step of FIG. 8 .

Referring to FIG. 11 , the object information estimation step (s32) includes an object candidate estimation layer selection step (s32-1) and image feature analysis and object information estimation step (s32-2). Each of the above steps is performed by the object information detection unit 30 . In the object candidate estimation layer selection step (s32-1), the object candidate estimation layers ('Conv8', 'Ext-1', 'Ext-2', 'Ext-3', and 'Ext-4') suitable for the input image multiple selection) is activated. The image feature analysis and object information estimation step (s32-2) analyzes the corresponding image feature and estimates the type and location information of the object by utilizing a previously selected object candidate estimation layer.

When connecting the 'Selective-Activator Feature Vector' stage and the 'Selective-Activator' stage of FIG. 8, they can all be connected. In the case of the 'Selective-Activator' stage, neurons (marked with blue circles) are set according to the number of target layers to be activated or deactivated. When all vector elements of the 'Selective-Activator Feature Vector' stage and neurons of the 'Selective-Activator' stage are connected and learned, which layer is selected from the 'Bounding Box Prediction Layer' column in FIG. 9 according to the characteristics of the input image Neurons in the learned 'Selective-Activator' stage decide whether to activate or inactivate.

In the output step (s33), the object information detection unit 30 calculates reliability by category, location reliability, and non-maxima suppression (NMS) for the plurality of object candidates extracted in the object information estimation step (s32). Through this, meaningful objects are selected and provided to the user.

12 is a diagram illustrating an example of metadata in which object information generated by the metadata generator of FIG. 1 is classified by time. FIG. 13 is a diagram showing an example in which the metadata of FIG. 12 is classified through an object-of-interest information log by time.

Referring to FIGS. 12 and 13 , the metadata generation unit 40 may generate metadata including visual/perceptual response information of the user based on the user gaze point information and the object of interest information (S40). The generated metadata may be stored in a storage device such as a memory, and the stored information may be loaded into the metadata provider 50 . The metadata generating unit 40 and the metadata providing unit 50 receive user 3D gaze point information from the gaze point information estimation unit 10 and user interest object information from the object information detection unit 30 . In addition, the metadata generating unit 40 and the metadata providing unit 50 receive user gaze images from the front camera 1 and user eye images from the left and right pupil cameras 2 and 3 .

The gaze point information estimated by the gaze point information estimation unit 10 includes the user's gaze point location and gaze time. The gaze point information estimator 10 may also obtain gaze frequency and gaze point formation time information for the object of interest. The object information detected by the object information detection unit 30 includes an ID assigned to each object and the location and size of each object. Each of the pieces of information acquired by the gaze point information estimation unit 10 and the object information detection unit 30 are provided to the metadata generation unit 40 .

The metadata generating unit 40 may classify objects of interest by object type, and may record gaze frequency and gaze point formation time information for each object of interest classified by object type. Through this, the metadata generation unit 40 may generate metadata for living environment intelligence.

The metadata generation unit 40 receives information on the user's gaze point from the gaze point information estimation unit 10 and includes time information when the user gazes at each object and location information of each object in the form of an object of interest information log by time. generate metadata that The generated metadata may be stored in the storage device. The location information of the object may be obtained using a known location tracking device such as GPS.

For example, the user determines the position of the lamp at times t1 and t2 (x1, y1, z1), the position of the table at time t3 (x3, y3, z3), and the position of the first chair at time t4 (x4, y4, z4 ), the position of the second chair at time t5 (x5, y5, z5), the position of the mirror at time t6 and t7 (x6, y6, z6), and the position of the table at time t8 (x3, y3, z3) In this case, the metadata generation unit 40 may classify the object information by time and generate an object of interest information log by time.

14 and 15 are diagrams illustrating examples of using metadata by the metadata providing unit of FIG. 1 . 14 shows an example of using environment/object recognition and analysis, and FIG. 14 shows an example of user behavior/intention recognition and analysis.

Referring to FIGS. 14 and 15 , the metadata providing unit 50 provides object of interest information of the surrounding environment where the user is located and visual/perceptual response information of the user based on the metadata generated by the metadata generating unit 40. at the same time to the user. The object-of-interest information of the surrounding environment includes the configuration and arrangement of the objects of interest belonging to the surrounding environment, and interest level information for each object.

The metadata providing unit 50 represents the size and position of each object of interest in the form of a bounding box in the space where the user is located, and creates an object of interest map reflecting priority according to the frequency of staring at the object of interest over time, so that the user can Information on the object of interest in the surrounding environment may be provided.

The metadata providing unit 50 may receive a change in the user's object of interest over time and gaze frequency information for each object of interest updated from the metadata generator, and classify the objects of interest in order of the user's level of interest and provide the objects of interest.

The metadata providing unit 50 stores the location of the current gaze point of the user, the object of interest corresponding to the gaze point, and the change of the gaze point for the change of the object of interest over time as a bounding box corresponding to each object of interest and each object of interest. It can be provided in the form of a movement path display indicating a change in .

For example, the user determines the position of the lamp at times t1 and t2 (x1, y1, z1), the position of the table at time t3 (x3, y3, z3), and the position of the first chair at time t4 (x4, y4, z4 ), the position of the second chair at time t5 (x5, y5, z5), the position of the mirror at time t6 and t7 (x6, y6, z6), and the position of the table at time t8 (x3, y3, z3) In this case, an example of using the metadata of the metadata providing unit 50 is as follows.

The metadata provider 50 displays the position of the lamp in the object interest map at time t1, the position of the table in the object interest map at time t3, the position of the first chair in the object interest map at time t4, and the object at time t7 In the interest map, object-of-interest information may be provided to the user in such a way that the position of the mirror is specified on the object interest map.

The metadata providing unit 50 may provide a change in a user's gaze point in the form of a movement path indicating a change in each object of interest. The movement route display shows the change in the gaze point for the change of the object of interest over time, and the location of the gaze point from the user's gaze point for the n-1th object of interest to the user's gaze point corresponding to the n-th object of interest. may be provided to indicate a change.

For example, the metadata providing unit 50 moves from the position of the lamp (x1, y1, z1) at time t2 to the position (x3, y3, z3) of the table at time t3, and also of the second chair at time t5. A change in the user's gaze point may be provided by changing the region of interest from the position (x5, y5, z5) to the position (x6, y6, z6) of the mirror at time t6.

In detail, when the information at time t3 is loaded, the metadata providing unit 50 provides the user with previous gaze point location information about the location (x1, y1, z1) of the lamp, which is the previous object of interest, to the user, and at the same time Current gazing point position information about the table position (x3, y3, z3) as an object of interest may be provided to the user together with a gaze point position change indication (indicated by a yellow arrow) from previous gazing point position information.

The scope of protection in this field is not limited to the descriptions and expressions of the embodiments explicitly described above. In addition, it is added once again that the protection scope of the present invention cannot be limited due to obvious changes or substitutions in the technical field to which the present invention belongs.

Claims (24)

An apparatus for detecting an object of interest based on 3D gaze point information and providing user visual perception metadata by receiving and processing an image of an external environment including a plurality of objects in the direction of the user's gaze from a camera and information on the position of the user's pupil in
a gaze point information estimator for estimating gaze point information according to the gaze of the user and the external environment through the image of the external environment and image information of the pupils of both eyes of the user;
Among the images of the external environment, a region of interest corresponding to the user's gaze point is created according to the user's gaze point information and a predetermined criterion, and the size of the region of interest matches the size of the object corresponding to the user's gaze point. a region-of-interest generating unit that adjusts;
Object information for detecting object information in the ROI using an object detection deep learning model having a structure of a plurality of feature map layers selected according to the ROI and the characteristics of the image of the external environment corresponding to the ROI. detection unit; and
a metadata generating unit generating metadata including visual/perceptual response information of a user based on the user gaze point information and the object of interest information; and
a metadata providing unit that simultaneously provides the object of interest information of a surrounding environment where the user is located and visual/perceptual response information of the user to a user based on the metadata;
The metadata providing unit expresses the size and location of each object of interest in the form of a bounding box on the space where the user is located, creates an object of interest map reflecting priority according to the frequency of staring at the object of interest over time, and generates an object of interest map where the user is located. An apparatus for detecting an object of interest based on 3D gaze point information and providing user visual perception metadata, characterized in that the object of interest information of the surrounding environment is provided.
According to claim 1,
Detecting an object of interest based on 3D gaze point information and providing user visual perception metadata, characterized in that the user's gaze point information includes a gaze point position in the space where the user is located and a gaze distance from the user to the gaze target. Device.
According to claim 2,
The region of interest generation unit defines a reference position of the region of interest using the position of the gaze point as a parameter, and determines the gaze distance, the width and length of the object corresponding to the gaze point, the width of the image relative to the external environment, and the external environment An apparatus for detecting an object of interest based on 3D gaze point information and providing user visual perception metadata, characterized in that the size of a region of interest is defined using a horizontal angle of view of a camera as a parameter.
According to claim 3,
The region of interest generator adjusts the width of the region of interest by including a color of the image of the external environment, an edge of the image, and depth map information of the image as parameters, based on 3D gaze point information. Object detection and user visual perception metadata provision device.
According to claim 1,
The object information detector sets a comparison feature between the region of interest generated by the region of interest generator and the image of the external environment as input information, learns the comparison feature, and selectively combines the plurality of feature map layers. An apparatus for detecting an object of interest based on 3D gazing point information and providing user visual perception metadata, characterized in that a neural network is configured to extract an object of interest.
According to claim 5,
The comparison feature may be the ratio of the width of the ROI to the image width of the external environment, the complexity of the image of the external environment corresponding to the ROI, or the color of the image of the external environment corresponding to the ROI. An apparatus for detecting an object of interest based on 3D gazing point information and providing user visual perception metadata, characterized in that it includes contrast.
According to claim 5,
The object information detection unit extracts the object of interest by activating at least one object candidate estimation layer determined for the input information among the plurality of feature map layers, and the type and location information of the object of interest through the activated object candidate layer. An apparatus for detecting an object of interest based on 3D gazing point information and providing user visual perception metadata, characterized in that estimating .
According to claim 1,
The user's gaze point information estimator obtains gaze frequency and gaze point formation time information for the object of interest;
The metadata generation unit classifies the object of interest by object type, and records gaze frequency and gaze point formation time information for each object of interest classified by object type. Based object of interest detection and user visual perception metadata provision device.
delete According to claim 1,
The metadata providing unit receives the change in the user's object of interest over time and the gaze frequency information for each object of interest updated by the metadata generating unit, and classifies the objects of interest in order of the user's level of interest and provides them. An apparatus for detecting an object of interest based on 3D gaze point information and providing user visual perception metadata.
According to claim 1,
The metadata providing unit stores the location of the user's current gaze point, the object of interest corresponding to the gaze point, and the change of the gaze point for the change of the object of interest over time in a bounding box corresponding to each object of interest and each object of interest. An apparatus for detecting an object of interest based on 3D gazing point information and providing user visual perception metadata, characterized in that it is provided in the form of a movement path indicating change.
According to claim 11,
The movement route display shows the change in the gaze point for the change of the object of interest over time, and the location of the gaze point from the user's gaze point for the n-1th object of interest to the user's gaze point corresponding to the n-th object of interest. An apparatus for detecting an object of interest based on 3D gazing point information and providing user visual perception metadata, characterized in that it is provided to indicate a change.
A method for detecting an object of interest based on 3D gaze point information and providing user visual perception metadata by receiving and processing an image of an external environment including a plurality of objects in the direction of the user's gaze from a camera and information on the location of the user's pupil in
(a) estimating gaze point information according to the external environment and the gaze of the user through the image of the external environment and the image information of the pupils of both eyes of the user;
(b) Among the images of the external environment, a region of interest corresponding to the user's gaze point is generated according to the user's gaze point information and a predetermined criterion, and the interest is matched to the size of the object corresponding to the user's gaze point. resizing the area;
(c) detecting object information in the region of interest using an object detection deep learning model having a structure of a plurality of feature map layers selected according to the region of interest and characteristics of the image of the external environment corresponding to the region of interest step;
(d) generating metadata including visual/perceptual response information of the user based on the user gaze point information and the object of interest information; and
(e) simultaneously providing the object of interest information of a surrounding environment where the user is located and visual/perceptual response information of the user to the user based on the metadata;
The step (e) represents the size and position of each object of interest in the form of a bounding box on the space where the user is located, and creates an object of interest map reflecting priority according to the frequency of staring at the object of interest over time, so that the user can A method of detecting an object of interest based on 3D gaze point information and providing user visual perception metadata, characterized in that providing the object of interest information of a located surrounding environment.
According to claim 13,
Detecting an object of interest based on 3D gaze point information and providing user visual perception metadata, characterized in that the user's gaze point information includes a gaze point position in the space where the user is located and a gaze distance from the user to the gaze target. method.
According to claim 14,
In the step (b), the reference position of the region of interest is defined using the location of the gaze point as a parameter, and the gaze distance, the width and length of the object corresponding to the gaze point, the width of the image relative to the external environment, and the external A method for detecting an object of interest based on 3D gaze point information and providing user visual perception metadata, characterized in that the size of a region of interest is defined using a horizontal angle of view of a camera photographing an environment as a parameter.
According to claim 15,
In the step (b), the width of the region of interest is adjusted by including a color of the image of the external environment, an edge of the image, and depth map information of the image as parameters, based on 3D gaze point information. A method for detecting an object of interest and providing user visual perception metadata.
The method of claim 13, wherein step (c) is
(c-1) setting a comparison feature between the ROI generated in step (b) and the image of the external environment as input information; and
(c-2) extracting an object of interest by constructing a variable neural network that learns the comparison feature and selectively combines the plurality of feature map layers, and detects the object of interest based on 3D gaze point information. and a method for providing user visual perception metadata.
According to claim 17,
The comparison feature may be the ratio of the width of the ROI to the image width of the external environment, the complexity of the image of the external environment corresponding to the ROI, or the color of the image of the external environment corresponding to the ROI. A method of detecting an object of interest based on 3D gaze point information and providing user visual perception metadata, characterized in that it includes contrast.
According to claim 17,
The step (c-2) activates at least one object candidate estimation layer determined for the input information among the plurality of feature map layers, and estimates the type and location information of the object of interest through the activated object candidate layer. A method of detecting an object of interest based on 3D gaze point information and providing user visual perception metadata, characterized in that:
According to claim 13,
Step (a) obtains information on the frequency of gaze and the formation time of gaze point for the object of interest;
In the step (d), the object of interest is classified by object type, and gaze frequency and gaze point formation time information for the object of interest are recorded for each object of interest classified by object type. A method for detecting an object of interest based on information and providing user visual perception metadata.
delete According to claim 21,
In the step (e), the change in the object of interest of the user over time and the gaze frequency information for each object of interest updated from the metadata generator are received, and the objects of interest are classified in order of the user's interest level and provided. A method for detecting an object of interest based on 3D gaze point information and providing user visual perception metadata.
According to claim 13,
In the step (e), the location of the user's current gaze point, the object of interest corresponding to the gaze point, and the change of the gaze point for the change of the object of interest over time are stored in a bounding box corresponding to each object of interest and each object of interest. A method of detecting an object of interest based on 3D gazing point information and providing user visual perception metadata, characterized in that providing it in the form of a movement path indicating a change in .
According to claim 23,
The movement route display shows the change in the gaze point for the change of the object of interest over time, and the location of the gaze point from the user's gaze point for the n-1th object of interest to the user's gaze point corresponding to the n-th object of interest. A method of detecting an object of interest based on 3D gaze point information and providing user visual perception metadata, characterized in that it is provided to indicate a change.

Images