KR101314687B1 - Providing device of eye scan path and mehtod for providing eye scan path - Google Patents

Abstract

Disclosed is a gaze path providing apparatus. The apparatus for providing a gaze path may include: an input unit for receiving a plurality of images photographed at spaced locations and a plurality of sound sources heard at spaced locations; a position determination unit for analyzing a plurality of sound sources to determine a location of a sound source; and a plurality of images A plurality of mono protrusion maps are generated for each of the plurality of mono protrusion maps, and a protrusion protrusion generation unit generating a dynamic protrusion map by using the generated plurality of mono protrusion maps. And a gaze path generator for generating a gaze path for the gaze path, and an output unit for outputting the generated gaze path.

Description

Device for providing a gaze path and method for providing a gaze path {PROVIDING DEVICE OF EYE SCAN PATH AND MEHTOD FOR PROVIDING EYE SCAN PATH}

The present invention relates to a gaze path providing device and a gaze path providing device, and more particularly, to a gaze path providing device capable of providing a gaze-like movement path similar to a human based on visual / audio fusion information.

Sensor technology began by mimicking human sense organs. Recently, the importance of active artificial vision system among the sensor technology has been highlighted.

However, many artificial visual systems developed so far focus on detecting and recognizing specific objects for the input image. There was a drawback to not having a solution to the problem at the beginning.

On the other hand, the human visual system has a robust visual processing performance against various and complex environmental factors. The human visual system has an efficient information processing mechanism that selectively and sequentially processes necessary information instead of processing information in a complex visual system collectively. This is achieved by the selective attention function, which allows visual cells with only simple functions to process complex image information quickly and quickly in real time, and the active visual function, which can freely move the eyes to the target object.

Therefore, in recent years, the importance of developing visual system models based on brain information processing and biological information processing mechanisms in human visual systems has been highlighted to overcome the limitations of existing artificial visual systems and to efficiently process complex real world visual information. It is becoming.

Accordingly, it is an object of the present invention to provide a gaze path providing apparatus capable of providing a gaze movement path similar to a human based on visual / audio fusion information.

In accordance with an embodiment of the present invention, a path path apparatus includes: an input unit configured to receive a plurality of images photographed at spaced locations and a plurality of sound sources heard at spaced locations, and analyzing the plurality of sound sources; Position determination unit for determining the position of the projection map generation unit for generating a plurality of mono protrusion map for each of the plurality of images, and generating a dynamic protrusion map using the generated plurality of mono protrusion map, the generated A gaze path generation unit configured to generate gaze paths for the plurality of images based on the dynamic protrusion map and the determined sound source position, and an output unit to output the generated gaze paths.

In this case, the protrusion map generating unit may include an image information extracting unit extracting at least one image information of brightness, edge, symmetry, and complementary colors of the input image, and a center-periphery difference of the extracted image information. a CSD processing unit for performing at least one feature map among a brightness feature map, a direction feature map, a symmetry feature map, and a color feature map by performing a surround difference (CSD) and normalization process, and an independent component of the output feature map It includes an ICA processing unit for generating a mono protrusion map by performing an independent component analysis.

In this case, the protrusion map generation unit may further include a merge unit that generates a dynamic protrusion map by merging a plurality of mono protrusion maps generated by the ICA processing unit.

On the other hand, the protrusion map generation unit, it is preferable to generate the plurality of mono protrusion map using a biological-based selective ambient concentration model.

The gaze path generation unit may reinforce or suppress a plurality of salient points included in the generated dynamic salient map based on the determined sound source position to prioritize the salient points. It is desirable to create a gaze path in accordance with the given priority.

On the other hand, it is preferable that the input unit receives a plurality of images and a plurality of sound sources at predetermined time intervals.

On the other hand, the gaze path providing method in the gaze path providing apparatus according to the present embodiment, the step of receiving a plurality of images taken from the spaced position and a plurality of sound sources heard from the spaced position, by analyzing the plurality of sound sources Determining a position of the sound source, generating a plurality of mono protrusion maps for each of the plurality of images, generating a dynamic protrusion map using the generated plurality of mono protrusion maps, and generating the generated dynamic protrusions Generating a gaze path for the plurality of images based on a map and the determined sound source position, and outputting the generated gaze path.

In this case, the generating of the plurality of mono protrusion maps may include extracting at least one image information of brightness, edge, symmetry, and complementary colors for each of the plurality of input images, and generating a center of the extracted image information. Performing a center-surround difference (CSD) and normalization process to output at least one feature map of a brightness feature map, a directional feature map, a symmetry feature map, and a color feature map, and the output feature And performing an independent component analysis on the map to generate a mono protrusion map.

In this case, in the generating of the dynamic protrusion map, it is preferable to generate the dynamic protrusion map by merging the generated plurality of mono protrusion maps.

Meanwhile, in the generating of the plurality of mono protrusion maps, it is preferable to generate the plurality of mono protrusion maps using a biologically-based selective ambient concentration model.

The generating of the gaze path may include reinforcing or suppressing a plurality of salient points included in the generated dynamic salient map based on the determined sound source position, thereby generating a plurality of salient points. It is preferable to give priority and generate a gaze path according to the given priority.

On the other hand, in the receiving step, it is preferable to receive a plurality of images and a plurality of sound sources at a predetermined time period.

Therefore, the apparatus for providing a gaze path and the method for providing a gaze path according to the present embodiment fuse the visual and auditory information and consider the dynamic movement of the image and the position of the sound source at the same time. You can choose.

1 is a block diagram showing the configuration of a gaze path providing apparatus according to an embodiment of the present invention;
FIG. 2 is a block diagram illustrating a specific configuration of the protrusion map generating unit of FIG. 1;
3 is a view for explaining the operation of the position determination unit of FIG.
4 is a view for explaining the operation of the protrusion map generation unit of FIG. 1, and
5 is a flowchart illustrating an operation of a gaze path method according to an exemplary embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a configuration of a gaze path providing apparatus according to an exemplary embodiment.

Referring to FIG. 1, the gaze path providing apparatus 100 according to the present exemplary embodiment may include an input unit 110, an output unit 120, a storage unit 130, a position determiner 140, and a protrusion map generator 150. The eyeball path generation unit 160 and the controller 170 may be configured. In the present embodiment, it will be described as being simply used as a device for identifying a user's gaze path, but the gaze path device according to the present embodiment may be implemented as one configuration of a vision system, a security system, and a surveillance system of a robot.

The input unit 110 receives a plurality of images photographed at spaced locations and a plurality of sound sources heard at spaced locations. In detail, the input unit 110 may receive a plurality of images photographed by a photographing device such as an external digital camera and an image reading apparatus (scanner). The input unit 110 may receive a plurality of sound sources through a microphone having a plurality of channels.

The output unit 120 may output the generated gaze path. In more detail, the output unit 120 may be implemented as a display device such as a monitor, and may display an image received through the input unit 110 and a gaze path for the corresponding image.

The storage unit 130 stores a plurality of input images and a plurality of input sound sources. In detail, the storage 130 may store a plurality of images and a plurality of sound sources input by the input unit 110 described above. The storage unit 130 may temporarily store the protrusion map generated by the protrusion map generator 150 to be described later, and may temporarily store the gaze path generated by the gaze path generator 160 to be described later. The storage unit 130 may be a memory mounted in the gaze path providing apparatus 100, for example, a ROM, a flash memory or an HDD, and an external HDD or a memory card connected to the gaze path providing apparatus 100. For example, it may be a flash memory (M / S, xD, SD, etc.) or a USB memory.

The position determiner 140 analyzes a plurality of sound sources (eg, stereo sound sources) to determine the position of the sound source. In detail, the position determiner 140 may determine a position where the sound source is generated by analyzing phase differences between the plurality of input sound sources. Since this operation is a well known technique, a detailed description thereof will be omitted.

The protrusion map generation unit 150 generates a plurality of mono protrusion maps for each of the plurality of images, and generates a dynamic protrusion map using the generated plurality of mono protrusion maps. A detailed operation and configuration of the protrusion map generator 150 will be described later with reference to FIG. 2.

The gaze path generator 160 generates gaze paths for a plurality of images based on the generated dynamic protrusion map and the determined sound source position. In detail, the gaze path generator 160 may reinforce or process the plurality of protrusion points included in the dynamic protrusion map generated by the protrusion map generator 150 based on the sound source position determined by the position determiner 140. The suppression process may be performed to give priority to the plurality of protruding points, and generate a gaze path according to the given priority. Meanwhile, the function of the gaze path generator 160 may be implemented in the form of being integrated into the protrusion map generator 150.

The operation of the gaze path generator 160 may use a biological-based selective ambient concentration model. Here, the biologically based selective ambient concentration model is a model of some processes of human rejection structure and processing. It is divided into concept-driven processing using data-driven processing that reacts instantly to the input image and learned information Loses. Data-Driven Processing and Concepts Driven processing is a well-known technology and its detailed description is omitted.

The controller 170 controls each component of the gaze path providing apparatus 100. In detail, when a plurality of images and a plurality of sound sources are input through the input unit 110, the controller 170 may control the protrusion map generator 150 to generate a dynamic protrusion map for the plurality of input images. The position determination unit 140 may be controlled to determine sound source positions for the plurality of sound sources. The controller 170 may control the gaze path generator 160 to generate a gaze path of the user based on the generated dynamic protrusion map and the sound source position. The controller 170 may control the output unit 120 to display the generated gaze path.

Therefore, the gaze path providing apparatus 100 according to the present embodiment fuses the visual and audio information, and considers the dynamic movement of the image and the position of the sound source at the same time, thereby generating a gaze path with high reliability in selecting information. It becomes possible.

In the present embodiment, only the operation of displaying the generated gaze path through the output unit 120 has been described. However, in the implementation, the generated gaze path is stored in the storage unit 130, printed through a printing device, or in a specific device. It may also be implemented in the form of being transmitted.

FIG. 2 is a block diagram illustrating a specific configuration of the protrusion map generating unit of FIG. 1.

Referring to FIG. 2, the protrusion map generator 150 includes an image information extractor 151, a CSD processor 152, an ICA processor 153, and a merger 154.

The image information extractor 151 extracts image information about brightness I, edge E, and complementary colors RG and BY of the input image. Specifically, at least one image information among brightness, edge, symmetry, and complementary colors of the input image may be extracted based on R (Red), G (Green), and B (Ble) values of the input image.

The CSD processor 152 may generate a brightness feature map, a directional feature map, a symmetry feature map, and a color feature map by performing a center-surround difference (CSD) and normalization process on the extracted image information. .

The ICA processor 153 generates a mono salient map (SM) by performing independent component analysis on the output feature map.

The image information extracting unit 151, the CSD processing unit 152, and the ICA processing unit 153 generate a mono protrusion map for each image.

The merger 154 merges the plurality of mono protrusion maps generated by the ICA processor 153 to generate a dynamic protrusion map. In detail, the dynamic protrusion map may be generated by Equations 1 to 4 below.

Sp (v) is a bottom-up protrusion map without considering depth information, and L (sp.v.σ) is a Laplace equation as shown in Equation (2).

Whether the dynamic protrusion map as shown in FIG. 2 produces a protrusion map similar to the selective attention function of a human, the protruding protrusion regions may be regions that are of no interest to humans, or regions that need more attention. . This is because the protrusion map is generated using only primitive features such as complementary color and brightness, edge, and symmetry information. In order to solve this problem, it is possible to suppress / reinforce each of the protruding regions of the dynamic salient map by reflecting it in the auditory information. This behavior can be modeled with Fuzzy ART neural network.

In FIG. 2, the protrusion map generating unit 150 is illustrated and described as including two image information extracting units 151, two CSD processing units 152, and two ICA processing units 153. When there are three or more images, the image information extracting unit 151, the CSD processing unit 152, and the ICA processing unit 153 corresponding to the number of input images may be implemented. In addition, in the present embodiment, only the generation of a mono protrusion map in parallel for a plurality of images has been described, but a mono protrusion map for each of the plurality of images is generated using one information extractor, a CSD processor, and an ICS processor. It can also be implemented in the form.

3 is a view for explaining an operation of the position determiner of FIG. 1.

Referring to FIG. 3, when two sound sources (eg, stereo sound sources) are input to the input unit, the position determination unit 140 analyzes the spectrum of each sound source, and through the spectrum of each analyzed sound source, The location generated can be predicted.

FIG. 4 is a diagram for describing an operation of the protrusion map generating unit of FIG. 1.

4, a plurality of protrusion maps 410 generated by the ICA processor 153, a dynamic protrusion map 420 generated by the merger 154, and a final protrusion map 430 are displayed.

The plurality of mono protrusion maps 410 are protrusions corresponding to each of a plurality of images input through the input unit 110.

The dynamic protrusion map 420 is a protrusion map generated by merging a plurality of mono protrusion maps output from the ICA processor 154.

The final protrusion map 430 is a protrusion map generated by performing a reinforcement and suppression operation on the protrusion region of the dynamic protrusion map 420 according to the position of the sound source determined by the position determiner 140.

5 is a flowchart illustrating an operation of a gaze path method according to an exemplary embodiment.

In operation S510, a plurality of images photographed at a spaced position and a plurality of sound sources heard at a spaced position are received. In detail, a plurality of images photographed by a photographing apparatus such as an external digital camera or an image reading apparatus (scanner) may be input. A plurality of sound sources may be input through a microphone having a plurality of channels.

The location of the sound source is determined by analyzing a plurality of sound sources (S520). Specifically, the position where the sound source is generated may be determined by analyzing the phase difference between the plurality of input sound sources.

In operation S530, a plurality of mono protrusion maps are generated for each of the plurality of images. Specifically, at least one image information of brightness, edge, symmetry, and complementary colors for each of the plurality of input images is extracted, and a center-surround difference (CSD) and normalization process for the extracted image information is performed. Outputs at least one feature map of a brightness feature map, a directional feature map, a symmetry feature map, and a color feature map, and generates a mono protrusion map by performing independent component analysis on the output feature map. can do.

A dynamic protrusion map is generated using the generated plurality of mono protrusion maps (S540). Specifically, the dynamic protrusion map may be generated by merging a plurality of generated mono protrusion maps.

A gaze path for a plurality of images is generated based on the generated dynamic protrusion map and the determined sound source position (S550). Specifically, reinforcement or suppression is performed on the plurality of protrusion points included in the generated dynamic protrusion map to give priority to the plurality of protrusion points, and according to the assigned priority. A gaze path may be generated.

The generated gaze path is output (S560). In detail, the gaze path generated through the display device such as a monitor may be output. In this case, the plurality of input images and the gaze paths of the plurality of images may be displayed together.

Therefore, in the gaze path providing method according to the present embodiment, the gaze path and the auditory information are merged to simultaneously consider the dynamic movement of the image and the position of the sound source, thereby making it possible to generate a gaze path with high reliability in selecting information. . The gaze path providing method as shown in FIG. 5 may be executed on the gaze path providing device having the configuration of FIG. 1 and may also be executed on the gaze path providing device having another configuration.

Although the above has been illustrated and described with respect to preferred embodiments of the present invention, the present invention is not limited to the above-described specific embodiments, and the present invention belongs to the present invention without departing from the gist of the present invention as claimed in the claims. Anyone of ordinary skill in the art can make various modifications, and such changes are within the scope of the claims.

100: gaze path providing apparatus 110: input unit
120: output unit 130: storage unit
140: position determination unit 150: protrusion map generation unit
160: gaze path generation unit 170: control unit

Claims (12)

In the gaze path providing apparatus,
An input unit configured to receive a plurality of images photographed at positions spaced at predetermined intervals and a plurality of sound sources heard at positions spaced at predetermined intervals;
A position determination unit determining the position of the sound source by analyzing the plurality of sound sources;
A protrusion map generation unit generating a plurality of mono protrusion maps for each of the plurality of images and generating a dynamic protrusion map using the generated plurality of mono protrusion maps;
A gaze path generation unit configured to generate gaze paths for the plurality of images based on the generated dynamic protrusion map and the determined sound source position; And
And an output unit for outputting the generated gaze path. The method of claim 1,
The protrusion map generation unit,
An image information extracting unit extracting at least one image information of brightness, edge, symmetry, and complementary colors of the input image;
A center-surround difference (CSD) and a normalization process on the extracted image information to output at least one feature map of a brightness feature map, a direction feature map, a symmetric feature map, and a color feature map A CSD processing unit; And
And an ICA processing unit for generating a mono protrusion map by performing independent component analysis on the output feature map. 3. The method of claim 2,
The protrusion map generation unit,
And a merging unit for generating a dynamic salient map by merging a plurality of mono salient maps generated by the ICA processing unit. The method of claim 1,
The protrusion map generation unit,
An apparatus for providing gaze paths, wherein the plurality of mono protrusion maps are generated using a biologically-based selective ambient concentration model. The method of claim 1,
The gaze path generation unit,
Based on the determined sound source position, reinforcement or suppression of a plurality of salient points included in the generated dynamic salient map gives priority to the plurality of salient points, and the assigned priority. Gaze path providing apparatus characterized in that for generating a gaze path according to. delete In the gaze path providing method in the gaze path providing apparatus,
Receiving a plurality of images photographed at positions spaced at predetermined time intervals and a plurality of sound sources heard at positions spaced at predetermined time intervals;
Analyzing the plurality of sound sources to determine a position of the sound source;
Generating a plurality of mono protrusion maps for each of the plurality of images;
Generating a dynamic protrusion map using the generated plurality of mono protrusion maps;
Generating a gaze path for the plurality of images based on the generated dynamic protrusion map and the determined sound source position; And
And outputting the generated gaze path. The method of claim 7, wherein
The generating of the plurality of mono protrusion maps may include:
Extracting at least one image information of brightness, edge, symmetry, and complementary colors for each of the plurality of input images;
A center-surround difference (CSD) and a normalization process on the extracted image information to output at least one feature map of a brightness feature map, a direction feature map, a symmetric feature map, and a color feature map step; And
And generating a mono protrusion map by performing independent component analysis on the output feature map. 9. The method of claim 8,
Generating the dynamic protrusion map,
And generating a dynamic protrusion map by merging the generated plurality of mono protrusion maps. The method of claim 7, wherein
The generating of the plurality of mono protrusion maps may include:
And generating the plurality of mono protrusion maps using a biologically-based selective ambient concentration model. The method of claim 7, wherein
The generating of the gaze path may include:
Based on the determined sound source position, reinforcement or suppression of a plurality of salient points included in the generated dynamic salient map gives priority to the plurality of salient points, and the assigned priority. The gaze path providing method of generating a gaze path according to the present invention. delete

Images