KR20150028589A - Apparatus and Method for Providing Object Information - Google Patents

Abstract

An embodiment of the present invention provides a method for providing object data and a device thereof. The method for providing object data in an image includes: an image receiving step of receiving a first image taken by a first camera; an image storage step of storing the first image received from the first camera and an image taken by one or more second cameras; an object extraction step of extracting a first object from the first image received from the first camera; an image extraction step of extracting one or more frames in which a similar object with a similarity equal to or greater than a certain threshold value to the first object is taken from the image stored in the image storage unit; an image analysis step of calculating an object data reliability for each of the first object and the similar objects in the frames and extracts a representative frame between the first image and the frames based on the object data reliability; and an object data analysis step of generating analysis data for the object from the representative frame.

Description

[0001] APPARATUS AND METHOD FOR PROVIDING OBJECT INFORMATION [0002]

An embodiment of the present invention relates to an apparatus and method for providing object information through information analysis of objects in an image. More particularly, the present invention relates to a method and apparatus for extracting an optimal image by calculating object information reliability of an object detected in an image, detecting object information in the extracted image, And a method for providing the same.

The contents described in this section merely provide background information on the embodiment of the present invention and do not constitute the prior art.

In recent years, there have been more and more attempts to promote products and brands to users by using digital signage, billboards, and closed circuit television ("CCTV") in places where the floating population such as department stores, shopping malls,

In the same context as this trend, there has been a demand for a customized advertisement providing information suitable for users by analyzing the user's gender and age range. However, due to the limitation of the information providing device, It is difficult to provide accurate and personalized information service through the Internet.

Conventional general methods to meet this need provide analysis results by analyzing the age and sex of the objects in the images taken by digital signage, billboards, etc. This means that when the object is photographed at a billboard, a digital signage, Distance limitation, and so on, it is difficult to accurately analyze because of low face size, angle, sharpness, surrounding illumination and camera resolution, and it is impossible to provide accurate real time information. .

According to an aspect of the present invention, there is provided a method of detecting an object in an image, the method comprising: extracting an optimal image by calculating object information reliability of an object detected in the image, And an object information providing apparatus and method for providing analyzed object information.

According to an aspect of the present invention, there is provided an apparatus for providing object information in an image, the apparatus comprising: an image receiving unit for receiving a first image photographed by a first camera; An image storage unit for storing a first image received from the first camera and an image captured by the at least one second camera; An object extraction unit for extracting a first object from a first image received from the first camera; An image extracting unit for extracting at least one frame in which a similar object having a degree of similarity higher than a predetermined threshold value with the first object is captured, from an image stored in the image storing unit; An image analyzer for calculating object information reliability for each of the similar objects for the first object and the one or more frames and for extracting representative frames from the first image and the one or more frames based on the object information reliability; And an object information analyzer for generating analysis information on the object from the representative frame.

In addition, the image extracting unit may include extracting the one or more frames from the image photographed by the second camera.

The object information analyzing unit extracts a face image of the object with respect to the object from the representative frame and generates at least one analysis information among the age and gender information of the object using the face image. To provide an object information providing apparatus.

Preferably, the object information providing apparatus stores the face image, extracts a face image having a degree of similarity equal to or greater than a predetermined threshold value from the previously stored plurality of face images, and transmits the extracted face image to the object information analyzing unit It is preferable to further comprise a part.

In addition, the object information providing apparatus may further include: a frame extracting unit that extracts a frame having a motion of an mouth portion of the similar object in the one or more frames before and after a predetermined time, Extracting voice information of the similar object, and transmitting the extracted voice information to the image analyzing unit.

 The object information providing apparatus may further include a product analyzing unit for extracting a type or trademark of a product worn by the similar object in the one or more frames.

In addition, the object information providing apparatus may further include a content linking unit for transmitting the analysis information to a content service providing apparatus for providing at least one content of an advertisement, a game, and an application to a user.

The image extracting unit may extract the similar object having a similarity degree of a predetermined threshold value or more based on a result of comparing at least one of at least one of pixel, color, and outline with the first object, And extracting it from the image.

In addition, the object information reliability may include an object characteristic information value based on the face size, the face angle, and the face sharpness of the first object and the similar object, and the first camera and the similar object, Which is determined according to the camera characteristic information value based on the distance and resolution between the first and second cameras.

In addition, the object characteristic information value may include a value proportional to a face size, a face angle, and a face sharpness for the first object and the similar object.

In addition, the camera characteristic information value may be inversely proportional to a distance between a first camera that captures the first object and a second camera that captures the similar object, and the first camera captures the first object, Which is proportional to the resolution of the second camera which has captured the image.

Preferably, the method further includes the steps of: receiving an image photographed by the first camera; An image storage step of storing a photographed image received from the first camera and an image photographed by the at least one second camera; An object extracting step of extracting a first object from the photographed image received from the first camera; An image extracting step of extracting at least one frame in which a similar object having a degree of similarity equal to or greater than a predetermined threshold value with the first object is captured, from an image stored in the image storing unit; An image analyzing process for calculating object information reliability for the first object and the similar object for the one or more frames to select representative frames among the captured image and the one or more frames; And an object information analyzing step of extracting a face image of the object in the representative frame and generating at least one analysis information of the age and gender information of the object using the face image.

In addition, a computer-readable medium storing a program for causing a computer to execute: an image receiving process for receiving a first image photographed by a first camera; An image storage step of storing a first image received from the first camera and an image captured by the at least one second camera; An object extracting step of extracting a first object from a first image received from the first camera; An image extracting step of extracting at least one frame in which a similar object having a degree of similarity equal to or greater than a predetermined threshold value with the first object is captured, from an image stored in the image storing unit; An image analyzing process of calculating object information reliability for each of the similar objects for the first object and the one or more frames and extracting a representative frame from the first image and the one or more frames based on the object information reliability; And a computer-readable recording medium storing a program for executing an object information analysis process for generating analysis information on the object from the representative frame.

As described above, according to the embodiment of the present invention, the object information providing apparatus has an advantage that it can extract an optimal frame for object information analysis by calculating object information reliability from an image of an object captured through a camera or the like .

According to the present invention, there is an effect that more accurate object information analysis can be performed through voice recognition of an object, recognition of a type of a product that an object is detached or possessed, and recognition of a brand or a trademark of a product in consideration of object face recognition limitations in a camera.

The object information providing device overcomes the difficulty of extracting and analyzing the object information due to low sharpness or resolution of the object image captured by the digital signage and the billboard, and by using the image collected or stored by the plurality of other cameras, It is possible to provide accurate object information through optimal combination and selection of advertisement information, and it can be widely used in advertising service field and marketing field by providing customized information.

In addition, it is possible to provide an infrastructure for building a new model for providing object information by effectively providing object analysis information to devices providing contents services such as advertisement, games, and applications, thereby creating new contents and creating new demands Economic effect can be achieved.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description of the invention given above, serve to provide a further understanding of the technical idea of the present invention. And should not be construed as limiting.
FIG. 1 is a block diagram illustrating an overall configuration of an object information providing apparatus according to an embodiment of the present invention.
2 is a block diagram illustrating a configuration of an object information providing apparatus according to an embodiment of the present invention.
3 is a flowchart illustrating a process of providing object information through data transmission / reception between an object information providing apparatus and a camera and a contents service providing apparatus according to an embodiment of the present invention.
4 is an example of an equation for calculating object information reliability in order to select an optimal representative frame from the image frames extracted from the object information apparatus according to an embodiment of the present invention.
FIG. 5 is a graph showing the relationship between object voice information, the type of a product and the type of a product detected from the object, in order to select an optimal representative frame from the image frames extracted from the object information apparatus according to an exemplary embodiment of the present invention. Is an example of a formula for summing the corresponding variables.

Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. It should be noted that, in adding reference numerals to the constituent elements of the drawings, the same constituent elements are denoted by the same reference numerals even though they are shown in different drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

FIG. 1 is a block diagram illustrating an overall configuration of an object information providing apparatus 100 according to an embodiment of the present invention. Referring to FIG. 1, a configuration of an object information providing apparatus 100 according to an exemplary embodiment of the present invention will be briefly described below.

The object information providing apparatus 100 according to an embodiment of the present invention is connected to the user 200 through the network 400 such as a wired or wireless network with the camera 400 and the content service providing apparatus 300.

The object information providing apparatus 100 according to an embodiment of the present invention is implemented in a server form or the like and each component of the object information providing apparatus 100 described below is a conceptual classification for distinguishing logical functions of the server The object information providing apparatus 100 mutually inputs and outputs various data for providing object information to the camera 400 and the contents service providing apparatus 300. [ Here, the camera 400 may include a digital signage, a CCTV camera, a surveillance camera, and the like that can take a path such as appearance, movement, and departure of an object, And transmits and receives video and data to and from the object-information providing apparatus 100. However, the present invention is not limited to this, and if it is possible to transmit and receive video and data such as a communication network according to a local area network or a communication base station installed in a predetermined company, But not limited to, any network.

Meanwhile, the content service providing apparatus 300 may be implemented in a form of a server for providing at least one content of an advertisement, a game, and an application to a user.

The object information providing apparatus 100 according to an embodiment of the present invention receives an image shot from a camera 400, detects an object in the received image to analyze object information, and provides the user with information such as an advertisement, a game, Means a device that performs the function of transmitting or providing the object analysis information to the service device that provides the content. Therefore, if the function is performed, the object information is not limited to the name, but may correspond to the object information providing device 100 It is self-evident.

Hereinafter, a specific configuration of the object information providing apparatus 100 will be described with reference to FIG. 2, which is a block diagram illustrating the configuration of an object information providing apparatus 100 according to an embodiment of the present invention.

An object storage unit 120, an object extraction unit 130, an image extraction unit 140, and an object extraction unit 140. The object extraction unit 130 extracts an object of interest from the object information providing apparatus 100 according to an embodiment of the present invention. An image analyzing unit 150, an object information analyzing unit 160, a database unit 170, a statistics unit 180, a content linking unit 190, a voice analyzing unit 121, and a product analyzing unit 123 And the like.

In addition, the object extracting unit 130 may include an image separating unit 131 and an object tracking unit 133.

The image receiving unit 110 of the object information providing apparatus 100 according to an exemplary embodiment of the present invention is connected to the camera 400 and receives a photographed image. Here, the camera 400 may include a plurality of cameras such as a digital signage, a CCTV, etc., and the photographed image is an image photographed by a plurality of cameras, Frame can be included.

The image storage unit 120 receives and stores a first image photographed by the first camera 410 through the image receiving unit 110 or directly receives and stores a photographed image from the second camera 420 And the stored image is used for frame tracking or image analysis of the image according to time.

The object extracting unit 130 extracts the first object from the photographed image received from the first camera 410. Here, the first camera 410 refers to a digital signage and a photographing unit of a billboard.

The object extracting unit 130 may include an image separating unit 131 and an object tracking unit 133. The image separator 131 separates the background from the foreground, which uses the Gaussian model of the pixel. Each pixel has a predetermined number of gaussian nodes. Each Gaussian node has mean, variance, and weight. The Gaussian model of the pixel is learned until the initial time after the start of the program. At this time, the average, variance, and weight are determined, and the node with the highest weight is designated as the background node.

When the above-described learning ends and an image frame comes in, each pixel is discriminated as belonging to one of the learned geosynchronous nodes, the average and variance of the corresponding geosynchronous node are updated, and the weights of all the geosynchronous nodes are updated .

Also, if a pixel belongs to a gaussian node other than a background node, the pixel becomes a foreground pixel. For example, when a person passes, a pixel corresponding to a person will not belong to a background node, and thus becomes a foreground pixel. A set of these foreground pixels will correspond to a person, and the foreground can be separated from the background through this process.

When the background separating unit 131 separates the background and foreground, the object tracking unit 133 traces the object. This uses the foreground image (background: 0, foreground: 1). A histogram and an object shape of a moving object corresponding to the foreground in the previous frame and a moving object corresponding to the foreground in the foreground image of the current frame are compared with each other, Identify the same object and track the object. For comparison of the color information, information such as a histogram, an object shape, and the like can be used.

The image extracting unit 140 extracts one or more frames captured from the first object extracted from the object extracting unit 130 and a similar object having a degree of similarity equal to or greater than a predetermined threshold value from the image stored in the image storing unit 120 . Here, the image picked up by the first camera 410 is received and the object extracted by the object extracting unit 130 is compared with a predetermined threshold value based on a result of comparison between at least one of the pixel, The similar object having the above similarity can be extracted from the image storage unit 120 in which the photographed image of the second camera 420 is stored.

The image analyzer 150 calculates the object information reliability for the first object and the similar object for one or more frames, and extracts an optimal representative frame from the stored images. Here, the object information reliability is determined based on the object characteristic information value based on the face size, the face angle and the face sharpness of the first object and the similar object, and the object characteristic information based on the position of the first camera 410, Can be calculated by summing the camera characteristic information values. For example, the object property information value is calculated as a sum of values obtained by multiplying each variable corresponding to a face size (Size), a face angle (Angle) and a face visibility (Visibility) of the first object and the similar object by a predetermined weight . Here, the face angle means the angle between the direction of the face of the object and the line of sight directed by the camera, and the face sharpness can mean the sharpness of the eye, nose, mouth, and face contour of the object. The camera characteristic information value is inversely proportional to the position or distance between the first camera 410 photographing the first object and the second camera 420 photographing the similar object, 2 camera 420 by multiplying each variable proportional to the resolution of the camera 420 by a predetermined weight.

Here, in order to overcome the limitation of face recognition of an object when the object information reliability is calculated by the image analyzer 150, the object's voice or object is analyzed by receiving information of the type of product worn or possessed by the object, . At this time, the voice analyzing unit 121 extracts information on the voice of the object, and the product analyzing unit 123 extracts the type, brand, or trademark of the item worn or possessed by the object in the product analyzing unit 123, .

The speech analyzing unit 121 extracts a frame having a motion of the mouth portion of a similar object around a predetermined time in one or more frames of the image and generates a voice corresponding to the time of the frame having the mouth portion motion And extracts voice information of a similar object at the start and end of a frame having a mouth motion, and transmits the extracted voice information to the image analysis unit 150.

The product analysis unit 123 extracts the types, brands, and brands of the products worn or owned by the similar objects in one or more frames, and transmits the extracted types, brands, and brands to the image analysis unit 150.

The database unit 170 stores a face image of the extracted object and extracts a face image having a degree of similarity higher than a predetermined threshold value from the object face image extracted from a plurality of previously stored object face images, As shown in FIG. This has the advantage that it can be used for face gender, age discrimination, or analysis of other objects by storing the object face image and constructing it as a database. For example, if the face image of the extracted object is difficult to analyze due to defects such as face size, face angle, face sharpness, camera illumination, etc., it is difficult to analyze the object information such as face recognition and sex and age. Can be looked up in the database unit 170 and used for object gender and age range analysis. As the accumulation amount of database increases after a certain period of time, the accuracy of object face recognition and object information analysis is improved.

The object information analyzing unit 160 extracts a face image of the object from the optimal representative frame extracted through the image analyzing unit 150, and analyzes at least one of the age and gender information of the object using the face image of the object As shown in FIG.

The statistics unit 180 numerically stores and stores object analysis information analyzed through the object information analysis unit 160 and transmits the object analysis information to the content link unit 190.

The content linking unit 190 transmits the object analysis information to the content service providing apparatus 300 that provides at least one of the advertisement, game, and application to the user. At this time, the object analysis information may be received through the statistics unit 180 which statistically analyzes the object analysis information, or may be received directly by the object information analysis unit 160 without passing through the statistics unit 180.

3 is a flowchart illustrating a process of providing object information through data transmission / reception between an object information providing apparatus and a camera and a contents service providing apparatus according to an embodiment of the present invention.

Hereinafter, a detailed procedure of the object information providing method according to the present invention will be described with reference to FIG. 3, which is a flowchart illustrating an object information providing method according to an embodiment of the present invention.

According to an embodiment of the present invention, an object information providing method in an image photographed by a camera includes the steps of: capturing an object through a first camera, which is a digital signage (S310) 2 camera image is received and stored (S320).

The first object is extracted from the image captured by the first camera in operation S330 and the extracted one or more frames of the similar object having a degree of similarity equal to or greater than a predetermined threshold value are extracted from the extracted image in operation S340.

The representative frame is extracted by calculating the reliability of the object information among the at least one frame (S350), the face image of the object is extracted from the representative frame, and the age and gender information of the object is analyzed (S360).

The object information analysis result is statistically stored (S370), and the statistical data is transmitted to the content linking unit and transmitted to the content service providing apparatus (S380).

4 is an example of an equation for calculating object information reliability in order to select an optimal representative frame from the image frames extracted from the object information apparatus according to an embodiment of the present invention.

FIG. 5 is a graph showing the relationship between object voice information, the type of a product and the type of a product detected from the object, in order to select an optimal representative frame from the image frames extracted from the object information apparatus according to an exemplary embodiment of the present invention. Is an example of a formula for summing the corresponding variables.

Hereinafter, an object information reliability calculation method will be described with reference to FIGS. 4 and 5, which are object information reliability calculation equations according to an embodiment of the present invention.

In Equation 4, Accuracy means the value of the object characteristic information indicating the degree of contribution of the extracted image to object analysis accuracy such as object gender and age, and Adequacy is a measure for evaluating suitability of the camera image, .

The object characteristic information value as the accuracy is set to a predetermined weight (e.g., w1, w2) for each variable corresponding to the object face size (f (size), face angle and face visibility) w2, w3). Here, the sum of the weights w1 and w2 and w3 may be one.

The camera characteristic information value as an adequacy is inversely proportional to the distance between the first camera 410 and the second camera 420 (f (1 / distance)) and the first camera 410 and the second camera 420 (Weight) (for example, w4, w5), and the sum of the weights w4 and w5 may be 1. The sum of the weights w4 and w5 may be 1,

The object information reliability Si is calculated as the sum of values obtained by multiplying each variable corresponding to the object characteristic information value and the camera characteristic information value Adequacy by a predetermined weight (e.g., w6, w7) The sum of the weights w6 and w7 may be one.

In order to overcome the face recognition limit of the object when the object information reliability (Si) is calculated, the equation of FIG. 5 may include an object voice information value, a brand information value of a product worn or owned by the object, Values are summed and calculated.

In FIG. 5, the object information reliability (Si +) is calculated based on the object information reliability Si, object voice information value f (voice), product brand information value f (brand) Is calculated as the sum of values obtained by multiplying each variable corresponding to the value f (goods) by a predetermined weight (e.g., w8, w9, w10, w11). Here, the sum of the weights w8, w9, w10 and w11 may be one.

In describing the present invention, each component of the object information providing apparatus 100 according to an embodiment of the present invention should be a logically separated component rather than a physically separated component as mentioned briefly above .

That is, since each configuration corresponds to a logical component for realizing the technical idea of the present invention, even if each component is integrated or separated, if the functions performed by the logical configuration of the present invention can be realized, It is to be understood that any component that performs the same or similar function should be interpreted as falling within the scope of the present invention regardless of the consistency of the name.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention.

Therefore, the embodiments disclosed in the present invention are not intended to limit the scope of the present invention but to limit the scope of the present invention.

The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

As described above, the present invention overcomes the difficulty of extracting and analyzing object information due to low sharpness or resolution of object images photographed in digital signage, billboards, etc., It is possible to provide accurate object information through optimal combination and selection of object information analysis, and it is a useful invention that is expected to be widely used in the field of advertising service and marketing through customized information provision in the future.

100: Object information providing apparatus 140:
200: user 150: image analysis unit
300: Content service providing apparatus 160: Object information analyzing unit
400: camera 170:
410: First camera 180: Statistics section
420: second camera 190: content interlocking unit
450: Nth camera 121: Voice analysis unit
500: network network 123: product analysis section
110: image receiving unit 131: image separating unit
120: Image storage unit 133: Object tracking unit
130:

Claims (13)

An apparatus for providing object information in an image,
An image receiving unit for receiving a first image photographed by a first camera;
An image storage unit for storing a first image received from the first camera and an image captured by the at least one second camera;
An object extraction unit for extracting a first object from a first image received from the first camera;
An image extracting unit for extracting at least one frame in which a similar object having a degree of similarity higher than a predetermined threshold value with the first object is captured, from an image stored in the image storing unit;
An image analyzer for calculating object information reliability for each of the similar objects for the first object and the one or more frames and for extracting representative frames from the first image and the one or more frames based on the object information reliability; And
An object information analyzer for generating analysis information on the object from the representative frame;
Wherein the object information providing apparatus comprises: The image processing apparatus according to claim 1,
And extracts the one or more frames from the image photographed by the second camera. The apparatus of claim 1, wherein the object information analyzer comprises:
Extracts a face image of the object with respect to the object from the representative frame, and generates at least one analysis information of the age and gender information of the object using the face image. 4. The apparatus according to claim 3,
Further comprising a database unit storing the face image and extracting a face image having a degree of similarity equal to or greater than a predetermined threshold value from a plurality of previously stored face images and transmitting the extracted face image to the object information analyzing unit. Device. The apparatus according to claim 1,
Extracting a frame in which the mouth portion of the similar object moves with a predetermined time before and after the one or more frames, extracting audio information of the similar object corresponding to the time of the frame in which the mouth portion is in motion, Analyzing unit for analyzing the object information, and transmitting the object information to the analysis unit. The apparatus according to claim 1,
Further comprising a product analysis unit for extracting a type or a trademark of a product worn by the similar object in the one or more frames. The apparatus according to claim 1,
A content service providing apparatus for providing at least one content of an advertisement, a game, and an application to a user,
The object information providing apparatus further comprising: The image processing apparatus according to claim 1,
Characterized in that the similar object having a degree of similarity equal to or greater than a predetermined threshold value is extracted from the photographed image of the second camera based on a result of comparison of at least one of the pixel, color and outline with the first object. Information providing device. 2. The method of claim 1,
A distance between a first camera that photographs the first object and a second camera that photographs the similar object, and a second camera that photographs the first object and the similar object based on the face size, the face angle, and the face sharpness, Wherein the object information is determined according to a camera characteristic information value based on the resolution. 10. The method of claim 9,
Wherein the first object and the similar object are proportional to a face size, a face angle and a face sharpness of the first object and the similar object. 10. The method of claim 9,
Wherein the distance between the first camera that photographs the first object and the second camera that photographs the similar object is proportional to the resolution of the first camera that photographs the first object and the second camera that photographs the similar object, The object information providing apparatus comprising: A method for providing object information in an image,
An image receiving step of receiving a first image photographed by a first camera; An image storage step of storing a first image received from the first camera and an image captured by the at least one second camera; An object extracting step of extracting a first object from a first image received from the first camera; An image extracting step of extracting at least one frame in which a similar object having a degree of similarity equal to or greater than a predetermined threshold value with the first object is captured, from an image stored in the image storing unit; An image analyzing process of calculating object information reliability for each of the similar objects for the first object and the one or more frames and extracting a representative frame from the first image and the one or more frames based on the object information reliability; And an object information analysis process for generating analysis information on the object from the representative frame
The object information providing method comprising: On the computer,
An image receiving step of receiving a first image photographed by a first camera; An image storage step of storing a first image received from the first camera and an image captured by the at least one second camera; An object extracting step of extracting a first object from a first image received from the first camera; An image extracting step of extracting at least one frame in which a similar object having a degree of similarity equal to or greater than a predetermined threshold value with the first object is captured, from an image stored in the image storing unit; An image analyzing process of calculating object information reliability for each of the similar objects for the first object and the one or more frames and extracting a representative frame from the first image and the one or more frames based on the object information reliability; And analyzing the object information to generate analysis information for the object from the representative frame.

Images