KR101914665B1 - Apparatus for providing image with additional information through automatic recognition of subject - Google Patents

Abstract

The present invention relates to an apparatus for providing an image displaying additional information through a subject automatic recognition function, comprising: a main body having a display unit on a front surface; a Kinect installed in the main body to acquire an image and audio on a front side; an information classification unit for acquiring information from the image acquired by the Kinect and classifying the same; an information suitability determination unit for receiving the information classified in the information classification unit, and determining information suitability according to a type and an environment of the classified information for identification as information that can be meaningfully used in a real-time reproduction environment of an image displayed through the display unit and for output of additional information; a database unit for storing a database for a plurality of additional information connected to the classified information; an output information selection unit for receiving the classified information determined to have suitability by the information suitability determination unit and related information on currently displayed additional information, determining a recognition range of a viewer and types of additional information that can be displayed on a real-time image, and selecting a type of additional information having suitability for a real-time expression on a current image among the determined types of the additional information from the database stored in the database unit; and an additional information output unit for outputting the additional information selected by the output information selection unit to synthesize the same with an image provided by the Kinect while satisfying predetermined output conditions.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an additional information display image providing apparatus,

More particularly, the present invention relates to an apparatus and method for providing an additional information display image through automatic recognition of a subject, and more particularly, To provide an additional information display image providing apparatus with a function of automatically recognizing a subject.

In recent years, a variety of kiosks have been installed and operated in advertisements for product or sales explanations at the store entrance or in the store, which contributes to the ease of use and the ability to convey information.

In the related art, Korean Patent Laid-Open No. 10-2013-0046867, entitled " Method of providing advertisement using a kiosk "has been proposed. When a call receiver installed in a kiosk installed in a public place is heard and a call waiting tone is transmitted, Sensing; Providing an advertisement to the kiosk and operating a timer at an operating server communicating with the kiosk when the call time is detected; When the call is terminated, terminating the advertisement provision and stopping the timer to calculate a call provision time; And the operation server includes imposing an advertisement charge on an advertiser of the provided advertisement based on the call providing time.

However, such a conventional art does not disclose a configuration for helping to increase the advertisement effect through providing interest and benefit of customers, and it has a problem that it is insufficient to attract public attention at the installation place.

In order to solve the problems of the prior art as described above, the present invention can output additional information such as subtitles, effects, sounds, etc. to a video of a participating customer obtained in the field without additional equipment or expert personnel, It is possible to provide an interest and a benefit to the surrounding audience including the audience, to easily allow anyone to watch his / her own video provided with the additional information, to participate in the event, to increase the meaning of the video, To maximize the value of

Other objects of the present invention will become readily apparent from the following description of the embodiments.

According to an aspect of the present invention, there is provided a display apparatus comprising: a main body having a display unit on a front surface thereof; A key knob installed to acquire a front image and audio in the main body; An information classifying unit for obtaining information from the video object acquired by the keynote; The information classification unit may receive the divided information and determine information suitability according to the type and environment of the classified information for identification and additional information output as information that can be used in a real time reproduction environment of the video through the display unit An information suitability judging unit; A database for storing a database of a plurality of additional information connected to the divided information; The information suitability determining unit receives the divided information determined to be conformable and the related information about the currently displayed additional information to determine the cognitive range of the viewer and the type of additional information that can be displayed in the real time video, An output information selection unit for selecting, in a database stored in the database unit, type of additional information for which a real-time representation is suitable for a current video among types of additional information; And an additional information output unit for outputting the additional information selected by the output information selection unit to synthesize the supplementary information selected by the output information selection unit to a video object provided by the keyboard while satisfying a predetermined output condition, An image providing apparatus is provided.

The information classification unit extracts the face from the video image photographed from the Kinect by the face extraction unit, extracts the voice by the audio extraction unit, extracts the motion by the motion extraction unit, And extracts a character or an image by the character / image extracting unit, thereby providing the information suitability determining unit with the face, the voice, the motion, the trademark, and the character or the image as the distinguished information with respect to the video.

The information suitability determining unit considers that the additional information is displayed on the screen for a predetermined period of time or output audio by the perceived time considering display unit in order to output identification and additional information as information that can be used in a real time broadcasting environment, Considering that the additional information is displayed within a predetermined time after the metabolism, gesture or trademark is exposed by the display unit, the information amount consideration display unit considers the position and size of the additional information to be displayed so as not to overlap with the object of the video object or to interfere with each other The consideration of the background sound of the video object and the effect sound of the additional information are not overlapped with each other, and it is considered by the recognizable consideration display section that the object image and the sound are separated and displayed in units of analyzing the video object. Can be processed within the cognitive range In consideration of the efficiency of data, weights are assigned to the type, processing speed, size, holding time, and category of additional information to determine information suitability according to the type and environment of the classified information, The priority information of the additional information to be displayed on the screen is determined and the remaining time and priority information of the currently outputted information are determined in real time to constitute a timeline, the additional information use history of the user is recorded, By storing the additional information list value on a user-by-user basis, preference information on the additional information list information can be reflected.

The event information additional information stored in the database unit is displayed on the screen of the display unit by the output information selection unit and the participant is displayed on the screen of the display unit by the keyboard, An event module for making a contact with the event additional information and issuing a coupon when the score obtained based on the frequency of contact with the event additional information according to the determined method exceeds a predetermined score; A notification unit for causing the participant to send a coupon issued by the event module to the mobile phone through which the participant inputs the telephone number through the input unit; And a communication unit for performing wireless communication with the cellular phone so that the coupon is transmitted to the cellular phone by the notifying unit.

Wherein the output information selection unit causes the event additional information to correspond to a request item of the predetermined advertiser, and the notification unit receives the image of the participant in which the event-use additional information is displayed from the event module, To the mobile phone, so that the participant can use the image.

According to the additional information display image providing apparatus through the automatic recognition of the object according to the present invention, additional information such as subtitles, effects, and sounds can be output to the video of participating customers acquired in the field without any additional equipment or expert personnel, It is possible to provide interest and benefit to the audience including the participating customers and to allow anyone to easily watch his or her own video provided with the supplementary information and to participate in the event, The advertisement effect can be maximized.

1 is a perspective view illustrating an apparatus for providing an additional information display image through a function of automatically recognizing a subject according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating an apparatus for providing an additional information display image through automatic recognition of a subject according to an embodiment of the present invention. Referring to FIG.
3 is a view for explaining an interaction of an additional information display image providing apparatus through a function of automatically recognizing a subject according to an embodiment of the present invention.
4 to 8 illustrate an image of a video image with additional information displayed by an additional information display image providing apparatus through an automatic recognition function of a subject according to an embodiment of the present invention.

The present invention is capable of various modifications and various embodiments, and specific embodiments are illustrated and described in detail in the drawings. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but is to be understood to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention, And the scope of the present invention is not limited to the following examples.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings, wherein like or corresponding elements are denoted by the same reference numerals, and redundant explanations thereof will be omitted.

FIG. 1 is a perspective view illustrating an apparatus for providing an additional information display image through an automatic recognition function of a subject according to an embodiment of the present invention. FIG. 2 is a perspective view of an additional information display image through an automatic recognition function of a subject according to an embodiment of the present invention. Fig.

1 and 2, an apparatus 100 for providing an additional information display image through a function of automatically recognizing a subject according to an embodiment of the present invention includes a main body 260, a key knot 170, an information classification unit 110, An information suitability determining unit 120, a database unit 130, an output information selecting unit 140, and an additional information output unit 150. [

The main body 260 is provided with a display unit 190 on the front surface 261 and a support table 262 for mounting on the ground. The main body 260 may be provided with a speaker (not shown) for outputting audio, or may be provided separately or integrally with the display unit 190. An information classification unit 110, an information suitability determination unit 120, a database unit 130, an output information selection unit 140, and an additional information output unit 140 are provided in the main body 260, A PC 270 in which a statistic calculating unit 160, an output controlling unit 180, a microcomputer 210, an event module 220, a notifying unit 230, and a communication unit 240, which will be described later, Can be accommodated in the inner space.

The key knob 170 is installed on the front surface of the main body 260 to acquire images and audio in front of the main body 260. The Kinect 170 includes a camera 171 for acquiring video, a microphone 172 for inputting audio, and a 3D depth sensor 173 for sensing the surrounding environment. The 3D depth sensor 173 is composed of, for example, a CMOS sensor and an infrared laser beam projector. When the infrared laser beam is irradiated, the reflected laser beam point is received by the CMOS sensor, and the distance per pixel is measured. So that the subject can be recognized.

The information classifying unit 110 extracts information from a video object obtained by the keyboard 170 and separates the face from the video object photographed by the keyboard 170 by the face extracting unit 111, Extracts the voice by the voice recognition unit 112 and extracts the operation by the operation extraction unit 113 and extracts the trademark by the trademark extraction unit 114. The character / The face, the voice, the operation, the trademark, and the character or the image to the information suitability judging unit 120 as separated information. The information classification unit 110 recognizes a subject such as an individual or a plurality of persons, objects, images, audio, and background appearing in a video object, divides the information of the corresponding object into stages, Analyze and classify according to the kind of information.

In the case of a type of information, when the subject is a person by the facial extracting unit 111 applying an image processing technique for extracting a specific image from the facial image, the facial position is determined and the facial main point By tracking the location information and the changing path. The voice can be obtained by determining whether music, a specific language, a word, or a sentence by the audio extracting unit 112 using a process technique of collecting audio signals from the video and converting the voice into text. The operation can be obtained by determining whether the specific type or gesture included in the database and the contents thereof are determined by the operation extracting unit 113 using the image process technique for recognizing the motion of the moving subject. A brand and a product together with a trademark by a trademark extracting unit 114 to which an image process technique for extracting a specific trademark, a brand, and a product stored in the database unit 130 from the image is applied. A character or an image can be obtained by analyzing a character image image by a character / image extracting unit 115 applying a process technique for extracting and converting text from an image process technique and judging whether it is a character or a QR code or an image have.

The information suitability judging unit 120 receives the information classified by the information classifying unit 110 that obtains information from the video object obtained from the keynote 170 and identifies the information as information that can be used in a real- The information suitability is judged according to the type and environment of the classified information for the additional information output.

In order to output identification information and additional information as information that can be used in a real-time broadcasting environment, the information suitability determining unit 120 is largely applied to two criteria, namely, a viewer's perception range and a technically processable range. Considering the processing within the perceptible time of the viewer, considering the display within the amount of perceptible reference data of the viewer, considering the display within the perceivable speed and effect of the viewer, The suitability of the information classified by the information classification unit 110 may be determined through a significant usability of the additional information connected thereto. More specifically, the information suitability determining unit 120 may consider that the additional information is displayed on the screen or output audio for a predetermined time, for example, at least one second, by the perceived time considering display unit 121, Gesture or trademark is exposed, the additional information may be displayed within a predetermined period of time, for example, within a maximum of 3 seconds, and the information amount consideration display unit 123 may display the information on the basis of the position and size of the additional information, The background sound of the video and the sound of the additional information do not overlap with each other, and the recognizable consideration display unit 124 separates the image and sound of the object in the video object into units And the processing time considering display section 125 displays the processing time in the viewer's perception range Li can take into account the efficiency of the available data.

The information suitability determining unit 120 assigns weights to the type, processing speed, size, holding time, and category of the additional information to determine information suitability according to the type and environment of the classified information, Determines the priority of the additional information to be displayed on the screen, determines the remaining time and priority information of the currently outputted information in real time to construct a time line (metadata), records the additional information usage details of the user , And the frequently used additional information list value is stored in the database unit 130 for each user, so that the efficiency of the information processing can be improved by reflecting the preference information. The conformity determination and information display order by the information suitability determining unit 120 is such that the displayed information and the newly displayed information are transmitted, the additional information usable for the displayed information is selected, and the weight of the selected additional information Determines a priority order, and transmits the information list to the viewer in order to output the additional information to the viewer in consideration of the information displayed on the current movie. In addition, the priority among the supplementary information connected to each of the plurality of pieces of information on the video object can be determined by weight. For example, the supplementary supplementary information has priority over the supplementary supplementary information, the Korean subtitle has priority over the foreign language subtitle, The higher priority is given to the larger one in the range and the higher the degree of similarity is given priority when the degree of similarity or similarity between the divided information of the video and the additional information is expressed in numerical value or percentage. Alternatively, the information suitability determining unit 120 may limit the amount of additional information such as an image or a video to a predetermined range, for example, not exceeding 70%. In case of additional information corresponding to music, 10 seconds, for example, about 5 seconds.

Referring to FIG. 3, the database unit 130 stores a database of a plurality of pieces of additional information connected to divided information. The database unit 130 stores a unique ID, output position, output duration, amount of information, And whether it is set or not, and may include customer information and purchase histories in the store as needed. Here, the unique ID is a unique identification ID of each additional information, and is used for identification and tracking. The type of information is a record of what information corresponds to face, voice, motion, text, image, and trademark. The output type includes information on the type of information to be displayed on the screen, subtitle, effect, sound, image, advertisement, and other functions such as face detection. The output position and output duration are the duration (possible) time of the output, the position information possible value and the attribute. The set whether or not the plurality of pieces of additional information can be combined and used together so as to have an association or the same concept.

The additional information display image providing apparatus 100 through the automatic recognition function of the subject according to the present invention controls to provide a video image photographed from the keynote 170 in real time broadcasting, An output control unit 180 for synthesizing the supplementary information provided by the supplementary information output unit 150 and providing the supplementary information to a video object in real time by broadcasting, a command input required for the operation control of the output control unit 180, An input unit 250 for performing a control operation, and a microcomputer 210 for performing an overall control operation. Here, the input unit 250 may be a touch panel integrated with the display unit 190.

3, in the client system, the additional information output unit 150 and the output control unit 180 enable the effect of the additional information exposed on the screen of the display unit 190 and the interaction of the subject, (Effects A and B) are applied to the object (subjects A and B) through additional information (real-time rendering, physical application, world coordinate, etc.) The direction, effect, and other values that are output due to the motion of the subject or other object motion can be varied in real time, and the effect (effect C) by the additional information can be used singly , A face of a subject, etc., and can use this as additional information by realizing data in real time.

2, the output information selection unit 140 receives the divided information determined to be conformable by the information suitability determining unit 120 and the related information about the currently displayed additional information, The type of additional information that can be displayed on the broadcast and the type of additional information whose real-time representation is suitable for the current video among the determined types of additional information is selected from the database stored in the database unit 130, Information about the output condition thereof is supplied to the additional information output unit 150 so that the additional information output unit 150 synthesizes the selected additional information so as to be exposed for a suitable position and time in the video material of the real- For example. Here, the related information on the currently displayed additional information may include a list of currently displayed additional information, a remaining exposure time, a weight in the screen, and a processing buffer. Further, the type of additional information having relevance may mean image, audio, screen weight, file size, processing speed, and the like. The information on the output condition may mean output time, screen weight, size, and the like.

The additional information output unit 150 outputs the additional information selected by the output information selection unit 140 applied to the subject of the video object to the image provided by the keynote 170 while satisfying the predetermined output condition , The additional information provided from the output information selection unit 140 is applied in real time to change the direction or effect of the movement of the subject or the effect of the motion of the other subject by applying the physical value, So that the additional effect can be used.

The additional information display image providing apparatus 100 for automatically recognizing a subject according to an embodiment of the present invention may further include a statistical calculation unit for calculating a preference degree based on usage frequency of each additional information, 160). The statistic calculation unit 160 records a customer's use history of additional information, measures a preference based on the frequency of use of each additional information, calculates a usage frequency for an event such as an advertisement, And history of additional information such as subtitles and effects reflected in the database in real time. Machine learning can be used to suggest customized services and functions that reflect preferences. The number of exposures of specific trademarks, We can collect the data by user and effect by calculating weights, and use the quantified data as the basis of indirect advertisement.

The apparatus 100 for providing an additional information display image through the automatic recognition of a subject according to an embodiment of the present invention may further include an event module 220, a notification unit 230, and a communication unit 240.

The event module 220 may be configured to display additional event information stored in the database unit 130 on the screen of the display unit 190 by the output information selection unit 140 when the event mode is selected by the input unit 250, And the participant is displayed on the screen of the display unit 190 by the key knob 170. The frequency of contact with the event additional information according to the determined method The coupon can be issued when the score obtained according to the predetermined number exceeds the predetermined score. Here, the additional information for the event may be additional information previously produced by requesting and approval of an advertiser who has an advertisement contract.

On the other hand, the output information selection unit 140 can make the event additional information correspond to the predetermined commodity of the advertiser, which can greatly contribute to increase the advertisement effect. The event module 220 controls the output information selection unit 140, the additional information output unit 150 and the output control unit 180 so that the drink corresponding to the commodity requested by the advertiser is poured If the participant is allowed to drink a drink on the screen in a predetermined manner, for example, in a state where the mouth is opened, it is possible to determine the percentage of the state in which the match state It is possible to judge by image processing and predetermined calculation and convert it into a score. The event module 220 may control the advertiser's commissioned product to float on the screen of the display unit 190 by controlling the output information selection unit 140, the additional information output unit 150 and the output control unit 180, for example. , And how much the participant handles such a product by the image processing and the predetermined calculation, and the participant can participate in the exposure of the commodity of the commodity by various methods Can

The notifying unit 230 causes the participant to send the coupon issued by the event module 220 to the cellular phone through which the participant inputs the telephone number through the input unit 250. [ Also, the notifying unit 230 receives the image of the participant displaying the event-use additional information from the event module 220 and transmits the image to the mobile phone through the communication unit 240, which will be described later, so that the participant transmits the image to the SNS, Uploading can share an interest, which can lead to an increase in advertising effectiveness by exposure of the product that the advertiser has previously commissioned. The coupon may be a gift certificate enabling the purchase of an item or a commodity, a point used at a specific site, or the like.

The communication unit 240 wirelessly communicates with the mobile phone so that the coupon is transmitted to the mobile phone corresponding to the phone number input through the input unit 250 by the notifying unit 230. To this end, the communication unit 240 transmits the 3G, LTE, Wi- And various communication methods can be applied.

An example of combining the additional information with the image of the real-time participant according to the additional information display image providing apparatus through the automatic recognition function of the subject according to the present invention will now be described.

As shown in FIG. 4, the audio extracting unit 112 of the information classifying unit 110 can recognize the audio of the subject and display it as a caption. For example, the audio extracting unit 112 It is recognized by extraction and displayed as a caption on the screen. At this time, the additional information is determined by the information suitability judging unit 120, and the output information selecting unit 140 selects the additional information, which is selected from the plurality of divided information, And display time.

5, when the audio extracting unit 112 of the information classifying unit 110 recognizes the audio of the subject, the audio classifying unit 110 displays the subtitle corresponding to the audio of the subject in the information classifying unit 110, the information suitability determining unit 120 ), The output information selection unit 140 and the additional information output unit 150, it is possible to recognize the foreign language and display the subtitle corresponding to the recognized language. On the other hand, when recognizing Chinese greetings as a voice, a type or a method for a display language can be selected so that subtitles in the language corresponding to the voice are displayed by the input by the input unit 250, or displayed as subtitles translated into Korean have.

As shown in FIG. 6, when the trademark extracting unit 114 of the information classifying unit 110 recognizes a specific trademark expressed on a product on the screen or the like, the information suitability determining unit 120 and / The output information selecting unit 140 can display the conforming trademark for a predetermined time in a predetermined position and size. At this time, the information suitability determining unit 120 and the output information selecting unit 140 may follow a predetermined specification such that the background, subtitle, and sound output on the screen match the advertisement.

7, when the information classification unit 110 extracts the character / image extraction unit 115 and recognizes the designated color on the screen, the information suitability determination unit 120 and the additional information output unit 150 The image or image can be converted to a color designated by a predetermined chroma key technique.

As shown in FIG. 8, the face extracting unit 111 of the information classifying unit 110 recognizes the face-to-face ratio of the subject, extracts the face, synthesizes the face matching the face as additional information, It can be synthesized as additional information, and when the change in expression such as the opening of the mouth is sensed, it is possible to synthesize additional information in the meaning of yawning associated with it or to take a predetermined drink or food. Can be dynamically displayed.

According to the present invention, additional information such as subtitles, effects, sounds, and the like can be output to the video of participating customers acquired in the field without additional equipment or professional personnel, .

In addition, according to the present invention, not only can anyone easily view his / her own video provided with additional information, but also participate in an event.

Further, according to the present invention, it is possible to maximize the advertisement effect by enhancing the semantic transfer power of the video object.

Although the present invention has been described with reference to the accompanying drawings, it is to be understood that various changes and modifications may be made without departing from the spirit and scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the scope of the appended claims and equivalents thereof.

110: information classifying unit 111: face extracting unit
112: Audio extraction unit 113: Operation extraction unit
114: trademark extracting unit 115: character / image extracting unit
120: information suitability judging unit 121:
122: plausibility consideration display section 123: information amount consideration display section
124: recognizable consideration display section 125: processing time consideration display section
130: Database part 131: Output database part
132: store database unit 133: customer database unit
140: output information selection unit 150: additional information output unit
160: Statistics calculation unit 170: Kinect
171: camera 172: microphone
173: 3D depth sensor 180: Output control section
190: display unit 210: microcomputer
220: Event module 230: Notification section
240: communication unit 250: input unit 184
260: main body 261: front
262: Support frame 270: PC

Claims (5)

A main body having a display part on the front side;
A key knob installed to acquire a front image and audio in the main body;
An information classifying unit for obtaining information from the video object acquired by the keynote;
The information classification unit may receive the divided information and determine information suitability according to the type and environment of the classified information for identification and additional information output as information that can be used in a real time reproduction environment of the video through the display unit An information suitability judging unit;
A database for storing a database of a plurality of additional information connected to the divided information;
The information suitability determining unit receives the divided information determined to be conformable and the related information about the currently displayed additional information to determine the cognitive range of the viewer and the type of additional information that can be displayed in the real time video, An output information selection unit for selecting, in a database stored in the database unit, type of additional information for which a real-time representation is suitable for a current video among types of additional information;
An additional information output unit for outputting the additional information selected by the output information selection unit to synthesize the supplementary information selected by the output information selection unit to a video image provided by the keynote while satisfying a predetermined output condition;
The event information additional information stored in the database unit is displayed on the screen of the display unit by the output information selection unit and the participant is displayed on the screen of the display unit by the keyboard, An event module for making a contact with the event additional information and issuing a coupon when the score obtained based on the frequency of contact with the event additional information according to the determined method exceeds a predetermined score;
A notification unit for causing the participant to send a coupon issued by the event module to the mobile phone through which the participant inputs the telephone number through the input unit; And
And a communication unit that performs wireless communication with the cellular phone so that the coupon is transmitted to the cellular phone by the notifying unit. The method according to claim 1,
The information classification unit,
Extracting the face from the video image photographed from the Kinect by the face extracting unit, extracting the voice by the audio extracting unit, extracting the motion by the motion extracting unit, extracting the trademark by the trademark extracting unit, And an information extraction unit for extracting a character or an image by the image extraction unit to provide the information suitability determination unit with the facial expression, voice, movement, trademark, and character or image as the divided information, Information display image providing device The method according to claim 1,
The information suitability determining unit may determine,
In order to identify and output additional information as information that can be used in a real-time broadcast environment, it is considered that the additional information is displayed on the screen for a predetermined time or the audio is outputted by the perceived time considering display unit, Or considering that the additional information is displayed within a predetermined time after exposing the trademark and that the information amount consideration display unit considers the position and the size of the additional information to be overlapped or not to interfere with the subject of the video image, Considering that the effect sounds of the additional information are not overlapped with each other and considering that the object image and the sound are separated and displayed in the unit of analysis by the recognizable consideration display unit, To consider the efficiency of , The type of the additional information, the processing speed, the size, the holding time, and the category weight are respectively assigned to determine the information suitability according to the type of the classified information and the environment and the additional information to be displayed on the screen based on the weight Determines the priority of the currently outputting information and determines the remaining time and priority information of the currently outputting information in real time to construct a time line, records the additional information usage history of the user, The additional information display image providing device is provided with the additional information display image providing device for automatically recognizing the subject. delete The method according to claim 1,
Wherein the output information selection unit comprises:
The additional information for the event corresponds to a request item of the predetermined advertiser,
Wherein,
An additional information display image through an automatic recognition function of the object, which allows the participant to use the image by receiving the image of the participant in which the event additional information is displayed from the event module, .

Images