KR20180048537A - Method, apparatus and computer program for providing search information from video - Google Patents

Abstract

The present invention relates to a method, an apparatus and a computer program for providing video search information. The method for providing video search information by an electronic apparatus comprises: a step of generating a query by using a video if a preset query generation event is generated while the video is played; a step of transmitting the query to a server; and a step of receiving a search result corresponding to the query from the server to display the search result on a screen. The search result is a search result for one or more objects included in the video. According to the present invention, a product similar to a product exposed in a video can be searched without inputting a separate search word.

Description

METHOD AND APPARATUS AND COMPUTER PROGRAM FOR PROVIDING SEARCH INFORMATION FROM VIDEO}

The present invention relates to a method, an apparatus, and a computer program for providing image retrieval information in an image reproducing apparatus, and more particularly, to a method, an apparatus, and a computer program for recognizing an object in an image without inputting a separate query word, .

In recent years, commerce such as promotion and marketing using online has been activated in the background of the increased wired and wireless Internet environment. In this regard, buyers searching for magazines, blogs, videos on YouTube, etc. from a desktop or mobile terminal connected to the Internet finds a product they like and searches for a product name and the like to purchase. Famous actresses list the name of the bags they listened to at the airport and the names of the childcare items in the entertainment program that rises above the ranking of real-time search terms on the portal site. However, at this time, the user has to separately open a web page for searching and search for a product name and a selling place, and if the name is not known, the user can not easily search for the inconvenience.

On the other hand, sellers often sponsor products for video content such as dramas and movies in order to promote their products. Most of them do not directly disclose product names due to indirect advertisement issues despite spending on promotional expenses.

Therefore, recently, sellers are searching for a variety of promotional methods such as producing video content tagged with promotional product information in a video frame so that the user can confirm the product information displayed on the video when playing the video. However, according to this method, since the user can confirm only the limited information (the product information tagged by the seller in the content) provided by the seller, if the favorite product is not the sponsored product in the image, . In addition, if the product exposed to the image is not at a price range that can be purchased, the consumer may search for a similar product at a lower price. If the product is identified by the image, it is difficult to specify the product name. have.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an image search information providing method, an apparatus, and a program for searching for a product similar to a product exposed to an image without inputting another search word.

Further, according to the present invention, it is possible to expose product information at the time of original image reproduction without producing separate content tagged with product information, thereby reducing the cost required for product promotion and providing image search information for more effectively promoting the product It is another object to provide a method and apparatus.

According to another aspect of the present invention, there is provided a method of providing image search information by an electronic device, the method comprising: generating a query using the image when a predetermined query generation event occurs during image playback; Receiving a search result corresponding to the query from the server and displaying the search result on a screen, wherein the search result is a search result of one or more objects included in the image.

According to another aspect of the present invention, there is provided an electronic device for providing image search information, comprising: a controller for generating a query using the image when a predetermined query generation event occurs during image playback; A communication unit for receiving a search result corresponding to the query; and a display unit for displaying the received search result on a screen, wherein the search result is a search result for an arbitrary object included in the image.

According to another aspect of the present invention, there is provided a method of providing image retrieval information to a client, the method comprising: generating a product database including an image feature information of the product and a label corresponding to the feature information of the product image, Receiving a query generated using an image from the client, analyzing the query, and if the query includes an object to which the label is attached as a result of the query analysis, Or product information corresponding to the image feature information of the search object in the goods database, and transmitting the search result to the client.

According to another aspect of the present invention, there is provided a server for providing image search information to a client, the server comprising: a database for creating and storing a product database including an image feature information of the product and a label corresponding to the feature information of the product image, A communication unit for receiving a query generated using an image from the client and transmitting a search result for the query to the client, analyzing the received query, and determining a label of the object included in the query, And a control unit for searching the database unit for product information corresponding to the image feature information and providing the retrieved product information.

According to the present invention as described above, a product similar to a product exposed to a video image can be searched without inputting another search word.

In addition, according to the present invention, it is possible to expose product information at the time of image reproduction without producing additional content tagged with product information, thereby reducing the cost of promoting the product and promoting the product more effectively.

1 is a conceptual diagram for explaining an embodiment of an electronic apparatus for providing image search information according to the present invention;
2 is a conceptual diagram for explaining an embodiment of a server for providing image search information to a client according to the present invention;
3 is a flowchart for explaining an embodiment of a method for providing image search information of an electronic device according to the present invention,
FIG. 4 is a flowchart for explaining an embodiment of a query generation method according to the present invention;
FIG. 5 is a flowchart for explaining an embodiment of an object extraction method according to the present invention,
6 is a flowchart for explaining an embodiment of an object extracting method according to the present invention,
FIG. 7 is a flowchart for explaining an embodiment of a method for providing image search information of a server according to the present invention;
8 is a view for explaining an embodiment of providing image search information of an electronic device according to the present invention.

The above and other objects, features, and advantages of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings, which are not intended to limit the scope of the present invention. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to designate the same or similar components, and all combinations described in the specification and claims can be combined in any manner. It is to be understood that, unless the context requires otherwise, references to singular forms may include more than one, and references to singular forms may also include plural forms.

In the present specification, 'object' may be understood as meaning an image of a product or an object that can be recognized as a product, or may be an image of an unspecified object including noise according to an object extraction condition, It may be an image of a corresponding specific object. An 'object' may correspond to a comprehensive category of products according to the extraction condition and the classification depth of the learned label, but it should be understood as a concept that can correspond to detailed product features such as color, pattern, and product name .

As used herein, the term 'label' is information defining the characteristics of a product (or object) as described above. For example, the label may include information on major classification such as a woman's clothing, a men's clothing, furniture, Information such as middle class information such as a dress, color, pattern, brand, and price. This is merely an example, and the label can be classified or defined in various ways according to the user setting. A label according to an embodiment of the present invention may be learned through machine learning, and a label corresponding to image feature information may be stored in advance in a server or a client (electronic device).

In this specification, the term 'electronic device' includes all electronic devices capable of reproducing images such as a desktop, a smart phone, and a tablet PC and capable of displaying search information, and has the same meaning as a client and an image search information providing device ≪ / RTI >

1 is a conceptual diagram for explaining an embodiment of an electronic device for providing image search information according to the present invention. Referring to FIG. 1, an electronic device 100 according to the present invention may include a control unit 130, a communication unit 150, and a display unit 170, and may further display an input unit 190 (not shown).

The control unit 130 generates a query using a video when a predetermined query generation event occurs during video playback.

The control unit 130 may determine that a query generation event has occurred when a preset time elapses or a search event occurs after a pause event occurs during video playback. For example, if the user temporarily pauses an image to be reproduced and remains in a paused state after a predetermined time (2 seconds) has elapsed, the electronic device 100 determines that a query generation event has occurred, You can create a query with

The search event is an event in which the electronic device 100 can recognize that the electronic device 100 is searching for a product. For example, the search event may be a click or a touch of a search button displayed after the user pauses, , An operation of clicking or touching one area or button recognized as an object in an image frame.

If there is no user input until the remaining time reaches 0, the control unit 130 displays a query generation event in which the query generation event is generated. .

The query generated by the control unit 130 at the time of the query generation event includes a paused image frame, an image of one region included in the image frame, a plurality of image frames satisfying predetermined conditions based on the paused screen, And object information extracted from the one-image frame included in the image. The object information may include at least one of image feature information extracted from an image, image feature information extracted from an arbitrary object region, image feature information learned or information corresponding to a specific object, or label information.

Here, the image feature information refers to information extracted locally (locally) or globally (globally) from an image of one region included in an image, an image frame, or an image. Image feature information can be used for machine learning for label generation. Further, when there is a pre-learned label, any image feature information may correspond to a specific label or a specific object. The image feature information may include vector information (descriptors) of a predetermined range of images (image pieces or patches) located around the entire image or object area or in the image, and the vector information may include a brightness change, a color, a pattern Frequency, vertical edge, and horizontal edge), the slope of the image slice, the brightness change over a predetermined threshold value, and the like. In addition, the image feature information may have a value in a range unique to each object, and may have a value in a range unique to each label according to a learning model.

The image of one region included in the image frame may be a polygon (B) as shown in FIG. When a pause event occurs, the controller 130 may provide a guide including a region A, which is recognized as an object, as a guide for the user. At this time, the user adjusts the size or shape of the polygon provided as a guide , And finally, the area included in the query can be selected.

In addition, when the paused image frame is blurred or the object is difficult to detect, or when it is necessary to increase the accuracy of object extraction from the server, the control unit 130 sets a predetermined condition based on the paused image frame One or more image frames that are satisfied may be included in the query.

The condition to be set in advance for selecting one or more image frames included in the query may be set using a time, a number of frames, a subtitle availability, and / or a sharpness. For example, the control unit 130 may generate image frames within 1 second before and after the image frame temporarily paused according to predetermined conditions as a single query. In addition, the control unit 130 can generate three image frames as one query by adding the previous / next image frames one by one based on the temporarily stopped image frame. In another embodiment, when the sharpness of the paused image frame is lowered, the controller 130 may search for and include an image frame having high sharpness in the pre / post image frame. As another example, when the caption is included in the temporarily stopped image, the control unit 130 may include in the query only the image frame from which the caption has been removed.

The control unit 130 may include an object extracting unit 135 for extracting an object from an image frame, and a query generating unit 137 for generating a query including the object.

The object extracting unit 135 extracts an area including the learned image feature information corresponding to the specific object as an object using the previously stored learning model and attaches a label corresponding to the image feature information to the object . The object in the present embodiment means an object that can be defined as a specific object, and can be variously defined according to the label or category.

The learned image feature information corresponding to a specific object using the pre-stored learning model may be a global descriptor or a local descriptor according to the setting. That is, the object extracting unit 135 may extract an object including the learned image feature information corresponding to the car in the image frame, extract an object including the learned image feature information corresponding to the clothing It also categorizes not only what objects are objects but also the characteristics of objects (product types, product brands, colors, materials, shapes, patterns, differences in some designs and feelings that can be defined by humans - cutes, You may. Therefore, it is possible to label specific product names and product characteristics by using global and local descriptors according to the purpose.

In another embodiment, the object extracting unit 135 extracts image feature information from an image frame, and uses the extracted feature information to divide one area within an image frame according to predetermined conditions, Can be set. Here, the object extracted by the electronic device according to the present invention may be determined as an arbitrary 'object' regardless of the type of the object, so that noise can be included, but extraction is possible at a high speed. The segment of the region may be made using the image feature information in the image frame such as the brightness change point, the overall color distribution, the edge information, the sharpness information, etc., and the setting of the object may be performed using the coordinate information on the image frame. A plurality of objects may be included in an image frame, and in this case, a plurality of objects may be extracted.

When the display unit 170 displays the search result, the control unit 130 preferentially displays a search result for an object including the same or similar label as the user searched by using the search log of the user Can be controlled. This is to expose the search results belonging to the user's interest to the screen in advance, thereby increasing the probability that the image search will lead to the purchase of the product and improving the user satisfaction. In addition, the control unit 130 can increase the accuracy of the search by controlling the display unit 170 to display the search results on the screen in the descending order of similarity with the image feature information of the objects included in the query.

The communication unit 150 transmits the query to the server 200 and receives the search result corresponding to the query from the server 200. [ The communication unit 150 may include a mobile communication module supporting wired / wireless communication modules for communication with the server 200, for example, GSM, CDMA, WCDMA, HSDPA and LTE. The communication unit 190 may include a wireless communication module such as WLAN, WiFi Direct, DLNA, Wibro, and Wimax. The communication unit 190 may include a communication module for short range communication such as Bluetooth, RFID, , ZigBee, NFC, Wi-Fi, Wi-Fi Direct, and the like. For reference, the communication unit 150 of the present invention is not limited to a communication module supporting data transmission / reception with the server 200.

The display unit 170 displays the received search result on the screen, where the search result means a search result for an arbitrary object included in the image. The display unit displays (outputs) information processed in the electronic device 100 and displays execution screen information of an application program driven by the electronic device 100 or UI (User Interface), GUI Interface information can be displayed.

The display unit 170 may include a display module, and the display module may include a touch screen having a mutual layer structure with the touch sensor or formed integrally with the touch sensor. Such a touch screen may function as an input 190 (not shown) that provides an input interface between the electronic device 100 and a user and may provide an interface for displaying information between the electronic device 100 and a user . The display module may be a stereoscopic display module for displaying a stereoscopic image. As such a stereoscopic display, a three-dimensional display system such as a stereoscopic system (eyeglass system), an autostereoscopic system (non-eyeglass system), and a projection system (holographic system) can be applied. The display unit 170 of the present invention is not limited by the type of the display module and may include a display module capable of displaying a moving image by all kinds of image reproducing programs capable of reproducing images, All included.

2 is a conceptual diagram for explaining an embodiment of a server for providing image search information to a client according to the present invention.

Referring to FIG. 2, the server 200 according to an embodiment of the present invention may include a controller 230, a communication unit 250, and a database unit 270.

The control unit 230 analyzes the query received from the client 100 and searches the database unit 270 for the label of the object included in the query or the product information corresponding to the image feature information of the object, .

If the query includes image feature information classified as noise or label information learned as noise, the control unit 230 may exclude the query or object from the search target.

In addition, if the query analysis result includes an image frame, an image of one region included in the image frame, or a first object to which no label is attached, the control unit 230 determines image feature information from the image frame, Can be extracted. Here, the first object may be an unlabeled object, an object recognized by the electronic device as an object regardless of the type of the product, and the image may be an image of a region captured or cropped in the electronic device.

The control unit 230 may include the image feature information in an image frame, an image, or the first object, if there exists image feature information that has been learned as a specific object in a search result of image feature information, an image frame, And a label corresponding to the image feature information can be attached to the search object.

The control unit 230 searches the database unit 270 for a product having the same / similar label or the same / similar image feature information using the label of the search object or the image feature information.

If the query includes an object to which a label is attached as a result of the query analysis of the control unit 230, the control unit 230 stores product information corresponding to the label attached to the object or the image feature information of the object, (270). And controls the communication unit 250 to transmit the search result to the client.

When the communication unit 250 receives the user's identification information and the user's search log from the client, the controller 230 uses the identification information and the search log received from the client to obtain product information corresponding to the label included in the search log Can be provided to the user through the communication unit 250. More specifically, the control unit 230 may assign a higher importance to the product information corresponding to the label included in the search log so that the product information is more preferentially exposed at the client.

The communication unit 250 receives the query generated by using the image from the client and transmits the search result for the query to the client 100. Also, the communication unit 250 can receive the user's identification information and the user's search log from the client together. An example of the communication module included in the communication unit 250 is the same as that of the communication module included in the communication unit 150 described above.

The database unit 170 generates and stores a product database including the image feature information of the product and the label corresponding to the feature information of the product image with respect to an arbitrary product.

Hereinafter, an embodiment of a method for providing image search information according to the present invention will be described with reference to FIG. 3 to FIG.

Referring to FIG. 3, according to the method of providing image search information according to the present invention, when a predetermined query generation event occurs during image playback (S100), the electronic device generates a query using the image (S200) To the server (S300). The query generated in step S200 may include a paused image frame, a region image included in the image frame, a plurality of image frames that satisfy predetermined conditions based on the paused image, or a one- And object information. The object information may include at least one of image feature information extracted from an image, image feature information extracted from an arbitrary object region, image feature information learned or information corresponding to a specific object, or label information.

Next, the electronic device receives the search result corresponding to the query from the server (S400), and displays the received search result on the screen (S500), thereby providing the image search information. In step S500 of displaying the received search result on the screen, the electronic device preferentially displays a search result on an object including a label identical or similar to an object previously searched by the user using the search log of the user . This is because the electronic device exposes the search results belonging to the user's interest area to the screen in advance, thereby increasing the probability that the image search leads to the purchase of the product and raising the user satisfaction. In step S500, the electronic device displays the search results on the screen in the order of high similarity to the image feature information of the objects included in the query, thereby improving the accuracy of the search.

The above-described query generation step S200 may be performed by extracting an object in an image frame as shown in FIG. 4 (S230) and creating a query including an object (S250).

Further, in the query generation step S200, the electronic device of the present invention can determine that a query generation event has occurred when a preset time elapses or a search event occurs after a pause event occurs during video playback.

Further, in the query generation step S200, the electronic device of the present invention displays the remaining time until the generation of the query on the screen when a pause event is generated during video playback, and if the user inputs until the remaining time reaches 0 If not, it can be determined that a query generation event has occurred.

Referring to FIG. 5, in the object extracting step S230, the electronic device extracts image feature information from an image frame (S232), divides one region in the image frame according to predetermined conditions using the extracted image feature information S234), and set one segmented area as a search object (S236).

The segment of the area in step S234 may be performed using the feature information in the image frame such as the brightness change point, the overall color distribution, the edge information, and the sharpness information, and the setting of the search object in step S236 may be performed based on the coordinate information . ≪ / RTI > A plurality of objects may be included in an image frame, and in this case, a plurality of objects may be extracted.

In another embodiment of the object extraction (S230), the object extraction (S230) extracts one region containing the learned image feature information corresponding to a specific object using a pre-stored learning model, as shown in FIG. 6, (S233), and attaching a label corresponding to the image feature information to the search object (S237).

7 is a flowchart for explaining an embodiment of a method of providing image search information of a server according to the present invention. Referring to FIG. 7, the server of the present invention creates a product database including image feature information of a product and a label corresponding to feature information of the product image with respect to an arbitrary product.

The server analyzes the query (S2000), receives the query generated using the image from the client (S1000), and analyzes the query (S2000). It determines whether the learned image feature information is included in the query as noise (S2500) Can be excluded from the search target.

As a result of the query analysis (S2000), if the query includes an image frame (Case 1), an image of one region included in the image frame (not shown), or a first object (Case 2) Image feature information learned as an image or a specific object from the first object is searched (S3000). In operation S3500, a region including the image feature information is extracted as a search object in the search result of step S3000, an image frame, an image, or a first object. Next, the server attaches a label corresponding to the image feature information to the search object (S3700). Then, a product having the same / similar label or the same / similar image feature information is searched in the product database using the label of the object to be searched or the image feature information (S4000).

If the query includes an object to be searched with a label (Case 3) as a result of the query analysis (S2000), the server obtains product information corresponding to the label attached to the search object or image feature information of the search object, (S4000). Then, the search result is transmitted to the client (S5000).

Although not shown in the figure, the server can receive the user's identification information and the user's search log from the client. In this case, in the search result transmission step (S5000), the server can provide the user with the goods information corresponding to the label included in the search log, using the identification information and the search log received from the client. The server can transmit the product information to the client with higher importance so that the product information is exposed more preferentially at the client.

8 is a view for explaining an embodiment of a method of providing image search information of an electronic device according to the present invention. In FIG. 8, the user reproduces an image using the electronic device 100. In the video, a woman wears a T-shirt (A), which is an example of a case where the user pauses the image for searching for product information on the T-shirt (A). When a pause event occurs and there is no user input for 2 seconds, a query generation event is generated by a preset algorithm.

The electronic device 100 generates a query using the paused image frame in response to the occurrence of the query generation event. The electronic device 100 may transmit the image frame itself to the server or may transmit an image of one area included in the image frame to the server. In addition, it is also possible to extract an object from a one-frame image included in the image, and to transmit the extracted object information to the server.

Assume that when a pause event occurs, a rectangle guide (B) is set to appear so that the user can specify an area. The user can select the t-shirt A by adjusting the size and shape of the guide B and move the position of the guide B so that the electronic device 100 does not recognize the object, ) Can be designated as an object.

In an embodiment in which the electronic device 100 recognizes the user's guide area B click or touch operation as a query generation event, the server 200 (not shown) performs the search using the guide area image B. The server 200 receives a query including an image with no label attached thereto from the electronic device 100 and searches whether the image includes feature information learned as a specific object. If the image has learned image feature information as noise, the server 200 will stop searching for the image, but since the image includes the T-shirt A, it also includes the image feature information learned by the T-shirt.

When the server 200 finds the image feature information (descriptor corresponding to the t-shirt) learned by the t-shirt, it sets an area including the image feature information as a search object. The object to be searched in this way has a form closest to the T-shirt A and may be a region excluding an area (background) irrelevant to the goods in the image B received at the beginning.

Next, the server 200 can attach a label such as a ladies' garment (major classification), an image (middle classification), blue, and a t-shirt (small classification) to the search target object. In addition, a product corresponding to the blue t-shirt can be searched in the product database, and a product including similar image feature information can be searched (image search) in the product database using the extracted image feature information. If the same and / or similar goods 20A, 10A, 10B, 10C and 10D are searched as a result of the search, the server 200 transmits the searched identical and / or similar goods information to the electronic device 100. [

The electronic device 100 receives the search result from the server 200 and displays it on the screen. When displaying the search result on the screen, the electronic device 100 may display the product information in one area 10 of the screen, It is also possible to display the product information in the area (B) or the object (A) 20 that has been designated. The merchandise information may be displayed in descending order or ascending order depending on the degree of similarity in the one area 10 or the degree of interest, price, popularity, etc. according to the user's search log. For example, 10A with the highest similarity in the most similar order can be displayed at the top, and 10B, 10C with the next highest similarity can be displayed in that order.

The electronic device also displays the product name 20B, the manufacturer of the product, the price 20C, the popularity, the category (label), the purchase place information (website, You can also set it to launch other applications by clicking or touching the information.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, But the present invention is not limited thereto.

100: electronic device, client
200: Server

Claims (10)

A method for an electronic device to provide image search information,
Generating a query using the image when a preset query generation event occurs during a video playback;
Transmitting the query to a server;
Receiving a search result corresponding to the query from the server and displaying the search result on a screen,
The query generation step
Displaying a remaining time until the generation of the query on the screen when a pause event occurs during the video playback;
Determining that the query generation event has occurred if there is no user input until the remaining time reaches 0,
Wherein the search result is a search result of one or more objects included in the image.
The method according to claim 1,
The query may include an image frame displayed on a paused screen, an image of one area included in the image frame, a plurality of image frames satisfying predetermined conditions based on the paused screen, And object information.
3. The method of claim 2,
Wherein the object information includes at least one of image feature information extracted from the image or label information learned corresponding to a specific object.
The method according to claim 1,
The query generation step
Determining that a query generation event has occurred when a pause event occurs during the video playback and a preset time has elapsed or a search event occurs
A method for providing image search information.
The method according to claim 1,
The query generation step
Extracting the one or more objects from a video frame displayed on a paused screen;
And generating a query including the one or more objects.
6. The method of claim 5,
The object extracting step
And extracting, as a search object, a region including the learned image feature information corresponding to the specific object using the pre-stored learning model,
And attaching a label corresponding to the image feature information to the object to be searched.
6. The method of claim 5,
The object extracting step
Extracting image feature information from the image frame;
Dividing a region in the image frame according to a predetermined reference using the extracted image feature information;
And setting the segmented region as a search object.
The method according to claim 1,
The step of displaying the search result on a screen may include:
Further comprising the step of preferentially displaying a search result on an object including a label identical or similar to an object previously searched by the user using the search log of the user.
An electronic device for providing image search information,
A controller for generating a query using the image when a predetermined query generation event occurs during video playback;
A communication unit transmitting the query to a server and receiving a search result corresponding to the query from the server;
And a display unit for displaying the received search result on a screen,
The control unit displays a remaining time until the generation of the query on the screen when the pause event occurs during the video playback and determines that the query generation event occurs when there is no user input until the remaining time reaches 0 In addition,
Wherein the search result is a search result for an arbitrary object included in the image.
A program for providing image search information stored in a computer-readable medium for executing a method according to any one of claims 1 to 8.

Images