KR20200088019A - Method and apparatus for searching broadcasting image - Google Patents

Abstract

The following embodiments relate to a method for searching an image and to an apparatus thereof. According to one embodiment, the method for searching an image comprises the steps of: receiving a captured broadcasting image; inputting the captured broadcast image to a plurality of image analysis servers, and receiving an image analysis result corresponding to the captured broadcasting image from each of the plurality of image analysis servers; scoring the image analysis result to determine the priority of the image analysis result; determining any one of the image analysis results as a search target based on the priority; inputting a search target into the service server to obtain service content corresponding to the search target; and processing the service content to be displayed on the screen.

Description

Broadcasting video search method and device {METHOD AND APPARATUS FOR SEARCHING BROADCASTING IMAGE}

The embodiments below relate to an image search method and apparatus.

With the continuous development of information and communication technology, an IPTV broadcasting system that broadcasts real-time broadcast content or on-demand broadcast content using an IP (Internet Protocol) communication network as a broadcast transmission medium has been commercialized. Furthermore, recently, a smart TV has appeared that supports interworking with a smartphone by adding interactive functions with users and various additional functions using the Internet to the existing traditional TV.

Looking at the trend of recently commercialized analog broadcasts, satellite digital broadcasts, terrestrial digital broadcasts, and cable broadcasts, it is often observed that the search terms related to the broadcast are ranked at the top of the search terms of online portals at the same time as the broadcast. Is done.

This means that broadcast and online search have already been fused by broadcast viewers, and viewers who watch the same broadcast content have the feature of solving the curiosity about the broadcast content through online search regardless of the broadcast time. Is doing.

The embodiments are intended to provide a customized service by automatically recognizing a TV screen viewed by a customer.

The embodiments are intended to provide a service such as VOD, shopping, knowledge search, etc. by simply analyzing images without additional meta.

The embodiments attempt to automatically provide a similar service by analyzing the scene and situation being watched.

The embodiments are intended to assist in content search through person search and to help customers purchase desired products through shopping search.

The embodiments are intended to generate revenue through recommendation of VOD and induction of viewing through provision of services such as searching for personal information and providing portraits.

An image search method according to an embodiment may include receiving a captured broadcast image; Inputting the captured broadcast images into a plurality of image analysis servers, and receiving image analysis results corresponding to the captured broadcast images from each of the plurality of image analysis servers; Scoring the image analysis results to determine a priority of the image analysis results; Determining any one of the image analysis results as a search target based on the priority; Inputting the search target into a service server to obtain service content corresponding to the search target; And processing to display the service content on the screen.

The image analysis server may include at least one of a person recognition server, a product recognition server, a knowledge recognition server, and a complex recognition server.

The image analysis result may include an area targeted for recognition included in the captured broadcast image, a recognition result for the area targeted for recognition, and an accuracy score of the recognition result.

The determining of the priority may include determining the priority based on the accuracy score of the recognition result included in the image analysis result.

The processing of displaying the service content on the screen may include processing the service content to be displayed on the screen together with an area to be recognized.

The obtaining of the service content may include determining the service server predetermined according to the type of the search target; And obtaining the service content by inputting the search target into the determined service server.

The receiving of the captured broadcast image may include receiving a search request for the broadcast image; Capturing a broadcast image at the time of the search request according to the search request; And receiving the captured broadcast image.

In the receiving of the video analysis result, when the captured broadcast video is input to the person recognition server, when the captured broadcast video includes a person image, a keyword corresponding to the person image and an accuracy score of the person image are obtained. And receiving.

In the receiving of the video analysis result, when the captured broadcast image is input to the product recognition server, and the product image is included in the captured broadcast image, the sales information corresponding to the product and the accuracy score of the product image are obtained. And receiving.

When the search target is an image analysis result received from the person recognition server, the step of obtaining service content corresponding to the search target may include inputting the search target into at least one of a person search server, a media search server, and a photo search server. And obtaining service content corresponding to the search target.

The image analysis server includes at least one of a person recognition server, a product recognition server, a knowledge recognition server, and a compound recognition server, and determining the priority includes the person recognition server, the product recognition server, the knowledge recognition server, and Comparing an accuracy score of the recognition result included in the image analysis result received from the composite recognition server; And determining the priority based on the accuracy score of the recognition result.

The processing of displaying the service content on the screen may include providing a list of the service content according to a predetermined criterion when the service content is plural.

An image search apparatus according to an embodiment may include a broadcast signal receiver configured to receive a captured broadcast image; And inputting the captured broadcast images to a plurality of image analysis servers, receiving image analysis results corresponding to the captured broadcast images from each of the plurality of image analysis servers, and scoring the image analysis results to score the image analysis results. Determines the priority of, and determines any one of the video analysis results based on the priority as a search target, inputs the search target to a service server to obtain service content corresponding to the search target, and the service And a processor that processes content to be displayed on the screen.

The image analysis server may include at least one of a person recognition server, a product recognition server, a knowledge recognition server, and a complex recognition server.

The image analysis result may include an area targeted for recognition included in the captured broadcast image, a recognition result for the area targeted for recognition, and an accuracy score of the recognition result.

The processor may determine the priority based on the accuracy score of the recognition result included in the image analysis result.

The processor may process the service content to be displayed on the screen together with an area to be recognized.

The processor may determine the predetermined service server according to the type of the search target, and obtain the service content by inputting the search target to the determined service server.

The image analysis server includes at least one of a person recognition server, a product recognition server, a knowledge recognition server, and a compound recognition server, and the processor is configured from the person recognition server, the product recognition server, the knowledge recognition server, and the compound recognition server. The accuracy score of the recognition result included in the received image analysis result may be compared, and the priority may be determined based on the accuracy score of the recognition result.

Embodiments may provide a customized service by automatically recognizing the TV screen viewed by the customer.

Embodiments can provide services such as VOD, shopping, knowledge search, etc. by simply analyzing images without additional meta.

Embodiments can automatically provide similar services by analyzing the scene and situation being viewed.

Embodiments may help to search for content through person search, and to help purchase a product desired by a customer through shopping search.

Embodiments may generate revenue through VOD recommendation and induction of viewing through provision of services such as searching for personal information and providing portraits.

1 is a diagram illustrating an entire system including an image search apparatus according to an embodiment.
2 is a diagram for describing a detailed configuration of an image search apparatus according to an embodiment.
3 is a diagram for describing a detailed configuration of a processor according to an embodiment.
4 is a flowchart illustrating a method of determining a priority of an image analysis result according to an embodiment.
5 is a diagram illustrating an embodiment in which a person recognition result according to an embodiment is a search target.
6 is a diagram illustrating an embodiment in which a product recognition result according to an embodiment is a search target.

The specific structural or functional descriptions disclosed in this specification are only for the purpose of describing the embodiments according to the technical concept, and the embodiments may be implemented in various other forms and are limited to the embodiments described herein. Does not work.

Terms such as first or second may be used to describe various components, but these terms should be understood only for the purpose of distinguishing one component from other components. For example, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

When an element is said to be "connected" or "connected" to another component, it is understood that other components may be directly connected to or connected to the other component, but there may be other components in between. It should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that no other component exists in the middle. Expressions describing the relationship between the components, for example, "between" and "immediately between" or "adjacent to" and "directly adjacent to" should be interpreted similarly.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, the terms "include" or "have" are intended to designate the presence of a feature, number, step, action, component, part, or combination thereof as described, one or more other features or numbers, It should be understood that the presence or addition possibilities of steps, actions, components, parts or combinations thereof are not excluded in advance.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person skilled in the art. Terms such as those defined in a commonly used dictionary should be interpreted as having meanings consistent with meanings in the context of related technologies, and should not be interpreted as ideal or excessively formal meanings unless explicitly defined herein. Does not.

Embodiments may be implemented in various types of products, such as personal computers, laptop computers, tablet computers, smart phones, televisions, smart home appliances, intelligent cars, kiosks, and wearable devices. For example, embodiments may be applied to authenticate a user in a smart phone, mobile device, smart home system, or the like. Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The same reference numerals in each drawing denote the same members.

1 is a diagram illustrating an entire system including an image search apparatus according to an embodiment.

1, the system according to an embodiment includes a TV 100, a set-top box (STB) 110, a communication network 120, a broadcast server 130, a service server 140, And an image analysis server 150.

According to an embodiment, when a viewer watching a broadcast content has a question about the broadcast content, the broadcast video being output from the TV is captured according to the viewer's input, and the captured video is automatically based on image recognition. It can be extracted to provide additional service content to viewers. The search target may include person information, product information, and knowledge information, and a score regarding the suitability of each search target may be calculated based on the analysis on the capture screen, and the most suitable search target based on the calculated score Can be determined.

The TV 100 may be connected to the communication network 120 through the set-top box 110. The TV 100 may receive broadcast information from the broadcast server 130 through the communication network 120 and display a broadcast screen. The broadcast server 130 may include a terrestrial broadcast server and a cable broadcast server.

The TV 100 may be applied to a TV having any form and function of the present invention. In the following description, a system in which a set-top box in which a predetermined application is installed in a general TV is connected is described as an example. However, the present invention can be applied to a smart TV with a predetermined application installed and a communication function with the Internet. According to the present invention, all or part of the functions implemented in the set-top box may be implemented to function in the smart TV. In addition, the present invention is not limited to a smart TV, and can be applied to any type of TV in which a predetermined application can be installed.

The set-top box 110 may be connected to the TV 100, receive signals from the outside, convert appropriately, and display the contents on the TV. The set-top box 110 is a home communication terminal required to use next-generation interactive multimedia communication services such as video on demand (VOD), home shopping, and network games, and is also referred to as a'subscriber signal conversion device'. The set-top box 110 may perform a function of receiving a broadcast signal from the broadcast server 130 and providing it to the TV 100. In addition, the set-top box 110 may perform communication with the service server 140 and the image search server 150 by connecting to the communication network 120 including an Internet connection function. In addition, the set-top box 110 can provide the functions of the present invention through the installation of a separate dedicated application.

In the specification of the present invention, the term'application (application)' or'app (App)' refers to an application program installed on a smartphone or a smart TV, by installing and running the application on the set-top box 110 of the present invention Each function according to the embodiment is performed. On the other hand, the present invention may use an application made exclusively to provide various functions according to an embodiment of the present invention, or may be implemented by including the function according to the present invention in other specific applications. In addition, the application to which the present invention is applied may be an application for driving a smart TV, and when the application is executed, log in by entering user information (eg, ID) and authentication information (eg, password), and then providing services. It can also be provided.

The communication network 120 may be configured regardless of its communication mode, such as wired and wireless, and a short-range communication network (PAN), a local area network (LAN), and a metropolitan area network (MAN). ), Wide Area Network (WAN), and various other communication networks. In addition, the communication network 120 may be a known World Wide Web (WWW), or may use a wireless transmission technology used for short-range communication such as Infrared Data Association (IrDA) or Bluetooth (Bluetooth). have. In addition, the communication network 120 may include a cable broadcast communication network, a terrestrial broadcast communication network, a satellite broadcast communication network, and the like for receiving broadcast signals from the TV 100.

The image analysis server 150 may perform a function of receiving the captured broadcast image and transmitting an image analysis result corresponding to the captured broadcast image. When an image being broadcast by a viewer is captured, the captured broadcast image is input to a plurality of image analysis servers 150, and each of the plurality of image analysis servers 150 outputs an image analysis result corresponding to the captured broadcast image. Can. The image analysis result may include an area to be recognized in the captured broadcast image, a recognition result for the area to be recognized, and an accuracy score of the recognition result. The recognition result may have a different form according to the type of the image analysis server 150. The accuracy score may be a value according to the similarity degree of the recognition result having the highest similarity among the recognition results analyzed by each of the plurality of image analysis servers 150. To calculate the accuracy score, a multidimensional vector comparison analysis operation can be performed. For example, accuracy scores can be calculated through various techniques such as vector spatial modeling, topic modeling, cosine similarity, and association rules. The method of expressing the accuracy score of the recognition result may vary. For example, the accuracy score of the recognition result may be expressed in %. The plurality of image analysis servers 150 may output the accuracy score of the recognition result with the same scoring system. Specific examples of the image analysis results will be described in detail with reference to FIGS. 5 to 6 below.

Each of the plurality of image analysis servers 150 may output an image analysis result corresponding to the captured broadcast image. Each of the image analysis servers 150 may output one or more image analysis results, and score the image analysis results of each of the plurality of image analysis servers 150 to determine the priority of the image analysis results. Any of the image analysis results may be determined as a search target based on the priority. The method of determining the priority and search target will be described in detail with reference to FIG. 3 below.

The service server 140 may receive a search target and output a service content corresponding to the search target. Hereinafter, each component of the application installed in the set-top box 110 will be described in detail.

2 is a diagram for describing a detailed configuration of an image search apparatus according to an embodiment.

2, the image search apparatus 200 according to an embodiment includes a broadcast signal receiving unit 210, a main control unit 220, a buffer unit 230, a broadcast data storage unit 240, a processor 250, It may be configured to include a data communication I/F unit 260, an external server 270, a TV image configuration unit 280, a broadcast data storage unit 290, and a display panel 291.

The broadcast signal receiving unit 210 receives various types of broadcast data transmitted from the outside, and may perform functions such as terrestrial broadcast signal reception, cable broadcast signal reception, IPTV broadcast signal reception, and satellite broadcast signal reception. The broadcast signal receiving unit 210 may include a cable TV receiving unit, an IPTV receiving unit, a satellite broadcasting receiving unit, and the like.

For example, in the broadcast signal receiving unit 210, the cable TV receiving unit demodulates the cable TV signal. At this time, the demodulated data is composed of compressed video, audio, and additional information. The compressed image may generate video raw data by performing decompression according to the MPEGx/H.264 standard. In addition, the compressed voice may perform decompression according to the MPEGx/AC3/AAC standard to generate audio raw data. In addition, additional information may be transmitted to the main control unit 220. The received broadcast signal may be transmitted to the display panel 291 through the TV image configuration unit 280 through the buffer unit 230.

The IPTV receiver in the broadcast signal receiver 210 may be composed of compressed video, audio, and additional information as IPTV data received through the Internet. At this time, the compressed image may generate video raw data by performing decompression according to the MPEGx/H.264 standard as in the cable TV receiver. In addition, the compressed voice may perform decompression according to the MPEGx/AC3/AAC standard to generate audio raw data. In addition, additional information may be transmitted to the main control unit 220.

The data communication I/F unit 260 is a data communication channel that performs data communication separate from TV broadcast content reception received from the broadcast signal receiving unit 210, and the communication network 120 is connected according to an embodiment of the present invention. Through this, data communication with the external server 270 including the service server 140 and the image analysis server 150 may be performed.

When the search request signal for the broadcast image is input from a viewer (eg, through a remote controller, etc.), the main control unit 220 captures the current broadcast image and provides it to the processor 250 to perform a search for the broadcast image Can be controlled. In addition, the main control unit 220 analyzes the captured broadcast image by capturing the broadcast image received through the broadcast signal receiving unit 210 according to another embodiment of the present invention, and analyzes the analyzed search term by the service server 140 and the image The main control function of the TV 100 for performing service content provision may be performed by transmitting to the external server 270 including the analysis server 150. The main control unit 220 may control TV setting changes, such as TV channel change and voice control, control Internet access, and control various applications (application programs) installed in the TV 100.

By exchanging information with the video analysis server 150 through the data communication I/F unit 260, the video analysis result is received, the priority is determined, and the search target determined based on the priority is input to the service server 140 Thus, it is possible to control the service content corresponding to the search target to be provided through the TV 100.

The buffer unit 230 may temporarily store data of a broadcast image received through the broadcast signal receiving unit 210 and then provide the TV image configuration unit 280. The TV image configuration unit 280 may configure an image to be displayed on a TV screen, including broadcast image data, and provide it to the display panel 291.

When capturing a broadcast video to search for a currently received broadcast video, capture the broadcast video at the time of the search request from the received video and temporarily store it in the buffer unit 230, and then supply the data of the temporarily stored broadcast video. It can be provided to the processor 250 through the control unit 220.

The broadcast image data temporarily stored in the buffer unit 230 may be stored in the broadcast data storage unit 240 for a predetermined period of time as a user function setting or default function. When the processor 250 receives an instruction according to an image search execution after the application is executed, the buffer unit 230 analyzes the temporarily stored broadcast captured image data to search for information related to the image.

As described above, the processor 250 may process various functions according to execution of an installed application according to an embodiment of the present invention. The processor 250 inputs the captured broadcast image to the plurality of image analysis servers 150, receives image analysis results corresponding to the captured broadcast images from each of the plurality of image analysis servers 150, and receives the image analysis results. Scoring determines the priority of the video analysis results, determines any one of the video analysis results based on the priority, and inputs the search target to the service server 140 to obtain service content corresponding to the search target The service content is displayed on the screen.

The TV image configuration unit 280 may combine each display area provided through the above-described broadcast signal receiving unit 210 and the processor 250 in a form set by a user to generate a TV display area. The TV display area may be divided into, for example, a TV display area, an Internet display area, a service content display area, or the like, or may be generated by overlapping each area. At this time, in a form in which each display area is combined, each display area is independently scaled according to a user's preference to generate a display screen for TV output, or a specific display area is transparent or translucent to process the TV. A display screen may be generated. In addition, the service content may be displayed on the screen together with an area to be recognized. Hereinafter, a detailed structure of the processor 250 will be described with reference to FIG. 3.

3 is a diagram for describing a detailed configuration of a processor according to an embodiment.

Referring to FIG. 3, the processor 250 according to an embodiment includes an image capture request unit 310, a capture information receiving unit 320, an image recognition unit 330, an analysis result display unit 340, an analysis result, and a target service It may be configured to include a display unit 360. In the embodiment of FIG. 3, each component of the processor 250 is separately illustrated in the drawings to indicate that it can be separated functionally and logically, and it is physically necessarily a separate component or is implemented as a separate code It is not done.

When a broadcast search request is performed to obtain information related to a screen currently being broadcast while watching a broadcast through a TV, the video capture request unit 310 may request a screen capture of the video currently being broadcast according to the execution of an application.

As described above, an image currently being broadcast is captured through the main control unit 220, and the captured information receiving unit 320 may receive the captured broadcast image (eg, an image file).

The image recognition unit 330 may input captured broadcast images to the plurality of image analysis servers 150 and receive image analysis results corresponding to the captured broadcast images from each of the plurality of image analysis servers. In addition, the video analysis results received from each of the plurality of video analysis servers 150 may be scored to determine the priority of the video analysis results, and any one of the video analysis results may be determined as a search target based on the priority. The image recognition unit 330 may include a person search unit 331, a shopping search unit 332, and a knowledge search unit 333, and through each unit, a person recognition server 334, a product recognition server 335, The captured broadcast image may be input to the knowledge recognition server 336.

The image analysis server 150 may include at least one of a person recognition server 334, a product recognition server 335, a knowledge recognition server 336, and a compound recognition server (not shown). When a person image is included in the captured broadcast image, the person recognition server 334 may output a keyword corresponding to the person image and an accuracy score of the person image. The keyword may be, for example, a person's name. The name may include the person's real name as well as the name of the activity. When a product image is included in the captured broadcast image, the product recognition server 335 may output sales information corresponding to the product and an accuracy score of the product image.

The image analysis server 150 may include an external image analysis server. For example, the image analysis server 150 may include a face recognition API, a shopping search API, a knowledge search API, and a complex search API. In the case of using an external image analysis server, the image retrieval device can utilize the image analysis results received from the external image analysis server, but the image retrieval device does not perform image analysis on its own, which can speed up processing.

Based on the accuracy score of the recognition result included in the image analysis result, the priority of the image analysis result may be determined. When each of the image analysis servers 150 outputs one or more image analysis results, priority may be determined based on the result with the highest accuracy score. For example, when the characters'a','b', and'c' are output from the person recognition server, another image analysis server may display the results with the highest accuracy scores among the keywords'a','b', and'c' ( Priority can be determined by comparing with the results of the video analysis of 150). The method of determining the priority may be defined in various ways and is not limited to the embodiments described herein.

The image recognition unit 330 may determine an image analysis result whose priority is equal to or higher than a predetermined priority as a search target. For example, the image recognition unit 330 may determine the image analysis result having the highest priority as a search target, and may also determine the image analysis results corresponding to the second priority as the search target. The method for determining the search target may be defined in various ways and is not limited to the embodiments described herein.

When the image recognition unit 330 determines the search target, the analysis result display unit 340 determines the service server according to the type of search target by providing the search target to the analysis result display unit 340, and determines the search target. The service content can be obtained by inputting it to a service server.

A service server 140 suitable for each search target may be determined in advance according to the type of search target. For example, when the search target is a video analysis result of the person recognition server, the service server 140 may include at least one of a person search server, a media search server, and a photo search server, and a person search server and a media search server. And inputting a video analysis result of the person recognition server to at least one of the photo search servers to output service content corresponding thereto. Alternatively, when the search target is an image analysis result of the product recognition server, the service server 140 may include various types of shopping search servers.

The analysis result and the target service display unit 360 provide service content information to the display panel 391 to display the service content in a specific area on the screen as shown in FIGS. 5 and 6. The analysis result and the target service display unit 360 may provide a list of service contents according to a predetermined criterion when there are multiple service contents. In addition, the analysis result and the target service display unit 360 may display the service content on the screen together with an area to be recognized.

In the present specification, each functional unit may mean a functional and structural combination of hardware for performing the technical idea of the present invention and software for driving the hardware. For example, each functional unit may mean a logical unit of a predetermined code and a hardware resource for performing the predetermined code, and does not necessarily mean a physically connected code or a type of hardware. It can be easily deduced from the average expert in the technical field of the invention.

4 is a flowchart illustrating a method of determining a priority of an image analysis result according to an embodiment.

Referring to FIG. 4, steps 410 to 470 may be performed by the image search apparatus 200 described above with reference to FIG. 2. The image retrieval apparatus 200 may be implemented by one or more hardware modules, one or more software modules, or various combinations thereof. Since the description of FIGS. 1 to 3 is also applicable to FIG. 4, description of overlapping contents is omitted.

In step 410, the application may be executed in the set-top box.

In step 415, the image search apparatus 200 may analyze a broadcast screen.

In step 420, the image retrieval apparatus 200 may capture a broadcast screen.

In step 425, the image search apparatus 200 may receive an image analysis result corresponding to the captured image (image) using the image recognition server 150.

For example, in step 430, the image retrieval apparatus 200 may obtain an accuracy score of the person recognition result from the person recognition server. In step 435, the image search apparatus 200 may obtain an accuracy score of the product recognition result from the product recognition server. In step 440, the image search apparatus 200 may obtain an accuracy score of the knowledge recognition result from the knowledge recognition server.

The image search apparatus 200 may determine the priority of the image analysis result by scoring the image analysis result. The image search apparatus 200 compares the accuracy scores of the recognition results included in the image analysis results received from the person recognition server, the product recognition server, the knowledge recognition server, and the composite recognition server, and prioritizes them based on the accuracy scores of the recognition results Can decide.

For example, in step 445, the image search apparatus 200 may compare the accuracy score of the person recognition result with the accuracy score of the product recognition result, and when the accuracy score of the person recognition result is higher, the accuracy of the person recognition result By comparing the score and the accuracy score of the knowledge recognition result, if the accuracy score of the person recognition result is high, the person recognition result is determined as a search target, and in step 455, the person recognition result is input to the service server 140 to input the person information. You can recommend content related to offers and people.

Similarly, when the accuracy of the knowledge recognition result is the highest, the knowledge service may be provided in step 465, and when the accuracy of the product recognition result is highest, the shopping information and service related to the product may be provided in step 470. Can.

5 is a diagram illustrating an embodiment in which a person recognition result according to an embodiment is a search target.

Referring to FIG. 5, the image search apparatus according to an embodiment may receive the captured broadcast image 510. The captured broadcast image may be input to a plurality of image analysis servers, and image analysis results corresponding to the captured broadcast image may be received from each of the image analysis servers. Since the accuracy score of the person recognition result is the highest, the person recognition result can be determined as a search target.

A person recognition result may be input to a service server to obtain service content corresponding to a search target. Since the search target is a person recognition result, the service server may include a person search server, a media search server, and a photo search server. The person recognition results can be input to the person search server to obtain the service content 520, the person recognition results can be entered to the media search server to obtain the service content 530, and the person recognition results to the photo search server. The service content 540 may be obtained by inputting it.

6 is a diagram illustrating an embodiment in which a product recognition result according to an embodiment is a search target.

Referring to FIG. 6, the image retrieval apparatus according to an embodiment may receive the captured broadcast image 610. The captured broadcast image may be input to a plurality of image analysis servers, and image analysis results corresponding to the captured broadcast image may be received from each of the image analysis servers. Since the accuracy score of the product recognition result is the highest, the product recognition result can be determined as a search target.

The service content corresponding to the search target may be obtained by inputting the product recognition result to the service server. Since the search target is a product recognition result, the service server may include a shopping search server. A service content 620 may be obtained by inputting a product recognition result to a shopping search server.

The image retrieval device may generate a TV display area by combining broadcast video captured in a form set by a user and service content. The TV display area may be divided into, for example, a TV display area, an Internet display area, a service content display area, or the like, or may be generated by overlapping each area. At this time, in a form in which each display area is combined, each display area is independently scaled according to a user's preference to generate a display screen for TV output, or a specific display area is transparent or translucent to process the TV. A display screen may be generated. In addition, the service content may be displayed on the screen together with an area to be recognized.

An image search method according to an embodiment may include receiving a captured broadcast image, inputting the captured broadcast image into a plurality of image analysis servers, and analyzing image analysis results corresponding to the captured broadcast image from each of the plurality of image analysis servers. Receiving, scoring video analysis results to determine the priority of the video analysis results, determining any one of the video analysis results based on the priority, and entering a search target into a service server to search targets And obtaining service content corresponding to and processing to display the service content on the screen.

The embodiments described above may be implemented with hardware components, software components, and/or combinations of hardware components and software components. For example, the devices, methods, and components described in the embodiments include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors (micro signal processors), microcomputers, and field programmable gates (FPGAs). It can be implemented using one or more general purpose computers or special purpose computers, such as arrays, programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodied in the transmitted signal wave. The software may be distributed on networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the medium may be specially designed and constructed for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specifically configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler.

As described above, although the embodiments have been described by the limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

Receiving the captured broadcast image;
Inputting the captured broadcast images into a plurality of image analysis servers, and receiving image analysis results corresponding to the captured broadcast images from each of the plurality of image analysis servers;
Scoring the image analysis results to determine a priority of the image analysis results;
Determining any one of the image analysis results as a search target based on the priority;
Inputting the search target into a service server to obtain service content corresponding to the search target; And
Processing the service content to be displayed on a screen
Including, image search method.
According to claim 1,
The video analysis server
An image retrieval method comprising at least one of a person recognition server, a product recognition server, a knowledge recognition server, and a compound recognition server.
According to claim 1,
The video analysis results
An image retrieval method comprising an area targeted for recognition included in the captured broadcast image, a recognition result for the area targeted for recognition, and an accuracy score of the recognition result.
According to claim 1,
Determining the priority is
Determining the priority based on an accuracy score of a recognition result included in the image analysis result
Including, image search method.
According to claim 1,
The step of processing to display the service content on the screen is
Processing the service content to be displayed on the screen together with an area to be recognized.
Including, image search method.
According to claim 1,
The step of obtaining the service content is
Determining the predetermined service server according to the type of the search target; And
Obtaining the service content by inputting the search target into the determined service server
Including, image search method.
According to claim 1,
The step of receiving the captured broadcast image is
Receiving a search request for a video being broadcast;
Capturing a broadcast image at the time of the search request according to the search request; And
Receiving the captured broadcast image
Including, image search method
According to claim 1,
The step of receiving the image analysis result
Inputting the captured broadcast image into a person recognition server, and when the captured broadcast image includes a person image, receiving a keyword corresponding to the person image and an accuracy score of the person image;
Including, image search method.
According to claim 1,
The step of receiving the image analysis result
Inputting the captured broadcast image into a product recognition server, and if a product image is included in the captured broadcast image, receiving sales information corresponding to the product and an accuracy score of the product image
Including, image search method.
According to claim 1,
When the search target is an image analysis result received from the person recognition server,
The step of obtaining the service content corresponding to the search target is
Inputting the search target into at least one of a person search server, a media search server, and a photo search server to obtain service content corresponding to the search target
Including, image search method.
According to claim 1,
The video analysis server
At least one of a person recognition server, a product recognition server, a knowledge recognition server, and a complex recognition server,
Determining the priority is
Comparing the accuracy scores of the recognition results included in the image analysis results received from the person recognition server, the product recognition server, the knowledge recognition server, and the composite recognition server; And
Determining the priority based on the accuracy score of the recognition result
Including, image search method.
According to claim 1,
The step of processing to display the service content on the screen is
When the service content is plural,
Providing a list of the service content according to a predetermined criterion
Including, image search method.
A computer program stored in a medium to execute the method of any one of claims 1 to 12 in combination with hardware.
A broadcast signal receiver for receiving the captured broadcast image; And
Input the captured broadcast image into a plurality of image analysis servers, and receive image analysis results corresponding to the captured broadcast image from each of the plurality of image analysis servers,
Scoring the video analysis results to determine the priority of the video analysis results,
Based on the priority, any one of the video analysis results is determined as a search target,
Input the search target to the service server to obtain service content corresponding to the search target,
Processor for processing to display the service content on the screen
Including, image search device.
The method of claim 14,
The video analysis server
An image retrieval device comprising at least one of a person recognition server, a product recognition server, a knowledge recognition server, and a compound recognition server.
The method of claim 14,
The video analysis results
An image retrieval apparatus including an area targeted for recognition included in the captured broadcast image, a recognition result for the area targeted for recognition, and an accuracy score of the recognition result.
The method of claim 14,
The processor
The image retrieval apparatus determines the priority based on the accuracy score of the recognition result included in the image analysis result.
The method of claim 14,
The processor
An image retrieval apparatus that processes the service content to be displayed on the screen together with an area to be recognized.
The method of claim 14,
The processor
The predetermined service server is determined according to the type of the search target,
An image search apparatus for acquiring the service content by inputting the search target into the determined service server.
The method of claim 14,
The video analysis server
At least one of a person recognition server, a product recognition server, a knowledge recognition server, and a complex recognition server,
The processor
Comparing the accuracy scores of the recognition results included in the image analysis results received from the person recognition server, the product recognition server, the knowledge recognition server, and the composite recognition server,
And determining the priority based on the accuracy score of the recognition result.

Images