KR102664418B1 - Display apparatus and service providing method of thereof - Google Patents

Abstract

Provided is a display device and a service provision method thereof. A display device according to an embodiment of the present disclosure recognizes a display unit that displays a content playback screen and content being played, recognizes at least one object on the currently displayed screen, and determines a user using the display device. It may include a processor that selects one of at least one recognized object based on the determined user's information and provides a service related to the selected object to the determined user.

Description

Display device and service providing method thereof {DISPLAY APPARATUS AND SERVICE PROVIDING METHOD OF THEREOF}

This disclosure relates to a display device and a service provision method thereof, and more specifically, to a display device capable of providing a customized ACR service to a user currently using the display device and a service provision method thereof.

Additionally, the present disclosure relates to an artificial intelligence (AI) system and its applications that mimic the functions of the human brain, such as cognition and judgment, using machine learning algorithms such as deep learning.

An artificial intelligence (AI) system is a computer system that implements human-level intelligence, and unlike existing rule-based smart systems, it is a system in which machines learn and make decisions on their own and become smarter. As artificial intelligence systems are used, the recognition rate improves and users' preferences can be more accurately understood, and existing rule-based smart systems are gradually being replaced by deep learning-based artificial intelligence systems.

Artificial intelligence technology consists of machine learning (deep learning) and element technologies using machine learning. Machine learning is an algorithmic technology that classifies and learns the characteristics of input data on its own, and elemental technology is a technology that uses machine learning algorithms such as deep learning to mimic the functions of the human brain such as cognition and judgment, including linguistic understanding and visual It consists of technical areas such as understanding, reasoning/prediction, knowledge expression, and motion control.

The various fields where artificial intelligence technology is applied are as follows. Linguistic understanding is a technology that recognizes and applies/processes human language/characters and includes natural language processing, machine translation, conversation systems, question and answer, and voice recognition/synthesis. Visual understanding is a technology that recognizes and processes objects like human vision, and includes object recognition, object tracking, image search, person recognition, scene understanding, spatial understanding, and image improvement. Inferential prediction is a technology that judges information to make logical inferences and predictions, and includes knowledge/probability-based inference, optimization prediction, preference-based planning, and recommendation. Knowledge expression is a technology that automatically processes human experience information into knowledge data, and includes knowledge construction (data creation/classification) and knowledge management (data utilization). Motion control is a technology that controls the autonomous driving of vehicles and the movement of robots, and includes motion control (navigation, collision, driving), operation control (behavior control), etc.

Meanwhile, in recent years, ACR (Automatic Content Recognition) technology has been developed that can recognize content currently displayed on a display device. If it can know what content the user is currently watching, the display device can provide intelligent services such as targeted advertising, content recommendation, and related information search.

However, display devices used at home or in public places are devices that are shared by many people. Therefore, information about the same product or service is inevitably provided to multiple people using the display device.

Since the products and services preferred by various users may be different, there is a problem in that conventional ACR-based services can provide services suitable only for some users.

The present disclosure is intended to solve the above-mentioned problems, and provides a personalized ACR-based service by clustering objects recognized within the screen, determining the user of the current display device, and selecting clustered objects suitable for the determined user. The purpose is to provide a display device capable of providing a display device and a method of providing its service.

In order to achieve the above object, a display device according to an embodiment of the present disclosure recognizes a display unit that displays a content playback screen and the content being played, recognizes at least one object on the currently displayed screen, and uses the display device to It may include a processor that determines the user using the device, selects one of the recognized at least one object based on information about the determined user, and provides a service related to the selected object to the determined user.

And, it further includes a communication unit that communicates with a server, wherein the processor extracts characteristics of the screen displayed on the display unit to generate a fingerprint, and queries the server for content information corresponding to the generated fingerprint. can be controlled, and content can be recognized using content information received from the server.

In addition, the processor recognizes an object corresponding to a product on the screen displayed on the display unit, recognizes an object corresponding to a person on the displayed screen based on metadata of the recognized content, and Products can be clustered.

In addition, the processor determines whether there is a relationship between the person and the product based on the position on the displayed screen, clusters the recognized person and the product if there is a relationship, and filters the product if there is no relationship. there is.

Additionally, if the determined number of users is two or more, the processor may determine one user among the two or more users as the user using the display device every time the screen is changed.

In addition, the processor may determine a preference ranking for the at least one object, and determine that, among the two or more users, the user with the highest preference for the object determined as the highest priority is the user using the display device. there is.

Additionally, it may further include an input unit that receives the user's biometric information, and the processor may determine the user using the display device by comparing the biometric information input through the input unit with pre-stored biometric information.

And, it may further include a camera for capturing an image, and the processor may determine that a user included in an image of a preset area located in front of the display device captured through the camera is a user using the display device. there is.

Meanwhile, a service providing method of a display device according to an embodiment of the present disclosure for achieving the above object includes recognizing content being played, recognizing at least one object on a currently displayed screen of the content, and displaying the display device. It includes determining a user using the device, selecting one of the recognized at least one object based on information on the determined user, and providing a service related to the selected object to the determined user. can do.

And, the step of recognizing the content includes extracting characteristics of the displayed screen to generate a fingerprint, querying the server for content information corresponding to the generated fingerprint, and content information received from the server. It may include a step of recognizing content.

In addition, the step of recognizing the object includes recognizing an object corresponding to a product on the displayed screen, recognizing an object corresponding to a person on the displayed screen based on metadata of the recognized content, and the recognition It may include a step of clustering the people and products.

And, the clustering step includes determining whether there is a relationship between the person and the product based on the position on the displayed screen, and if there is a relationship, clustering the recognized person and the product, and if there is no relationship, clustering the product. A filtering step may be included.

Additionally, in the step of determining the user, if the determined users are two or more, one user among the two or more users may be determined as the user using the display device every time the screen is changed.

And, the step of determining the user includes determining a preference ranking for the at least one object, and selecting the user with the highest preference for the object determined as the highest priority among the two or more users using the display device. It may include a step of determining the user.

In addition, the step of determining the user may include receiving the user's biometric information and comparing the input biometric information with pre-stored biometric information to determine the user using the display device. .

And, the step of determining the user includes taking an image of a preset area located in front of the display device and determining the user included in the captured image as a user using the display device. can do.

In addition, in a display device using an artificial intelligence neural network model according to an embodiment of the present disclosure to achieve the above object, the display unit that displays the content playback screen and the content being played are recognized, and the recognized Input content into an artificial intelligence neural network model to recognize at least one object on the currently displayed screen, determine a user using the display device, and at least one object recognized based on the determined user information. It may include a processor that selects one of the objects and provides a service related to the selected object to the determined user.

According to various embodiments of the present disclosure as described above, a display device can provide customized ACR-based services to each user.

1 is a schematic block diagram for explaining the configuration of a display device according to an embodiment of the present disclosure;
2 is a block diagram illustrating in detail the configuration of a display device according to an embodiment of the present disclosure;
3 is a block diagram of a processor according to some embodiments of the present disclosure;
4A is a block diagram of a data learning unit according to some embodiments of the present disclosure;
4B is a block diagram of a data recognition unit according to some embodiments of the present disclosure;
5 is a diagram for explaining a display system according to an embodiment of the present disclosure;
Figure 6 is a diagram for explaining the ACR process;
7 is a diagram for explaining the object recognition process;
Figure 8a is a diagram for explaining a method of recognizing an object by extracting feature points;
Figure 8b is a diagram for explaining a method of recognizing objects through learning;
9 is a diagram for explaining object clustering;
Figure 10 is a diagram illustrating a case where multiple user information is stored, and
11 and 12 are flowcharts illustrating a method of providing a service by a display device according to various embodiments of the present disclosure.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the attached drawings. In describing the present disclosure, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, the detailed description will be omitted. Additionally, the terms described below are defined in consideration of the functions in the present disclosure, and may vary depending on the user, operator, or custom. Therefore, the definition should be made based on the contents throughout this specification.

Terms containing ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by the terms. Terms are used only to distinguish one component from another. For example, a first component may be named a second component without departing from the scope of the present invention, and similarly, the second component may also be named a first component. The term and/or includes a combination of a plurality of related items or any one item among a plurality of related items.

The terms used in this specification are used to describe embodiments and are not intended to limit and/or limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as include or have are intended to designate the presence of features, numbers, operations, components, parts, or a combination thereof described in the specification, but are not intended to indicate the presence of one or more other features, numbers, operations, or configurations. It should be understood that this does not exclude in advance the possibility of the presence or addition of elements, parts, or combinations thereof.

In an embodiment, a 'module' or 'unit' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware or software. Additionally, a plurality of 'modules' or a plurality of 'units' may be integrated into at least one module and implemented with at least one processor, except for 'modules' or 'units' that need to be implemented with specific hardware.

Meanwhile, in the present disclosure, the term user may refer to a person using an electronic device or a device (eg, an artificial intelligence electronic device) using the electronic device.

Hereinafter, the present disclosure will be described in detail using the attached drawings.

FIG. 1 is a schematic block diagram for explaining the configuration of a display device 100 according to an embodiment of the present disclosure. The display device 100 may be a smart TV, but this is only an example and may be implemented as various types of devices such as projection TVs, monitors, kiosks, laptop PCs, tablets, smartphones, PDAs, electronic photo frames, and table displays. You can.

Referring to FIG. 1, the display device 100 may include a display unit 110 and a processor 120.

The display unit 110 can display various video contents, information, UI, etc. provided by the display device 100. For example, the display unit 110 may display a playback screen of various contents provided in the form of live broadcasting or VOD.

The processor 120 can recognize the content currently being played. For example, the processor 120 may generate a fingerprint by extracting features of the displayed screen. Additionally, the processor 120 may perform Automatic Content Recognition (ACR) by querying the server 200 for the generated fingerprint. As another example, the processor 120 may perform ACR by comparing the generated fingerprint with a fingerprint database stored in the memory 160.

Fingerprints are feature data extracted from video and audio signals included in each frame of content. Unlike text-based metadata, fingerprints can reflect the unique characteristics of the signal itself.

The term 'fingerprint' used in this specification may mean a single fingerprint for a specific image, or in some cases, a fingerprint list consisting of multiple fingerprints for a specific image.

The processor 120 may determine the user using the display device 100. For example, the processor 120 may receive biometric information and determine who is currently using the display device 100. As another example, the processor 120 may photograph a user and perform face detection (FD) to determine who is currently using the display device 100.

The processor 120 may provide a customized service using the determined user's preferences, usage history, etc. If multiple users are determined, the processor 120 can determine the user to whom the service will be provided for each frame (or screen) of content.

The term 'frame' used in this specification refers to a series of data containing information about audio or images. A frame can be data for audio or images over a certain period of time. In the case of digital video content, it may consist of 30 to 60 image data per second, and each of these 30 to 60 image data can be referred to as a frame. For example, when used as the current frame and next frame of video content, a frame can refer to each continuously displayed video screen included in the content.

The processor 120 may recognize at least one object in the currently displayed screen (or frame). For example, the object may be a person or product on the screen. The processor 120 may cluster recognized people and products. That is, the processor 120 can classify objects recognized as products related to people.

Based on the determined user's preference for the person, the processor 120 may select one of the classified objects. And the processor 120 may provide services related to the selected object to the determined user.

As in the above-described embodiment, the display device 100 may provide a personalized ACR-based service.

FIG. 2 is a block diagram for explaining in detail the configuration of the display device 100 according to an embodiment of the present disclosure. Referring to FIG. 2, the display device 100 includes a display unit 110, a processor 120, a communication unit 130, an input unit 140, a camera 150, a memory 160, an image receiver 170, and an image It may include a processing unit 180.

In addition to the components shown in the embodiment of FIG. 2, the electronic device 100 may include various components such as an audio processing unit (not shown), an audio output unit (not shown), and a power supply unit (not shown). Additionally, of course, the electronic device 100 is not necessarily limited to being implemented by including all of the configurations shown in FIG. 2 .

The display unit 110 can display various video contents, information, UI, etc. provided by the display device 100. The display unit 110 is implemented with a liquid crystal display (LCD), an organic light emitting diode (OLED), or a plasma display panel (PDP), and is used as a display device 100. You can display various screens that can be provided through .

The communication unit 130 performs a function of communicating with the server 200 according to various types of communication methods. The communication unit 130 can transmit and receive fingerprint data with the server 200 connected wired or wirelessly. Additionally, the communication unit 130 may transmit information on a specific frame of content to the server 200 and request recognition of an object included in the frame. Additionally, the communication unit 130 may stream video data from an external server. The communication unit 130 may include various communication chips that support wired and wireless communication. For example, the communication unit 130 may include a chip that operates in a wired LAN, wireless LAN, WiFi, BT (Bluetooth), or NFC (Near Field Communication) method.

The input unit 140 may receive various user commands for controlling the display device 100. Additionally, the input unit 140 can receive input of the user's biometric information. The user's biometric information may include fingerprint information, iris information, voiceprint information, etc. For example, the input unit 140 is implemented as a fingerprint recognition sensor located on the ACR function execution button, and can collect fingerprint information of the user who pressed the ACR function execution button.

The input unit 140 may be implemented as a button, a touch pad, etc., or may be implemented as a separate device such as a remote control device. When the input unit 140 is implemented as a touch pad, it may be implemented in the form of a touch screen that is combined with the display unit 110 to form a mutual layer structure. The touch screen can detect the touch input location, area, and pressure of the touch input.

The camera 150 can capture still images or videos. For example, the camera 150 may always capture a specific capture area. Using the difference between captured image frames, the display device 100 can detect that a change has occurred in the captured image frame. For example, the camera 150 may capture a certain area in front of the display device 100, and the processor 120 may determine whether a user exists from the captured image. Additionally, the processor 120 may determine who the user included in the image is through facial recognition or the like.

The camera 150 may be implemented with an image sensor such as a charge coupled device (CCD) or complementary metal oxide semiconductor (CMOS). A CCD is a device in which each MOS (Metal-Oxide-Silicon) capacitor is located very close to each other, and charge carriers are stored and transported in the capacitor. CMOS image sensors use CMOS technology that uses control circuits and signal processing circuits as peripheral circuits to create MOS transistors as many as the number of pixels and adopt a switching method that sequentially detects output using these. It is a device that does.

The memory 160 can store various modules, software, and data for driving the display device 100. For example, the memory 160 may store biometric information, viewing history information, preference information, etc. for at least one user.

The memory 160 can be implemented in the form of flash memory, hard disk, etc. For example, the memory 160 may include ROM for storing programs for performing operations of the display device 100, RAM for temporarily storing data according to operating operations of the display device 100, etc. . In addition, an Electrically Erasable and Programmable ROM (EEPROM) for storing various reference data may be further provided.

The video receiver 170 receives video content data through various sources. For example, the video receiver 170 may receive broadcast data from an external broadcasting station. As another example, the video receiver 170 may receive video data from an external device (e.g., set-top box, DVD player), or receive video data streamed from an external server through the communication unit 130.

The image processing unit 180 performs image processing on image data received from the image receiving unit 170. The image processing unit 180 may perform various image processing such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion on image data.

The processor 120 can control the above-described configurations of the display device 100. For example, the processor 120 may control the communication unit 130 to query content information corresponding to the fingerprint generated in the server 200. The processor 120 may be implemented as a single CPU, or may be implemented as a plurality of processors and an IP that performs a specific function.

Meanwhile, the processor 120 may be an existing general-purpose processor (e.g., CPU or application processor) to perform the above-described operations, but for specific operations, a dedicated hardware chip for artificial intelligence (AI) may perform the operations. It can be done.

Hereinafter, the operation of the processor 120 will be described in more detail with reference to the drawings.

Figure 3 is a block diagram of the processor 120 according to some embodiments of the present disclosure. Referring to FIG. 3, the processor 120 according to some embodiments may include a data learning unit 121 and a data recognition unit 122.

The data learning unit 121 can learn standards for image analysis. The processor 120 may recognize an object in each image frame according to the learned criteria. The data learning unit 121 may determine what data to use to recognize objects included in the image. Additionally, the data learning unit 121 may learn a standard for recognizing an object using the determined data. The data learning unit 121 may learn standards for image analysis by acquiring data to be used for learning and applying the obtained data to a data recognition model to be described later.

The data recognition unit 122 can recognize the situation from predetermined data using a learned data recognition model. The data recognition unit 122 may acquire predetermined data according to a preset standard through learning and use a data recognition model using the acquired data as an input value. For example, using the learned feature extraction model, the data recognition unit 122 can recognize an object on the currently displayed screen. Additionally, the data recognition unit 122 may update the data recognition model by using the data obtained as a result of applying the data recognition model again as an input value.

At least one of the data learning unit 121 and the data recognition unit 122 may be manufactured in the form of one or more hardware chips and mounted on the display device 100. For example, at least one of the data learning unit 121 and the data recognition unit 122 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general-purpose processor (e.g., It may be manufactured as part of a CPU or application processor) or a graphics processor (eg, GPU, ISP) and mounted on the various display devices 100 described above.

In the embodiment of FIG. 3, the data learning unit 121 and the data recognition unit 122 are both mounted on the display device 100, but they may be mounted on separate devices. For example, one of the data learning unit 121 and the data recognition unit 122 may be included in the display device 100, and the other may be included in the server 200. In addition, the data learning unit 121 and the data recognition unit 122 are connected to each other by wire or wirelessly, so that model information built by the data learning unit 121 can be provided to the data recognition unit 122, and the data recognition unit 122 The data input at 122 may be provided to the data learning unit 121 as additional learning data.

Meanwhile, at least one of the data learning unit 121 and the data recognition unit 122 may be implemented as a software module. When at least one of the data learning unit 121 and the data recognition unit 122 is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable recording medium. At least one software module may be provided by an operating system (OS) or a predetermined application. Alternatively, part of at least one software module may be provided by the OS, and the remaining part may be provided by a predetermined application.

FIG. 4A is a block diagram of the data learning unit 121 according to some embodiments of the present disclosure. Referring to FIG. 4A, the data learning unit 121 according to some embodiments includes a data acquisition unit 121-1, a preprocessing unit 121-2, a learning data selection unit 121-3, and a model learning unit 121. -4) and a model evaluation unit 121-5.

The data acquisition unit 121-1 may acquire data necessary for determining the situation. For example, the data acquisition unit 121-1 may acquire an image frame by capturing a screen played on the display unit 110. Additionally, the data acquisition unit 121-1 may receive image data from an external device such as a set-top box. Image data may consist of multiple image frames. Additionally, the data acquisition unit 121-1 may receive video data for learning from the server 200 or a network such as the Internet.

The preprocessor 121-2 may preprocess the acquired data so that the acquired data can be used for learning to determine the situation. The pre-processing unit 121-2 may process the acquired data into a preset format so that the model learning unit 121-4, which will be described later, can use the acquired data for learning to determine the situation.

For example, the preprocessor 121-2 may create image frames of the same format by performing decoding, scaling, noise filtering, and resolution conversion on the input image data. Additionally, the preprocessor 121-2 may remove the background portion from the input image frame and convert it into an image suitable for object recognition.

The learning data selection unit 121-3 may select data required for learning from preprocessed data. The selected data may be provided to the model learning unit 121-4. The learning data selection unit 121-3 may select data required for learning from preprocessed data according to preset standards for situation determination. Additionally, the learning data selection unit 121-3 may select data according to preset criteria through learning by the model learning unit 121-4, which will be described later.

For example, at the beginning of learning, the learning data selection unit 121-3 may remove image frames with high similarity from preprocessed image frames. That is, for initial learning, the learning data selection unit 121-3 may select data with low similarity to learn a standard that is easy to distinguish.

Additionally, the learning data selection unit 121-3 may select preprocessed image frames that commonly satisfy one of the criteria preset by learning. Through this, the model learning unit 121-4 can learn a different standard from the already learned standard.

The model learning unit 121-4 can learn standards for how to judge a situation based on learning data. Additionally, the model learning unit 121-4 may learn standards for what learning data should be used to determine the situation.

For example, the model learning unit 121-4 may learn physical characteristics that distinguish images by comparing a plurality of image frames. The model learning unit 121-4 can learn image analysis standards through the ratio of foreground and background in the image frame, object size, location, arrangement type, feature point extraction, etc.

Additionally, the model learning unit 121-4 can train a data recognition model used for situation determination using learning data. In this case, the data recognition model may be a pre-built model. For example, a data recognition model may be a pre-built model that receives basic training data (for example, sample image frames, etc.) as input.

A data recognition model may be built considering the application field of the recognition model, the purpose of learning, or the computer performance of the device. The data recognition model may be, for example, a model based on a neural network. For example, models such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), and Bidirectional Recurrent Deep Neural Network (BRDNN) may be used as data recognition models, but are not limited thereto.

Additionally, the electronic device 100 may use an artificial intelligence agent to perform the operations described above. At this time, the artificial intelligence agent is a dedicated program to provide AI (Artificial Intelligence)-based services (e.g., voice recognition service, secretary service, translation service, search service, etc.), and is an existing general-purpose processor (e.g., It may be executed by a CPU) or a separate AI-specific processor (e.g., GPU, etc.).

According to various embodiments, when there are a plurality of pre-built data recognition models, the model learning unit 121-4 selects a data recognition model with a high correlation between the input learning data and the basic learning data as a data recognition model to be learned. You can decide. In this case, the basic learning data may be classified by data type, and the data recognition model may be built in advance for each data type. For example, the basic training data is categorized by various criteria such as the region where the training data was created, the time the training data was created, the size of the training data, the genre of the training data, the creator of the training data, the type of object in the training data, etc. It may be.

Additionally, the model learning unit 121-4 may train a data recognition model using, for example, a learning algorithm including error back-propagation or gradient descent. .

For example, the model learning unit 121-4 can learn a data recognition model through supervised learning using training data as input. As another example, the model learning unit 121-4 creates a data recognition model through unsupervised learning, which discovers a standard for situation judgment by learning the type of data needed for situation judgment without separate guidance. It can be learned. As another example, the model learning unit 121-4 may learn a data recognition model through reinforcement learning that uses feedback on whether the result of situational judgment based on learning is correct.

Additionally, when the data recognition model is learned, the model learning unit 121-4 may store the learned data recognition model. In this case, the model learning unit 121-4 may store the learned data recognition model in the memory 160 of the display device 100. Alternatively, the model learning unit 121-4 may store the learned data recognition model in the memory of the server 200 connected to the electronic device through a wired or wireless network.

In this case, the memory 160 where the learned data recognition model is stored may also store commands or data related to at least one other component of the display device 100. Additionally, memory 160 may store software and/or programs. For example, a program may include a kernel, middleware, an application programming interface (API), and/or an application program (or “application”), etc.

The model evaluation unit 121-5 inputs evaluation data into the data recognition model, and, if the recognition result output from the evaluation data does not meet a predetermined standard, can cause the model learning unit 121-4 to learn again. there is. In this case, the evaluation data may be preset data for evaluating the data recognition model.

In the initial recognition model construction stage, evaluation data may be image frames for two types of objects. The evaluation data can then be replaced by a set of image frames with increasing types of objects. Through this, the model evaluation unit 121-5 can gradually verify the performance of the data recognition model.

For example, the model evaluation unit 121-5 applies a predetermined criterion when the number or ratio of evaluation data for which the recognition result is inaccurate among the recognition results of the data recognition model learned for the evaluation data exceeds a preset threshold. It can be evaluated as unsatisfactory. For example, when the predetermined standard is defined as a ratio of 2%, and the learned data recognition model outputs incorrect recognition results for more than 20 evaluation data out of a total of 1000 evaluation data, the model evaluation unit 121-5 It can be evaluated that the learned data recognition model is not suitable.

Meanwhile, when there are a plurality of learned data recognition models, the model evaluation unit 121-5 evaluates whether each learned video recognition model satisfies a predetermined standard, and recognizes the model that satisfies the predetermined standard as the final data. You can decide as a model. In this case, when there are multiple models that satisfy the predetermined criteria, the model evaluation unit 121-5 may determine one or a predetermined number of models preset in descending order of evaluation scores as the final data recognition model.

Meanwhile, the data acquisition unit 121-1, pre-processing unit 121-2, learning data selection unit 121-3, model learning unit 121-4, and model evaluation unit 121 in the data learning unit 121. At least one of -5) may be manufactured in the form of at least one hardware chip and mounted on an electronic device. For example, at least one of the data acquisition unit 121-1, the pre-processing unit 121-2, the learning data selection unit 121-3, the model learning unit 121-4, and the model evaluation unit 121-5. One may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general-purpose processor (e.g., CPU or application processor) or a dedicated graphics processor (e.g., GPU, ISP). It may be manufactured as part of and mounted on the various display devices 100 described above.

In addition, the data acquisition unit 121-1, the preprocessing unit 121-2, the learning data selection unit 121-3, the model learning unit 121-4, and the model evaluation unit 121-5 are one electronic unit. It may be mounted on a device, or it may be mounted on separate electronic devices. For example, some of the data acquisition unit 121-1, preprocessing unit 121-2, learning data selection unit 121-3, model learning unit 121-4, and model evaluation unit 121-5. may be included in the display device 100, and the remaining portion may be included in the server 200.

Meanwhile, at least one of the data acquisition unit 121-1, the pre-processing unit 121-2, the learning data selection unit 121-3, the model learning unit 121-4, and the model evaluation unit 121-5 It can be implemented as a software module. At least one of the data acquisition unit 121-1, the preprocessing unit 121-2, the learning data selection unit 121-3, the model learning unit 121-4, and the model evaluation unit 121-5 is a software module. When implemented as a program module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable recording medium. At least one software module may be provided by an operating system (OS) or a predetermined application. Alternatively, part of at least one software module may be provided by the OS, and the remaining part may be provided by a predetermined application.

FIG. 4B is a block diagram of the data recognition unit 122 according to some embodiments of the present disclosure. Referring to FIG. 4B, the data recognition unit 122 according to some embodiments includes a data acquisition unit 122-1, a preprocessing unit 122-2, a recognition data selection unit 122-3, and a recognition result providing unit ( 122-4) and a model update unit 122-5.

The data acquisition unit 122-1 may acquire data necessary for determining the situation, and the preprocessing unit 122-2 may preprocess the acquired data so that the acquired data can be used for determining the situation. The pre-processing unit 122-2 may process the acquired data into a preset format so that the recognition result providing unit 122-4, which will be described later, can use the acquired data to determine the situation.

The recognition data selection unit 122-3 may select data necessary for situation determination from preprocessed data. The selected data may be provided to the recognition result provider 122-4. The recognition data selection unit 122-3 may select some or all of the preprocessed data according to preset criteria for situation determination. Additionally, the recognition data selection unit 122-3 may select data according to preset criteria through learning by the model learning unit 142-4, which will be described later.

The recognition result provider 122-4 may determine the situation by applying the selected data to the data recognition model. The recognition result providing unit 122-4 can provide recognition results according to the recognition purpose of the data. The recognition result providing unit 122-4 can apply the selected data to the data recognition model by using the data selected by the recognition data selection unit 122-3 as an input value. Additionally, the recognition result may be determined by a data recognition model. For example, according to the criteria determined in the data recognition model, the recognition result provider 122-4 may recognize the object by analyzing the selected image frame.

The model updating unit 122-5 may update the data recognition model based on the evaluation of the recognition result provided by the recognition result providing unit 122-4. For example, the model updating unit 122-5 provides the recognition result provided by the recognition result providing unit 122-4 to the model learning unit 121-4, so that the model learning unit 121-4 The data recognition model can be updated.

Meanwhile, the data acquisition unit 122-1, the pre-processing unit 122-2, the recognition data selection unit 122-3, the recognition result providing unit 122-4, and the model update unit in the data recognition unit 122 ( At least one of 122-5) may be manufactured in the form of at least one hardware chip and mounted on an electronic device. For example, among the data acquisition unit 122-1, pre-processing unit 122-2, recognition data selection unit 122-3, recognition result providing unit 122-4, and model update unit 122-5. At least one may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general-purpose processor (e.g., CPU or application processor) or graphics-specific processor (e.g., GPU, ISP). ) and may be mounted on the various display devices 100 described above.

In addition, a data acquisition unit 122-1, a preprocessing unit 122-2, a recognition data selection unit 122-3, a recognition result providing unit 122-4, and a model updating unit 122-5 are integrated into one electronic unit. It may be mounted on a device, or it may be mounted on separate electronic devices. For example, among the data acquisition unit 122-1, pre-processing unit 122-2, recognition data selection unit 122-3, recognition result providing unit 122-4, and model update unit 122-5. Some may be included in the display device 100, and others may be included in the server 200.

Meanwhile, at least one of the data acquisition unit 122-1, the pre-processing unit 122-2, the recognition data selection unit 122-3, the recognition result providing unit 122-4, and the model update unit 122-5. Can be implemented as a software module. At least one of the data acquisition unit 122-1, the preprocessing unit 122-2, the recognition data selection unit 122-3, the recognition result providing unit 122-4, and the model update unit 122-5 is software. When implemented as a module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable recording medium. At least one software module may be provided by an operating system (OS) or a predetermined application. Alternatively, part of at least one software module may be provided by the OS, and the remaining part may be provided by a predetermined application.

FIG. 5 is a diagram for explaining the display system 1000 according to an embodiment of the present disclosure. Referring to FIG. 5, the display system 1000 may be comprised of a display device 100 and a server 200.

At this time, the electronic device 100 may include a general-purpose processor, and the server 200 may include an artificial intelligence-specific processor. Alternatively, the electronic device 100 may include at least one application, and the server 200 may include an operating system. The server 200 is a component that is more integrated, dedicated, has a smaller delay, has better performance, or has more resources than the electronic device 100, and is required when creating, updating, or applying a recognition model. It can be a component that can process many calculations more quickly and effectively than the electronic device 100.

In this case, an interface for transmitting/receiving data between the electronic device 100 and the server 200 may be defined.

For example, an API (application program interface) may be defined that has learning data to be applied to a recognition model as an argument value (or a parameter value or transfer value). An API is a set of subroutines or functions that can be called for certain processing in one protocol (e.g., a protocol defined in the electronic device 100) of another protocol (e.g., a protocol defined in the server 200). can be defined. In other words, an environment in which operations of another protocol can be performed in one protocol can be provided through the API.

In the embodiment of FIG. 5, the display device 100 may query the server 200 and receive a response from the server 200. For example, the display device 100 may transmit a query including a fingerprint to the server 200 and receive a response including content information from the server 200. Content information may include at least one of the location of the current frame in the entire content, playback time, content name, content ID, performer appearing in the current frame, product included in the current frame, content genre, and series information.

As another example, the display device 100 may transmit a query including the current frame to the server 200 and receive a response including a result of object recognition included in the current frame from the server 200.

Both ACR and object recognition may be performed in the display device 100 itself, but ACR and object recognition may also be performed in the server 200. Hereinafter, the description will focus on an embodiment in which ACR and object recognition are performed in the server 200, but of course, the display device 100 may operate alone.

Figure 6 is a diagram for explaining the ACR process. In the embodiment of FIG. 6, the display device 100 may generate a fingerprint by periodically extracting features of the displayed screen. And the display device 100 may query the server 200 for content information corresponding to the generated fingerprint. The server 200 can perform ACR for real-time broadcasting and ACR for VOD, respectively.

A fingerprint is feature data extracted from video and audio signals included in a frame. Unlike text-based metadata, fingerprints can reflect the unique characteristics of the signal itself. For example, if the signal included in the frame is an audio signal, the fingerprint may be data expressing characteristics such as frequency and amplitude of the audio signal. Additionally, if the signal included in the frame is a video (or still image) signal, the fingerprint may be data expressing characteristics such as a motion vector and color.

In FIG. 6, the server 200 is shown to be composed of a plurality of devices, but all operations may be performed by a single server 200. Referring to FIG. 6, the server 200 includes a capture server 210, a live indexing server 220, a live data server 230, a metadata server 240, a VOD storage unit 250, and a VOD indexer 260. , VOD data server 270, and search server 280.

First, in the case of real-time broadcast ACR, the capture server 210 can extract each video frame from the broadcast signal. And the capture server 210 can generate a fingerprint by analyzing the extracted frame. In the case of real-time broadcasting, the capture server 210 may receive several seconds of video information before the display device 100. The capture server 210 may receive Electronic Program Guide (EPG) data including a channel and broadcast time from the metadata server 240. The capture server 210 can use the EPG data to determine the content of the currently received broadcast signal and the position of the current frame in the overall content.

The live indexing server 220 may distribute and store the fingerprint data and content information received from the capture server 210 across a plurality of live data servers 230. For example, the live indexing server 220 may transmit data to one of a plurality of live data servers 230 for each broadcast channel or content.

The search server 280 may search the live data server 230 for a fingerprint corresponding to the fingerprint included in the query in response to a query about real-time broadcasting. And, the search server 280 may transmit content information corresponding to the searched fingerprint to the display device 100.

Next, in the case of VOD ACR, the VOD to be provided as a service may be stored in the VOD storage unit 250. In the case of VOD, the difference from real-time broadcasting is that the server 200 may have information about all video frames in advance. The server 200 may generate a fingerprint for each VOD stored in the VOD storage unit 250. In addition, the VOD indexer 260 can match the metadata and fingerprint of the corresponding VOD received from the metadata server 270 and store them in a distributed manner across a plurality of VOD data servers 270. VOD metadata may include the title, genre, director, writer, characters, playback time, etc. of the program or content.

The search server 280 may search for a corresponding fingerprint in the VOD data server 270 in response to a query about VOD. And, the search server 280 may transmit content information corresponding to the searched fingerprint to the display device 100.

In this way, the display device 100 can request corresponding content information by transmitting a fingerprint to the server 200. The server 200 may search for a corresponding fingerprint from the live data server 230 or the VOD data server 270 depending on whether the requested content is real-time broadcasting or VOD. Additionally, the server 200 may transmit content information corresponding to the searched fingerprint to the display device 100.

Figure 7 is a diagram for explaining the object recognition process. The display device 100 can receive content information from the server 200 and know which frame of which content the current frame is. Additionally, the display device 100 may transmit the current image frame data to the object recognition server 290. The object recognition server 290 can recognize an object in a received video frame. In FIG. 7, the server 200 providing the ACR function and the object recognition server 290 are shown as separate devices, but ACR and object recognition may be performed in the same server device. Additionally, as described above, it may be implemented in an embodiment in which object recognition is performed in the display device 100.

A method for recognizing an object will be described in more detail with reference to FIGS. 8A and 8B. The display device 100 or the object recognition server 290 may recognize objects corresponding to products on the displayed screen. Hereinafter, for convenience of explanation, it will be described that the display device 100 performs an object recognition operation.

The display device 100 may recognize objects corresponding to people as well as products. However, since information on people currently appearing on the screen can be known from content information, the display device 100 can be implemented to recognize objects that mainly correspond to products on the displayed screen.

Figure 8a is a diagram for explaining a method of recognizing an object by extracting feature points. The display device 100 may extract feature points from the displayed screen and match them with pre-stored product images. According to the matching result, the display device 100 may determine which product the object corresponds to.

For example, the display device 100 may extract feature points that are invariant to size and rotation from an image. Mainly, the outer part of the object, letters or shapes, etc. can be extracted as feature points. As another example, the display device 100 may extract feature points that are invariant to environmental changes such as scale, lighting, and viewpoint from multiple image frames.

Figure 8b is a diagram for explaining a method of recognizing an object through learning. The display device 100 can recognize products displayed on the screen through learning using artificial intelligence. The display device 100 can learn criteria for distinguishing objects through supervised learning or unsupervised learning methods.

For example, the display device 100 may learn physical characteristics that distinguish images by comparing a plurality of image frames. The display device 100 can learn image analysis standards through the ratio of the foreground and background in the image frame, the size, location, and arrangement type of the object.

Additionally, the display device 100 may recognize objects displayed on the screen based on learned image analysis standards.

Figure 9 is a diagram for explaining object clustering. By the method shown in FIGS. 8A and 8B, the display device 100 can recognize an object corresponding to a product. The display device 100 may cluster recognized product 1 and product 2 with a person. Also, the display device 100 can store the clustered objects by matching them with each image frame.

The display device 100 may recognize objects corresponding to people on the displayed screen based on metadata of content acquired during the ACR process. Metadata may include information about people appearing in each frame. For example, the display device 100 may perform clustering focusing on the main character of the displayed content. This is because the products that users of the display device 100 are mainly interested in are products used by the main characters.

The upper diagram of FIG. 9 is a diagram showing that the display device 100 recognizes three people and two products. And, the lower diagram of FIG. 9 is a diagram showing the result of the display device 100 performing object clustering.

First, the display device 100 can use metadata to filter out the middle person among the three recognized people who are not the main person. Additionally, the display device 100 may determine whether there is a relationship between the recognized person and the product based on the location on the displayed screen. If determined to be related, the display device 100 may cluster the recognized people and products. Conversely, if it is determined to be irrelevant, the display device 100 may filter the product.

In FIG. 9, the display device 100 may determine that the two products recognized based on their positions on the screen are related to the people on the left and right, respectively. Since it is possible to know who the left and right people are from metadata, the display device 100 can cluster products and person objects so that they are represented by the person's name.

Through this, the display device 100 can organize recognized objects centered on information of interest to the user, such as the bag worn by the male protagonist and the bag worn by the female protagonist.

In the case of real-time broadcasting, the above-described object recognition and clustering processes are performed in real time. In the case of VOD, the object recognition and clustering process for each frame is completed in advance, and the clustered object information can be built into a database.

According to an embodiment of the present disclosure, the display device 100 may determine who the user currently using the display device 100 is. For example, the display device 100 may collect the user's biometric information. Additionally, the display device 100 may determine who the user is by comparing the collected biometric information with previously stored biometric information. Specifically, the display device 100 may recognize the user's fingerprint using a remote control or recognize the user's iris using a camera.

As another example, the display device 100 may capture a certain area in front of the display device 100, where the user is mainly located, with a camera. Additionally, the display device 100 may determine that the user included in the captured image is the current user of the display device.

The display device 100 may collect and store log information, whether relevant information is clicked, gender information, age information, genre preference information, etc., of each user. For example, the display device 100 may store information directly input by the user and information collected from the usage history of the display device 100. As another example, the display device 100 may communicate with IoT devices and store information collected by the IoT devices while tracking the user.

The display device 100 may select one object from among the clustered objects using the determined user information. Additionally, the display device 100 may provide services related to the selected object.

For example, the display device 100 may use the determined user's gender and age information to select objects clustered with characters of the same gender and similar age. If the determined user is a woman, the display device 100 may select the bag worn by the female protagonist in FIG. 9 . Additionally, the display device 100 can provide a shopping application execution service that allows purchasing the bag worn by the female protagonist.

When a single user is determined, it is sufficient to provide a service suitable for the user. However, when multiple users are determined, it must be decided which of the users to provide a more suitable service to. For example, if it is determined through the camera that two users are watching content played on the display device 100, the display device 100 determines which of the two users a more appropriate service should be provided to. Must decide.

The display device 100 can store a plurality of user information. Referring to FIG. 10 , the display device 100 may collect and store information about the first user 1010 and information about the second user 1020 . The display device 100 may determine a preference ranking for clustered objects recognized on the current screen. For example, the display device 100 may determine a preference ranking in order of sales volume among clustered products.

Additionally, the display device 100 may determine the preferences of a plurality of users for the object with the highest preference ranking. For example, the display device 100 may determine the preferences of the first user 1010 and the second user 1020 for the bag of the female protagonist determined to have the highest preference. Based on the stored plurality of user information, the display device 100 may determine that the first user 1010's preference for the female protagonist's bag is higher than the second user's 1020 preference. The display device 100 may determine the user of the display device 100 as the first user 1010 for the current screen (current frame). Additionally, the display device 100 may provide a service related to the female protagonist's bag to the determined first user.

In this way, even when multiple users use the display device 100, the display device 100 can determine a more suitable user to provide the service to. Additionally, the display device 100 can select one of a plurality of users each time the screen changes. That is, even while providing a suitable service to the first user 1010, if a screen containing a product preferred by the second user 1020 is displayed, the display device 100 may not provide a suitable service to the second user 1020. You can.

According to various embodiments of the present disclosure as described above, the display device 100 may determine the user of the display device 100 and provide a personalized ACR-based service to the determined user.

FIG. 11 is a flowchart illustrating a method of providing a service by the display device 100 according to an embodiment of the present disclosure. Referring to FIG. 11, the display device 100 can recognize content being played (S1110). For example, the display device 100 may recognize content through a server ACR method that transmits a fingerprint extracted from the characteristics of the currently displayed screen to the server 200 and inquires about content information on the current screen.

And the display device 100 may recognize at least one object on the currently displayed screen of content (S1120). For example, the display device 100 may recognize an object using a feature point extraction algorithm. As another example, the display device 100 may learn image analysis standards using artificial intelligence. And using the learned standard, the display device 100 can recognize an object on the displayed screen.

The display device 100 may determine the user currently using the display device 100 (S1130). The display device 100 may provide a personalized ACR-based service using the determined user preference information. For example, the display device 100 may determine who the user is by collecting the user's biometric information, such as a fingerprint. As another example, the display device 100 may extract a user from an image captured by a camera using a face recognition algorithm.

Based on the determined user information, the display device 100 may select one of the recognized objects (S1140). And the display device 100 can provide a service related to the selected object (S1150). The display device 100 can provide a personalized service by selecting a user's preferred object among a plurality of objects recognized on the displayed screen.

FIG. 12 is a flowchart illustrating a method of providing a service by the display device 100 according to another embodiment of the present disclosure. Referring to FIG. 12, the display device 100 can recognize the content being viewed (S1210). The display device 100 can determine what content is currently being played by matching the fingerprint generated from the currently displayed screen with the fingerprint stored in the server 200.

And, the display device 100 can recognize people or products on the screen (S1220). The display device 100 can distinguish objects included in the screen by extracting feature points from the currently playing content screen or through a learning method using artificial intelligence. Next, the display device 100 may cluster the people and products recognized within the screen (S1230). The display device 100 may determine the main person in the content using metadata and cluster products located close to the main person into products used by the main person.

The display device 100 may determine who the user currently using the display device 100 is (S1240). For example, the display device 100 may use biometric information such as fingerprint recognition, iris recognition, or voiceprint recognition. As another example, the display device 100 may recognize a user watching content using a camera.

When it is determined that there is only one user (S1250-Y), the display device 100 can provide a service suitable for the determined user. However, if there are multiple users (S1250-N), the display device 100 must select a user to provide the service (S1260). For example, taking into account information about people appearing on the screen and the gender, age, preferences, etc. of the determined plurality of users, the display device 100 may target a target for providing services.

When the target to provide the service is determined, the display device 100 may select a product according to the determined user's preference (S1270). The display device 100 may collect user profile information, preference information, etc. For example, the display device 100 may collect account information entered by the user, usage information of IoT devices around the display device 100, etc. Using the collected information, the display device 100 can select the user's most preferred product among the recognized products.

And the display device 100 can provide services related to the selected product (S1280).

The term “unit” used in the present disclosure includes a unit comprised of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrated part, a minimum unit that performs one or more functions, or a part thereof. For example, a module may be comprised of an application-specific integrated circuit (ASIC).

Various embodiments of this document may be implemented as software including instructions stored in a machine-readable storage media that can be read by a machine (e.g., a computer). The device may read instructions stored from the storage medium. A device capable of making a call and operating according to a called instruction may include an electronic device (e.g., electronic device A) according to the disclosed embodiments, and when the instruction is executed by a processor, the processor directly: Alternatively, the function corresponding to the instruction may be performed using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter. , It can be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium does not contain signals and is tangible. There is no distinction between semi-permanent or temporary storage.

According to one example, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed on a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or online through an application store (e.g. Play Store™). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or created temporarily in a storage medium such as the memory of a manufacturer's server, an application store server, or a relay server.

Each component (e.g., module or program) according to various embodiments may be composed of a single or multiple entities, and some of the above-described sub-components may be omitted or other sub-components may be used. It may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity and perform the same or similar functions performed by each corresponding component prior to integration. According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added. It can be.

100: display device 110: display unit
120: processor 130: communication department
140: input unit 150: camera
160: memory 170: video receiver
180: Image processing unit

Claims (20)

In the display device,
A display unit that displays a content playback screen;
Ministry of Communications; and
Recognizes the content being played,
Recognizes at least one object on the currently displayed screen,
Identify the user using the display device,
Selecting one of the at least one recognized object based on the identified user information,
A processor that provides services related to the selected object to the identified user,
The processor,
Recognize the first object corresponding to the product on the screen,
Recognize a second object corresponding to a person on the screen based on metadata of the recognized content,
If no relationship is identified between the product and the person, filter the first object,
Based on the identified user information, select one of the at least one object through which the first object has been filtered,
Generates a fingerprint by extracting features of the screen displayed on the display unit,
Controlling the communication unit to query the server for content information corresponding to the generated fingerprint,
A display device that recognizes the content using content information received from the server. delete delete According to paragraph 1,
The processor,
Based on the position on the displayed screen, identify whether there is a relationship between the person and the product,
A display device that clusters the first object and the second object if there is a relationship. According to paragraph 1,
The processor,
If there are two or more identified users, a display device that identifies one of the two or more users as a user using the display device every time the screen is changed. According to clause 5,
The processor,
A display device that determines a preference ranking for the at least one object, and identifies the user with the highest preference for the object determined as the highest priority among the two or more users as the user using the display device. According to paragraph 1,
It further includes an input unit that receives the user's biometric information,
The processor,
A display device that compares biometric information input through the input unit with pre-stored biometric information to identify a user using the display device. According to paragraph 1,
Further including a camera for taking images,
The processor,
A display device that identifies a user included in an image of a preset area located in front of the display device captured by the camera as a user using the display device. In a method of providing a service for a display device,
Recognizing the content being played;
Recognizing at least one object on the currently displayed screen of the content;
identifying a user using the display device;
selecting one of the at least one recognized object based on the identified user information; and
It includes providing a service related to the selected object to the identified user,
The step of recognizing at least one object includes:
Recognizing a first object corresponding to a product on the screen;
Recognizing a second object corresponding to a person on the screen based on metadata of the recognized content; and
If no relationship is identified between the product and the person, filtering the first object; Including,
The step of recognizing the content is,
Based on the identified user information, selecting one of the at least one object through which the first object has been filtered;
generating a fingerprint by extracting features of the displayed screen;
transmitting a query for content information corresponding to the generated fingerprint to a server; and
Recognizing the content using content information received from the server; Steps comprising; Service provision method including. delete delete According to clause 9,
The step of recognizing at least one object includes:
Identifying whether there is a relationship between the person and the product based on the position on the displayed screen; and
A service providing method comprising: clustering the first object and the second object if the relationship exists. According to clause 9,
The step of identifying the user is,
If there are two or more identified users, a service providing method that identifies one user among the two or more users as a user using the display device each time the screen is changed. ◈Claim 14 was abandoned upon payment of the setup registration fee.◈ According to clause 13,
The step of identifying the user is,
determining a preference ranking for the at least one object; and
Among the two or more users, identifying the user with the highest preference for the object determined as the highest priority as the user using the display device. ◈Claim 15 was abandoned upon payment of the setup registration fee.◈ According to clause 9,
The step of identifying the user is,
Receiving biometric information of the user; and
Comparing the input biometric information with pre-stored biometric information to identify a user using the display device. ◈Claim 16 was abandoned upon payment of the setup registration fee.◈ According to clause 9,
The step of identifying the user is,
Taking an image of a preset area located in front of the display device; and
A service providing method comprising: identifying a user included in the captured image as a user using the display device. In a display device using an artificial intelligence neural network model,
A display unit that displays a content playback screen;
Ministry of Communications; and
Recognizes the content being played,
Input the recognized content into an artificial intelligence neural network model to recognize at least one object on the currently displayed screen,
Identify the user using the display device,
A processor that selects one of the recognized at least one object based on information on the identified user and provides a service related to the selected object to the identified user,
The processor,
Recognize the first object corresponding to the product on the screen,
Recognize a second object corresponding to a person on the screen based on metadata of the recognized content,
If no relationship is identified between the product and the person, filter the first object,
Based on the identified user information, select one of the at least one object for which the first object has been filtered,
Generates a fingerprint by extracting features of the screen displayed on the display unit,
Controlling the communication unit to query the server for content information corresponding to the generated fingerprint,
A display device that recognizes the content using content information received from the server. delete delete ◈Claim 20 was abandoned upon payment of the setup registration fee.◈ According to clause 17,
The processor,
Based on the position on the displayed screen, identify whether there is a relationship between the person and the product,
A display device that clusters the first object and the second object if the relationship exists.

Images