KR102302368B1 - Operating method of server and user device for providing linked service and user device thereof - Google Patents

Abstract

According to an embodiment, an operation of a user device that captures and analyzes any one frame included in the image content, transmits the analysis result to the server, and receives a linked service determined by the server based on the analysis result and provides it to the user Methods and user devices may be provided.

Description

A server for providing linked services, a method of operating a user device, and a user device thereof

Embodiments relate to a server for providing a linked service, a method of operating a user device, and a user device thereof.

With the development of information and communication technology, an IPTV (Internet Protocol Television) broadcasting system that provides real-time broadcasting contents or on-demand broadcasting contents using an IP (Internet Protocol) communication network as a broadcasting transmission medium is being commercialized. The IPTV broadcasting system can provide various information related to an object displayed on a TV screen through interactive communication with a user and various additional functions through the Internet. However, it is difficult to clearly process an image search request or provide a linked service at the time the user requests due to a decrease in the recognition rate due to rapid screen switching and changes in facial expressions of characters.

According to an embodiment, various linked services may be provided to viewers through image recognition and/or image analysis of a scene the user is watching.

According to an embodiment, the user device captures and analyzes any one frame of the image content that the user is watching, and transmits the analysis result to the server, so that the linked service determined by the server can be quickly provided to the user.

According to an embodiment, the user device may provide various linked services and/or customized services such as shopping information and travel information related to the frame at a time desired by the user according to the result of analyzing the frame and/or meta information. have.

According to an embodiment, a method of operating a user device includes: capturing any one frame included in image content; analyzing the frame; transmitting the analysis result to a server; receiving a linked service determined by the server based on the analysis result; and providing the linked service to the user.

The analyzing of the frame may include analyzing the frame by an edge computing technique.

The analyzing of the frame may include recognizing at least one object included in the frame.

Recognizing the at least one object may include recognizing the at least one object by analyzing the frame in real time using at least one of a graphics processing unit (GPU) and a neural process unit (NPU). .

Recognizing the at least one object by analyzing the frame in real time may include: classifying a feature of the at least one object included in the frame using a first neural network based on deep learning; and recognizing the at least one object by accumulating the classified feature as a histogram value over a plurality of frames included in the image content.

Recognizing the at least one object may include calculating a score of the feature accumulated as a histogram value over the plurality of frames.

The method of operating the user device may further include extracting meta information corresponding to the frame, and transmitting the analysis result to the server may include transmitting the analysis result and the meta information to the server. can

The extracting of the meta information may include extracting the meta information corresponding to the frame based on at least one of an electronic program guide (EPG) and VOD programming information.

The extracting of the meta information may include extracting the meta information corresponding to the frame using a pre-trained second neural network.

Receiving the link service may include receiving the link service determined by the server as the analysis result is applied to a search engine of the server.

The image content may include any one of TV broadcast content, movie content, and video on demand (VOD) content.

The analysis result may include at least one of a person, a scene, a product, a place, a character included in the frame, and a score of a feature classified from the frame.

According to an embodiment, the method of operating a server includes: receiving, from a user device, at least one of an analysis result of a frame captured by the user device and meta information corresponding to the captured frame; searching for a linked service corresponding to the frame based on at least one of the analysis result and the meta information; and transmitting the association service to the user device.

The searching for the associated service may include searching for the associated service using a third neural network trained to determine the associated service corresponding to a score included in the analysis result.

The method of operating the server may include: assigning a first weight to the analysis result; and assigning a second weight to the meta information.

The searching for the linked service may include searching for the linked service corresponding to the frame based on at least one of the first weighted analysis result and the second weighted meta information. .

According to one embodiment, the user device includes a sensor for capturing any one frame included in the image content; a processor that analyzes the frame; a communication interface for transmitting the analysis result to the server, and receiving a linked service determined by the server based on the analysis result; and a display that provides the linked service to a user.

The processor may recognize the at least one object by analyzing the frame in real time.

The processor classifies a feature of at least one object included in the frame using a first neural network based on deep learning, and accumulates the classified feature as a histogram value over a plurality of frames included in the image content. The at least one object may be recognized.

According to one side, various linked services may be provided to viewers through image recognition and/or image analysis of a scene the user is watching.

According to one side, the user device captures and analyzes any one frame of the image content that the user is watching, and transmits the analysis result to the server, so that the linked service determined by the server can be quickly provided to the user.

According to one side, the user device may provide various linked services and/or customized services such as shopping information and travel information related to the frame at a time desired by the user according to the result of analyzing the frame and/or meta information. .

1 is a flowchart illustrating a method of operating a user device according to an embodiment;
2 is a diagram for explaining an operation method between a user device and a server according to an embodiment;
3 to 4 are diagrams for explaining a method of recognizing at least one object by a user device according to an exemplary embodiment;
5 is a flowchart illustrating a method of operating a server according to an embodiment.
6 is a diagram for explaining a method for a third neural network to determine a connection service according to an embodiment;
Fig. 7 is a block diagram of a user device according to an embodiment;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Like reference numerals in each figure indicate like elements.

Various modifications may be made to the embodiments described below. It should be understood that the embodiments described below are not intended to limit the embodiments, and include all modifications, equivalents or substitutes thereto.

Terms used in the examples are only used to describe specific examples, and are not intended to limit the examples. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In the description of the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

1 is a flowchart illustrating a method of operating a user device according to an exemplary embodiment. Referring to FIG. 1 , a user device according to an embodiment captures any one frame included in image content ( 110 ). The image content may include, for example, TV broadcast content, movie content, and video on demand (VOD) content, but is not limited thereto. The user device is, for example, when a user watching video content clicks a hot key of a remote control unit (RCU) wirelessly connected to the user device, or is displayed on the screen of the user device. By touching a user interface (user interface) is triggered (triggered), it is possible to capture the frame of the scene the user is watching. Here, the frame may be an image frame. The user device may include, for example, a display device and a communication device such as a smart phone, a smart TV (Smart Television), and an IPTV (Internet Protocol Television), and may be a device capable of providing an IPTV service or an image providing service. The user device may capture any one frame using, for example, an image sensor, a camera sensor, or the like.

The user device analyzes the frame ( 120 ). The user device may analyze the frame by, for example, an edge computing technique. The 'edge computing technique' is a technique for performing computing functions at the edge of a network close to each user device. In other words, the edge computing technique is a computing method that supports the acceleration of data flow by processing data generated from various user devices in real time at the site or near the data center without sending it to a centralized data center such as the cloud. The user device may recognize at least one object included in the frame. For example, as shown in FIG. 2 , the user device may recognize at least one object by analyzing the frame in real time using at least one of a graphics processing unit (GPU) and a neural process unit (NPU). The user device may classify the feature of at least one object included in the frame using, for example, a first neural network based on deep learning. The user device may recognize at least one object by accumulating the classified features as histogram values over a plurality of frames included in the image content. A method for the user device to classify the characteristics of at least one object will be described in detail with reference to FIG. 3 below. Also, a method for the user device to recognize at least one object based on the accumulated histogram value will be described in detail with reference to FIG. 4 below.

According to an embodiment, the user device may extract meta information corresponding to the frame. The user device may extract meta information corresponding to the frame based on at least one of, for example, electronic program guide (EPG) and video on demand (VOD) programming information. Alternatively, the user device may extract meta information corresponding to the frame by using the second neural network learned in advance. Here, 'meta information' may be information given to content according to a certain rule in order to efficiently find and use the information that is being sought from among a large amount of information. Meta information is not data that actually exists or can be used, such as a video, sound, document, etc., but may correspond to data that provides information directly or indirectly related to actual data. The meta information may include, for example, title, provision time, rating, location and content of content, information about a creator (director or producer), rights conditions, conditions of use, history of use, and the like. Meta information may be used to represent data, or may serve as an index for quickly finding data. The server can easily find desired specific information by searching a search engine or the like using meta information.

The user device transmits the analysis result to the server (130). When the user device extracts meta information corresponding to the frame, the user device may transmit the analysis result and meta information to the server. The analysis result may include, for example, a person, a scene, a product, a place, a character included in the frame, a score of a feature classified from the frame, and the like.

The user device receives the association service determined by the server based on the analysis result ( 140 ). The user device may receive the linked service determined by the server as the analysis result is applied to the search engine of the server.

The user device provides the association service to the user ( 150 ).

2 is a diagram for explaining an operation method between a user device and a server according to an embodiment. 2 , a server 205 and a user device 210 are shown according to one embodiment. The user device 210 may include a CPU 213 and a GPU (or NPU) 216 .

For example, suppose that the user device 210 receives a real-time scene recognition request triggered by a user's touch input or click. The CPU 213 may capture any one frame of image content according to the real-time scene recognition request, and transmit the real-time scene recognition request together with the captured frame to the GPU (or NPU) 216 ( 220 ).

GPU (or NPU) 216 may recognize the frame in real time in response to the request of step 220 (230). The GPU (or NPU) 216 may analyze the frame to recognize at least one object included in the frame. In this case, according to an embodiment, the GPU (or NPU) 216 may extract meta information within the frame ( 240 ). The GPU (or NPU) 216 may recognize at least one object by analyzing the frame in real time, update the recognition result list, and perform scoring according to the analysis result ( 250 ). The GPU (or NPU) 216 may transmit the analysis result and/or score to the server. The server 205 may search for a link service corresponding to the frame based on the analysis result. The server 205 may transmit the search result, that is, the searched associated service, to the user device 210 or the CPU 213 ( 260 ). For example, the server 205 may deliver the search result after the score is higher than a preset reference value or after the reproduction of the video content or the broadcast content is finished. An operation of the GPU (or NPU) 216 will be described in detail with reference to FIGS. 3 to 4 below.

3 to 4 are diagrams for explaining a method of recognizing at least one object by a user device according to embodiments.

Referring to FIG. 3 , a process of classifying a feature of at least one object included in a frame by a user device using a first neural network according to an exemplary embodiment is illustrated. The first neural network may be, for example, a region with convolutional neural network (R-CNN). R-CNN can perform high-performance object detection by connecting a convolutional neural network (CNN) that performs image classification and a region proposal algorithm that suggests regions in which objects exist in an image.

When any one captured frame is input ( 310 ), the user device may obtain a plurality of candidate regions in which an object may exist, that is, region proposals from the frame ( 320 ).

The user device cuts each candidate region from the frame to make warped regions of the same size, and then extracts at least one feature of each region using CNN ( 330 ).

The user device may classify at least one characteristic of each area ( 340 ). The user device may calculate a classification result for at least one feature of each region and a bounding box regression. The user device may improve detection accuracy through bounding box regression.

Referring to FIG. 4 , a process of recognizing at least one object based on a classified feature by a user device according to an exemplary embodiment is illustrated. The user device according to an embodiment may recognize at least one object or a class of at least one object by accumulating the features classified through FIG. 3 as a histogram value over a plurality of frames included in the image content. The user device may calculate a score of the accumulated feature as a histogram value over a plurality of frames.

More specifically, the user equipment may perform clustering on the classified features to find codewords that are cluster centers. The user equipment may perform clustering by, for example, a k-means clustering technique.

The user device may generate a codebook composed of codewords. At this time, the codebook is not generated for each class of object, but is generated in common and may include codewords of all classes. The user device may represent each image as a histogram value of codewords as shown in FIG. 4 . Here, the 'histogram value' may correspond to counting the number of times each codeword appears in a frame.

The user device may recognize at least one object or a class of at least one object by a discriminative method of classifying the object by, for example, interpreting a histogram value for each class of the object as a probability. The discriminative method interprets a histogram value as a feature vector, puts it in a classifier such as a support vector machine (SVM), etc., and learns the class boundary to recognize an object or a class of an object.

5 is a flowchart illustrating a method of operating a server according to an embodiment. Referring to FIG. 5 , the server according to an embodiment receives, from the user device, at least one of an analysis result of a frame captured by the user device and meta information corresponding to the captured frame ( S510 ). The server according to an embodiment may utilize information accumulated by automatically capturing frames through continuous monitoring as well as frames captured at a specific point in time desired by the user.

The server searches for a link service corresponding to the frame based on at least one of the analysis result and meta information ( 520 ). Here, the 'linked service' may be a service having a relationship with the image content that the user is watching or an object exposed to any one frame included in the image content. The linked service may include, for example, a shopping service for products that appeared in a drama that the user is watching, a VOD recommendation for an actor who appeared in the drama, and a tour service to a famous tourist destination appearing in the corresponding drama. A method for the server to search for a linked service will be described in detail with reference to FIG. 6 below.

The server transmits the association service to the user device (530).

FIG. 6 is a diagram for describing a method for a third neural network to determine an associated service according to an embodiment. Referring to FIG. 6 , an operation of a third neural network according to an embodiment is illustrated.

The server may search for an association service using, for example, a third neural network trained in advance to determine an association service corresponding to a score included in an analysis result received from the user device. Alternatively, the server may search for a link service using, for example, a fourth neural network trained to determine a link service corresponding to the score and meta information included in the analysis result. In this case, for example, the server assigns a first weight (w _o ) to the analysis result (more specifically, a score) (x _o ), and gives a second weight (w ₁ ) to the meta information (x _{1 ).} can The server may search for a link service corresponding to the frame, for example, based on at least one of the first weighted analysis result and the second weighted meta information.

When the score exceeds a preset threshold, the server may execute a service result matched to the score.

7 is a block diagram of a user device according to an embodiment. Referring to FIG. 7 , a user device 700 according to an embodiment includes a sensor 710 , a processor 730 , a communication interface 750 , and a display 770 . User device 700 may further include memory 790 . Sensor 710 , processor 730 , communication interface 750 , display 770 , and memory 790 may communicate with each other via communication bus 705 .

The sensor 710 captures any one frame included in the image content. The sensor 710 may be, for example, any one of an image sensor, a vision sensor, and a camera sensor. The sensor 710 may, for example, capture any one frame of the image content that the user is watching according to a signal from a remote control connected to the user device.

The processor 730 analyzes the frame captured by the sensor 710 . The processor 730 recognizes at least one object by analyzing the frame in real time. The processor 730 classifies the features of at least one object included in the frame using, for example, a first neural network based on deep learning. The processor 730 recognizes at least one object by accumulating the classified features as histogram values over a plurality of frames included in the image content.

The communication interface 750 transmits the analysis result of the processor 730 to the server. The communication interface 750 receives the association service determined by the server based on the analysis result.

The display 770 provides a link service received by the communication interface 750 to the user.

In addition, the processor 730 may perform the at least one method described above with reference to FIGS. 1 to 4 or an algorithm corresponding to the at least one method. The processor 730 may be a hardware-implemented data processing device having a circuit having a physical structure for executing desired operations. For example, desired operations may include code or instructions included in a program. For example, a data processing device implemented as hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , an Application-Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA).

The processor 730 may execute a program and control the user device 700 . The program code executed by the processor 730 may be stored in the memory 790 .

The memory 790 may store an analysis result of the processor 730 .

The memory 790 may store various pieces of information generated in the process of the above-described processor 730 . In addition, the memory 790 may store various data and programs. Memory 790 may include volatile memory or non-volatile memory. The memory 790 may include a mass storage medium such as a hard disk to store various data.

The method according to an embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

As described above, although the present invention has been described with reference to limited embodiments and drawings, the present invention is not limited to the above embodiments, and various modifications and variations from these descriptions are provided by those skilled in the art to which the present invention pertains. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the following claims as well as the claims and equivalents.

700: user device
705: communication bus
710: sensor
730: processor
750: communication interface
770: display
790: memory

Claims (20)

capturing any one frame included in the image content;
analyzing the frame;
transmitting the analysis result to a server;
receiving a linked service determined by the server based on the analysis result; and
providing the linked service to the user
including,
Analyzing the frame
classifying a feature of at least one object included in the frame using a first neural network based on deep learning;
finding codewords by performing clustering on the classified features;
generating a codebook composed of the codewords for each frame, the codebook including codewords of all classes for each frame; and
Recognizing the at least one object by calculating an accumulated score with a histogram value for each class of the object included in the codebook
A method of operating a user device comprising a. According to claim 1,
Analyzing the frame
analyzing the frame by an edge computing technique
A method of operating a user device comprising a. According to claim 1,
Analyzing the frame
Recognizing at least one object included in the frame
A method of operating a user device comprising a. 4. The method of claim 3,
Recognizing the at least one object comprises:
Recognizing the at least one object by analyzing the frame in real time using at least one of a graphics processing unit (GPU) and a neural process unit (NPU)
A method of operating a user device comprising a. delete delete According to claim 1,
extracting meta information corresponding to the frame
further comprising,
The step of transmitting the analysis result to the server is
transmitting the analysis result and the meta information to the server
A method of operating a user device comprising a. 8. The method of claim 7,
The step of extracting the meta information is
extracting the meta information corresponding to the frame based on at least one of an electronic program guide (EPG) and VOD programming information;
A method of operating a user device comprising a. 8. The method of claim 7,
The step of extracting the meta information is
extracting the meta information corresponding to the frame using a second neural network learned in advance
A method of operating a user device comprising a. According to claim 1,
The step of receiving the linked service is
Receiving the linked service determined by the server as the analysis result is applied to the search engine of the server
A method of operating a user device comprising a. According to claim 1,
The video content is
A method of operating a user device, comprising any one of TV broadcast content, movie content, and video on demand (VOD) content. According to claim 1,
The analysis result is
A method of operating a user device, comprising at least one of a person, a scene, a product, a place, a character included in the frame, and a score of a feature classified from the frame. receiving, from a user device, at least one of an analysis result of a frame captured by the user device and meta information corresponding to the captured frame;
searching for a linked service corresponding to the frame based on at least one of the analysis result and the meta information; and
transmitting the association service to the user device;
including,
The analysis result is
The user device classifies features of at least one object included in the captured frame using a first neural network based on deep learning, performs clustering on the classified features to find codewords, and a plurality of frames Generates a codebook composed of the codewords for each frame, the codebook includes codewords of all classes for each frame, and calculates an accumulated score with a histogram value for each class of an object included in the codebook. A method of operating a server, comprising recognizing an object. 14. The method of claim 13,
The step of searching for the linked service is
searching for the linked service using a third neural network trained to determine a linked service corresponding to a score included in the analysis result;
Including, the method of operation of the server. 14. The method of claim 13,
assigning a first weight to the analysis result; and
assigning a second weight to the meta information
Further comprising at least one of, the method of operation of the server. 16. The method of claim 15,
The step of searching for the linked service is
searching for a linked service corresponding to the frame based on at least one of the first weighted analysis result and the second weighted meta information;
Including, the method of operation of the server. A computer program stored in a computer-readable recording medium in combination with hardware to execute the method of any one of claims 1 to 4 and 7 to 16. a sensor for capturing any one frame included in the image content;
a processor that analyzes the frame;
a communication interface for transmitting the analysis result to the server, and receiving a linked service determined by the server based on the analysis result; and
A display that provides the linked service to a user
including,
the processor
Classifying features of at least one object included in the frame using a first neural network based on deep learning, performing clustering on the classified features to find codewords, and selecting the codewords for each frame A codebook consisting of - the codebook includes codewords of all classes for each frame - is generated, and the at least one object is recognized by calculating an accumulated score with a histogram value for each class of the object included in the codebook. Device. 19. The method of claim 18,
the processor
A user device for recognizing at least one object by analyzing the frame in real time. delete

Images