KR20210015417A - System and method for providing service based on object detection - Google Patents

Abstract

The present invention provides a system for providing a service based on object recognition to provide the convenience to a user. The system comprises: a secretary terminal receiving a voice command of a user and outputting a response to the voice command; a server receiving the voice command from the secretary terminal, detecting food from video content that a user is watching according to the voice command, and searching food name of the food and product information related to food ingredients to output the food name and product information to the secretary terminal; and a display device receiving the food name and product information from the secretary terminal and displaying the food name and product information.

Description

Object recognition-based service provision system and method {SYSTEM AND METHOD FOR PROVIDING SERVICE BASED ON OBJECT DETECTION}

The present invention relates to a system and method for providing service based on object recognition, and more particularly, to a system and method for providing a product link service for each user.

Recently, speech recognition technology is developing at a rapid pace. Through such speech recognition technology, services such as providing a lot of information and contents to users in various fields such as broadcasting, movies, music, internet, and shopping are being implemented.

For example, using a voice recognition service, a user may receive or purchase a desired product through speech expressing a shopping intention.

However, since the existing voice recognition-based service has to perform shopping by searching for each product, it is inconvenient to users who want to collectively purchase related products. In addition, even though the products they prefer are different for each user, the utility of service use is deteriorating because products are uniformly recommended.

An object of the present invention is to provide a system and method for providing an object recognition-based service capable of recommending products for each user and providing convenience to users.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned can be clearly understood by those of ordinary skill in the technical field to which the present invention belongs from the following description. There will be.

In order to solve the above-described problems, the present invention provides a secretary terminal that receives a user's voice command and outputs a response to the voice command, and receives a voice command from the secretary terminal, and is viewed by the user according to the voice command. A server that detects a food object in the video content being processed, searches for product information related to the food name and food material of the food object, and outputs it to the secretary terminal, and a display device that receives food name and product information from the secretary terminal and displays it It provides an object recognition-based service providing system including.

Here, the server may search for product information based on the user's product purchase history or product search history.

In addition, product information may be classified into a plurality of items including information on the lowest price product, organic product, and retort product.

In addition, the voice command may include at least one of a food object detection command, a product purchase or shopping cart addition command included in the product information, a food recipe detection command, and a food ingredient detection command.

In addition, the server may perform at least one of a product purchase, a product shopping cart, a food recipe search, and a food material search according to a voice command, and output the execution result to the secretary terminal.

Also, the display device may receive an execution result according to the voice command from the secretary terminal and display it.

In addition, the server includes a learning model generation unit that generates a learning model by learning food image data, a food object detection unit that detects a food object from image content, and a food name of the food object by inputting the food object into the learning model. It may include a food name search unit.

Here, when a plurality of food names are searched, the food name search unit may output at least one food name to the secretary terminal according to a matching degree of the food object and the food name.

In addition, the server may store a food image including a food object according to a voice command, and output the food image to the secretary terminal during or after viewing the video content.

In addition, the server may include a database for storing product purchase history or product search history of the user and storing product information.

In addition, the present invention provides a step of uttering a voice command to a secretary terminal while a user is watching video content through a display device, a step of receiving the voice command from the secretary terminal and outputting the voice command to the server, and the server according to the voice command. Detecting a food object from the video content, searching for product information related to the food name and food material of the food object based on the user's product purchase history or product search history, and outputting it to the secretary terminal, and the display device is the secretary terminal Provides an object recognition-based service providing method comprising the steps of receiving food name and product information from and displaying them, and outputting a response to a voice command by a secretary terminal.

In addition, the object recognition-based service providing method of the present invention includes the steps of a user firing a command to purchase a product or add a shopping cart included in product information to a secretary terminal, and a command to purchase a product or add a shopping cart included in the product information by the secretary terminal. Receiving the input and outputting it to the server, the server performing a product purchase or adding a product shopping cart and outputting the execution result to the secretary terminal, and the display device receiving the execution result from the secretary terminal and displaying it. Can include.

According to the present invention, it is possible to collectively purchase products related to a specific food at a time, thereby providing convenience to a user, and analyzing a user's product preference tendency to recommend products for each user.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood by those of ordinary skill in the art from the following description. will be.

1 is a block diagram of an object recognition-based service providing system according to an embodiment of the present invention.
2 is a detailed block diagram of a secretary terminal according to an embodiment of the present invention.
3 is a detailed block diagram of a server according to an embodiment of the present invention.
4 is a detailed block diagram of the control unit of FIG. 3.
5 is a diagram for describing a method of detecting a food object in video content.
6 and 7 are diagrams illustrating an example of displaying food names and product information on a display device according to an exemplary embodiment of the present invention.
8 is a diagram illustrating an example in which food name, food recipe, and product information are displayed on a display device according to an embodiment of the present invention.
9 is a flowchart of an object recognition-based service providing method according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The detailed description to be disclosed hereinafter together with the accompanying drawings is intended to describe exemplary embodiments of the present invention, and is not intended to represent the only embodiments in which the present invention may be practiced. In the drawings, parts irrelevant to the description may be omitted in order to clearly describe the present invention, and the same reference numerals may be used for the same or similar components throughout the specification.

In an embodiment of the present invention, expressions such as "or" and "at least one" may represent one of words listed together, or a combination of two or more. For example, “A or B” and “at least one of A and B” may include only one of A or B, and may include both A and B.

1 is a block diagram of an object recognition-based service providing system according to an embodiment of the present invention.

Referring to FIG. 1, an object recognition-based service providing system according to an embodiment of the present invention may include a display device 100, a secretary terminal 200, and a server 300.

The display apparatus 100 may receive and display image content from the server 300 through the secretary terminal 200 and may provide a time-based service during image content display.

Such a display device 100 includes a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, and a micro electromechanical system (MEMS). mechanical systems) displays and electronic paper displays, and the like.

The assistant terminal 200 may be connected to the server 300 or the display device 100 at the same time or at a time interval through a network.

Here, the network includes a local area network (LAN), a wide area network (WAN), the Internet (WWW), a wired/wireless data communication network, a telephone network, a wired/wireless television communication network, and the like. Examples of wireless data networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, and ultrasound. Communication, visible light communication (VLC), LiFi, and the like, but are not limited thereto.

The secretary terminal 200 receives a user's voice command and outputs a response to the received voice command. For example, when the user utters the voice "Gi Genie", the secretary terminal 200 may recognize the call word "Gi Genie" and expose a command waiting UI according to the recognition result. Thereafter, when the user utters a voice “Tove on”, the secretary terminal 200 may recognize the voice command “Tove on” and request the server 300 to analyze the voice command. In addition, the secretary terminal 200 may receive an analysis of a voice command from the server 300 and control the display apparatus 100 to be powered on according to the analysis result.

For another example, when the user utters the voice "Who is Hong Gil-dong", the secretary terminal 200 recognizes the voice command "Who is Hong Gil-dong?", and requests the server 300 to analyze the voice command. have. And, the secretary terminal 200 receives the analysis of the voice command from the server 300, according to the analysis result, "Kil-dong Hong is OO" can utter, through the display device 100, "Kil-dong Hong is OO” can be displayed.

In this way, the secretary terminal 200 may provide a time-based service through the display device 100 connected to the secretary terminal 200 and may provide a voice-based service through its own voice signal input/output unit.

In addition, the assistant terminal 200 may include a set-top box, through which video content can be provided from the server 300, and the received video content is output to the display device 100 to be displayed by the user. The video content can be viewed through the device 100.

2 is a detailed block diagram of a secretary terminal according to an embodiment of the present invention.

Referring to FIG. 2, the secretary terminal 200 may include a communication unit 210, an output unit 220, an input unit 230, a memory 240, and a control unit 250.

The communication unit 210 may include a short-range communication module or a wired/wireless communication module. Here, the short range communication module is for short range communication, and includes Bluetooth™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and NFC ( Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless Universal Serial Bus (USB) technologies may be used to support short-range communication.

The output unit 220 is for generating an output related to visual, auditory or tactile sense, and may include at least one of a display unit, an audio output unit, a hap tip module, and a light output unit.

The sound output unit may output audio data stored in the memory 240. The sound output unit also outputs sound signals related to functions performed by the secretary terminal 200. The sound output unit may include a speaker and a buzzer.

The input unit 230 may include a microphone for inputting an audio signal. The microphone processes external sound signals into electrical voice data. The processed voice data may be used in various ways according to a function (or an application program being executed) being executed by the secretary terminal 200. Meanwhile, various noise removal algorithms may be implemented in the microphone to remove noise generated in the process of receiving an external sound signal.

The memory 240 stores data supporting various functions of the secretary terminal 200. The memory 240 may store an application program or application driven by the assistant terminal 200, data for the operation of the assistant terminal 200, and instructions.

The controller 250 controls an operation related to an application program stored in the memory 240 and, in general, an overall operation of the secretary terminal 200. Furthermore, in order to implement various embodiments described below on the secretary terminal 200 according to the present invention, the controller 250 may control by combining at least one of the above-described components.

The control unit 250 may further include a voice recognition module 255. The voice recognition module 255 drives a voice recognition engine to which a voice recognition algorithm is applied to recognize an external voice input through a microphone.

That is, the voice recognition module 255 converts the external voice input through the microphone into digital data, amplifies the converted digital data (Pre-emphasis), and then detects the start and end points of the digitally converted voice. do. Subsequently, the voice recognition module 255 extracts voice feature values for the voice between the detected start point and the end point to recognize a unique voice or tone.

Meanwhile, in the present embodiment, although the speech recognition module 255 is exemplified in the controller 250, it is not limited thereto, and it will be apparent to those skilled in the art that the voice recognition module 255 may be configured independently of the controller 250.

The server 300 may provide video content to the secretary terminal 200. In addition, the server 300 may receive a user's voice command from the secretary terminal 200 and output a response according to the voice command to the secretary terminal 200.

3 is a detailed block diagram of a server according to an embodiment of the present invention.

Referring to FIG. 3, a server 300 according to an embodiment of the present invention may include a communication unit 310, a control unit 320, and a database 330.

The communication unit 310 is a component for performing communication with the secretary terminal 200, and may include a communication module for supporting wired communication and a mobile communication module for supporting wireless communication. Here, the mobile communication module includes technical standards or communication methods for mobile communication (eg, Global System for Mobile Communication (GSM), Code Division Multi Access (CDMA)), Code Division Multi Access 2000 (CDMA2000), and EVDO ( Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term) Evolution-Advanced), etc.), transmits and receives radio signals with at least one of a base station and an external terminal on a mobile communication network.

The database 330 stores a user's product purchase history or product search history, and stores product information related to food ingredients. Here, the product purchase history may be a history of purchasing a product using the secretary terminal 200, and the product search history may be a history of searching for a product using the secretary terminal 200.

The controller 320 controls the overall operation of the server 300, detects a food object from the video content being viewed by the user according to the user's voice command input from the secretary terminal 200, and detects the food name of the food object and Search for product information related to food ingredients. In this case, the controller 320 may search for product information based on the user's product purchase history or product search history.

In this way, product information searched by the control unit 320 is output to the secretary terminal 200 through the communication unit 310.

FIG. 4 is a detailed block diagram of the controller of FIG. 3, and FIG. 5 is a diagram illustrating a method of detecting a food object from image content.

4, the controller 320 includes a learning model generation unit 321, a food object detection unit 322, a food name search unit 323, a product classification unit 324, and a product information search unit 325. Can include.

The learning model generation unit 321 generates a learning model (eg, a Darknet YOLO model) by learning food image learning data. Here, the food image learning data may be generated by image crawling on the Food 101 dataset and various portal sites such as google and naver.

Referring to FIG. 5, when a user makes a voice command (eg, what food is on TV?) to the secretary terminal 200 while a user is watching video content through the display device 100, the food object detection unit 322 is viewed, and the food object detection unit 322 is viewed. The food object 10a is detected from the video content 10 being processed.

The food name search unit 323 searches for a food name (eg, fried pork chop) of the food object 10a by inputting the food object 10a detected by the food object detection unit 322 into the learning model.

Meanwhile, when a plurality of food names are searched, the food name search unit 323 may output at least one food name to the secretary terminal 200 according to a matching degree of the food object and the food name. For example, when there are a plurality of visually similar foods and it is difficult to search for an accurate food name for a food object (e.g., stir-fried jeyuk, stir-fried sundae, and stir-fried octopus), the names of the top three foods with high matching degree are selected from the secretary terminal ( 200), and the user can select one of these food names by voice command.

6 and 7 are views showing an example in which food names and product information are displayed on a display device according to an embodiment of the present invention, and FIG. 8 is a diagram illustrating a food name, food recipe, and information on a display device according to an embodiment of the present invention. It is a diagram showing an example in which product information is displayed.

The product classification unit 324 may classify product information related to food ingredients into a plurality of items, including information on the lowest price product, organic product, and retort product.

The product information search unit 325 searches for product information related to the ingredients of the food searched by the food name search unit 323. Then, the product information searched by the product information search unit 325 is output to the secretary terminal 200 through the communication unit 310.

6 and 7, the display device 100 receives food name (eg, bag stew) and product information from the secretary terminal 200 and displays it. In addition, when the user utters a purchase command (batch purchase or individual purchase) for a product displayed on the display device 100 to the secretary terminal 200, the secretary terminal 200 outputs the product purchase command to the server 300 , The server 300 purchases a corresponding product and outputs the result of the execution to the secretary terminal 200. In addition, the display apparatus 100 receives the execution result according to the product purchase command from the secretary terminal 200 and displays it.

As described above, the object recognition-based service providing system according to an exemplary embodiment of the present invention can provide convenience to a user since products related to a specific food can be collectively purchased at a time.

In particular, the product information search unit 325 may search for product information based on the user's product purchase history or product search history. For example, referring to FIG. 6, in the case of a user who mainly purchases organic products through the secretary terminal 200, organic product information among products classified by the product classification unit 324 is searched and output, and FIG. 7 For reference, in the case of a user who mainly purchases a retort product, retort product information may be searched and output from among the products classified by the product classification unit 324. In the case of a user who mainly purchases the lowest price product, information about the lowest price product among the products classified by the product classification unit 324 may be searched and output.

More specifically, when it is determined that the user prefers organic products, the product information search unit 325 first searches and outputs organic products with trademarks frequently purchased by the user for all ingredients required for food cooking. I can.

In addition, when it is determined that the user prefers the lowest organic product, the product information search unit 325 may preferentially search and output the lowest organic product for all ingredients required for food cooking, and at this time, two products of the same price If this is the case, products with trademarks purchased more often can be searched and printed first.

In addition, when it is determined that the user prefers the retort product, the product information search unit 325 may preferentially search and output a retort product with a trademark that the user has frequently purchased.

In addition, when it is determined that the user prefers the lowest price retort product, the product information search unit 325 may preferentially search and output the lowest priced retort product. In this case, when there are two or more products of the same price, the trademark purchased more often You can preferentially search and print the products with is attached.

In addition, when it is determined that the user prefers the lowest price product, the product information search unit 325 may preferentially search and output the lowest price product for all ingredients required for food cooking. In this case, products with trademarks purchased more often can be searched and printed first.

As described above, the object recognition-based service providing system according to an embodiment of the present invention may analyze a user's product preference tendency to recommend a product for each user.

The database 330 stores a user's product purchase history or product search history, and stores product information related to food ingredients. In this way, the information stored in the database 330 is used by the control unit 320 to search food names and product information of the food object.

As an embodiment of the present invention, the user may utter a command to add a shopping cart (add collectively or individually) to the product displayed on the display device 100 to the secretary terminal 200. In this case, the secretary terminal 200 outputs a command to add a product shopping cart to the server 300, the server 300 performs the product shopping cart addition, and outputs the execution result to the secretary terminal 200. In addition, the display device 100 receives the execution result according to the command to add the product shopping cart from the secretary terminal 200 and displays the result. Thereafter, the purchase of products in the shopping cart may be performed according to the user's selection.

Referring to FIG. 8, as another embodiment of the present invention, a user may ignite a food recipe detection command and a food ingredient detection command corresponding to a food name displayed on the display device 100 to the secretary terminal 200. In this case, the secretary terminal 200 outputs a food recipe detection command and a food material detection command to the server 300, and the server 300 performs a food recipe search and food material search, and the result of the execution is sent to the secretary terminal ( 200). In addition, the display apparatus 100 receives the food recipe detection command and the execution result according to the food material detection command from the secretary terminal 200, that is, the recipe and ingredients of the food and displays them.

When a food object of interest is displayed while the user is watching video content through the display device 100, a storage command for the video may be issued to the secretary terminal 200. In this case, the secretary terminal 200 outputs an image storage command to the server 300, and the server 300 stores a food image including a food object according to the user's voice command, and while viewing or ending the video content Then, the stored food image may be output to the secretary terminal 200 according to the user's voice command. In addition, the secretary terminal 200 may output the received food image to the display device 100, and the display device 100 may display the food image.

Table 1 below is a table showing various situations in which a secretary terminal provides a service to a user.

Service contents Conversation example Product purchase User : What is that food on the TV?
Secretary's Terminal : This is Jeyuk-bokkeum.
User : Show product information related to Jeyuk-bokkeum.
Secretarial Terminal : This is a product related to Jeyuk-bokkeum. If you want to purchase in bulk, please tell us to purchase in bulk, if you want to purchase individually, please purchase *times.
User : I buy in bulk.
Secretary's Terminal : I bought pork-bokkeum related products in bulk. Product Add to Cart User : What is that food on the TV? Secretary's Terminal : This is Jeyuk-bokkeum.
User : Show product information related to Jeyuk-bokkeum.
Secretarial Terminal : This is a product related to Jeyuk-bokkeum.
User : Add to cart.
Secretary's Terminal : The product related to stir-fried pork is put in the shopping cart. Recipe and Ingredient Guide User : What is that food on the TV? Secretary's Terminal : This is Jeyuk-bokkeum.
User : Show me the recipe and ingredients for stir-fried pork.
Secretarial Terminal : Recipes and ingredients for pork stir-fry. Food video memo User : What is that food on the TV? Secretary's Terminal : This is Jeyuk-bokkeum.
User : Please take note of the video of Jeyuk-bokkeum.
Secretary terminal : Save the video of stir-fried pork.
User : Go to the video scene of Jeyuk-bokkeum.
Secretary's Terminal : This is a video of Jeyuk-bokkeum.

As shown in Table 1, various services may be provided to the user based on object recognition through a conversation between the user and the assistant terminal 200.

9 is a flowchart of an object recognition-based service providing method according to an embodiment of the present invention.

Hereinafter, a method for providing an object recognition-based service according to an embodiment of the present invention will be described with reference to FIGS. 1 to 9, but the same contents as in the object recognition-based service providing system according to the embodiment of the present invention will be omitted.

First, the display device 100 receives image content from the server 300 through the secretary terminal 200 and displays it.

Next, while the user is watching video content through the display device 100, the user speaks a voice command to the secretary terminal 200 (S10). Here, the voice command may be a command for detecting food objects in the video content currently being viewed.

Next, the secretary terminal 200 receives a voice command and outputs it to the server 300 (S20).

Next, the server 300 detects a food object from the video content according to a voice command (S31), and searches for product information related to the food name and food ingredient of the food object based on the user's product purchase history or product search history. (S32) This is output to the secretary terminal 200 (S33).

Next, the display apparatus 100 receives food name and product information from the secretary terminal 200 (S41) and displays them (S42).

Next, the secretary terminal 200 outputs a response to the voice command to the user (S50).

Next, the user ignites a command to purchase a product or add a shopping cart included in the product information to the secretary terminal 200 (S60).

Next, the secretary terminal 200 receives a command to purchase a product or add a shopping cart included in the product information and outputs it to the server 300 (S70).

Next, the server 300 performs product purchase or product shopping cart addition (S81), and outputs the execution result to the secretary terminal 200 (S82).

Next, the display apparatus 100 receives the execution result from the secretary terminal 200 (S91) and displays it (S92).

As described above, the object recognition-based service providing method according to an embodiment of the present invention can provide convenience to a user because it is possible to purchase products related to a specific food at once, and analyze the user's product preference tendency to customize the user. You can recommend products for each.

The embodiments of the present invention disclosed in the present specification and drawings are only provided for specific examples to easily explain the technical content of the present invention and to aid understanding of the present invention, and are not intended to limit the scope of the present invention. Therefore, the scope of the present invention should be construed that all changes or modified forms derived based on the technical idea of the present invention in addition to the embodiments disclosed herein are included in the scope of the present invention.

100: display device
200: secretary terminal
300: server

Claims (12)

A secretary terminal receiving a user's voice command and outputting a response to the voice command;
The secretary terminal receives the voice command from the secretary terminal, detects a food object in the video content being viewed by the user according to the voice command, searches for product information related to the food name and food material of the food object, and obtains it from the secretary terminal. Output to the server; And
Display device that receives the food name and product information from the secretary terminal and displays it
Object recognition-based service providing system comprising a.
The method of claim 1,
The server is
Searching for the product information based on the user's product purchase history or product search history
Object recognition-based service provision system.
The method of claim 2,
The above product information
Including the lowest price product, organic product and retort product information,
Object recognition-based service provision system.
The method of claim 1,
The voice command is
Including at least one of the food object detection command, a product purchase or shopping cart addition command included in the product information, the food recipe detection command, and a food ingredient detection command
Object recognition-based service provision system.
The method of claim 4,
The server is
Performing at least one of the product purchase, product shopping cart, food recipe search, and food material search according to the voice command, and outputting the execution result to the secretary terminal
Object recognition-based service provision system.
The method of claim 5,
The display device
Receives an execution result according to the voice command from the secretary terminal and displays it
Object recognition-based service provision system.
The method of claim 1,
The server is
A learning model generation unit that generates a learning model by learning food image data;
A food object detection unit that detects the food object in the video content; And
A food name search unit that searches for a food name of the food object by inputting the food object into the learning model
Object recognition-based service providing system comprising a.
The method of claim 7,
The food name search unit
When a plurality of food names are searched, outputting at least one food name to the secretary terminal according to the matching degree of the food object and food name
Object recognition-based service provision system.
The method of claim 1,
The server is
According to the voice command, storing a food image including the food object, and outputting the food image to the secretary terminal during or after viewing the video content
Object recognition-based service provision system.
The method of claim 2,
The server is
Database for storing the product purchase history or product search history of the user and storing the product information
Object recognition-based service providing system comprising a.
Uttering a voice command to the secretary terminal while the user is watching the video content through the display device;
Receiving, by the secretary terminal, the voice command and outputting it to a server;
The server detects a food object in the video content according to the voice command, and searches for product information related to the food name and food ingredient of the food object based on the user's product purchase history or product search history, and the secretary Outputting to the terminal;
Receiving, by the display device, the food name and product information from the secretary terminal and displaying the information; And
Outputting, by the secretary terminal, a response to the voice command
Object recognition-based service providing method comprising a.
The method of claim 11,
Igniting a command to purchase a product or add a shopping cart included in the product information to the secretary terminal by the user;
Receiving, by the secretary terminal, a command to purchase a product or add a shopping cart included in the product information and output it to the server;
Performing, by the server, purchasing the product or adding a product shopping cart and outputting a result of the execution to the secretary terminal; And
Receiving, by the display device, the execution result from the secretary terminal and displaying it
Object recognition-based service providing method further comprising.

Images