KR101976366B1 - Unattended food store system based multimedia that can recommend customized menu - Google Patents

Abstract

The present invention relates to a multimedia-based unattended food store system capable of recommending a menu customized for a customer. More specifically, the multimedia-based unattended food store system capable of recommending a menu customized for a customer is configured to allow for an order inside the food store in real time using a kiosk or a customer terminal having a built-in dedicated application, and an order outside the food store in real time through the customer terminal having a built-in dedicated application, thereby selling food in an unmanned manner. The multimedia-based unattended food store system is configured to photograph a customer who stares menu information to select a menu, analyze gender information, age information, emotion information for each menu of the customer from image information of photographed images, generate, by a management server, recommended menu information regarding a menu selected the most and recommended menu information regarding a menu that receives attention the most, and provide information about a general menu that can be provided in the corresponding food store as well as a recommended menu customized for a customer.

Description

Unattended food store system based multimedia that can recommend customized menu}

The present invention relates to an unmanned food retailer system capable of recommending a customized menu based on multimedia, and more particularly, an order can be performed in real time using a customer terminal equipped with a kiosk or a dedicated app in a store, and outside the store. Provides the effect that unmanned foods can be sold by allowing customers to place orders in real time through a customer terminal equipped with a dedicated app, photographing customers who stare at the menu information for menu selection, and from the video information of the recorded video. After analyzing the gender information, age information, and emotional information of each menu, the recommendation menu information most selected by the management server and the recommendation menu information showing the most interest, and the menu that can be provided in the store are not only general menus but also customers Multimedia based customer customization to provide customized recommendation menu information Relate to unattended food retailers like system menus available.

Our society is undergoing a hyper-connectivity revolution based on the Internet of Things (IoT) where everything is connected to the Internet after the Industrial Revolution and the Information Revolution.

Things that are currently connected to the Internet are less than 1% of all things, but a variety of innovations and business opportunities can be created in the future.

As an example of an IoT service that can be found in the surrounding area, a connected car has been introduced to support automatic transmission of emergency rescue messages and unmanned autonomous driving services by connecting a vehicle to the Internet.

In addition, healthcare products support the health care service by measuring the body's heart rate and exercise.

In the smart home, it supports services such as remote control of home appliances or home CCTV.

Key technologies of IoT include sensing technology, wired and wireless communication and network infrastructure technology, and IoT service interface technology.

First, sensing technology can be used to obtain information from tangible objects and surroundings such as remote sensing, Synthetic Aperture Radar (SAR), radar, position, motion, and image sensors from traditional temperature / humidity / heat / gas / illuminance / ultrasonic sensors. Means detection in the sensor.

Physical sensors are evolving into smart sensors with standardized interfaces and information processing capabilities to improve application characteristics. In addition, physical sensors include a virtual sensing function that extracts specific information from data that has already been sensed. Implemented in the IoT service interface.

IoT sensing technology uses multi-sensor technology that is one level higher than standard independent and individual sensors, enabling more intelligent and higher-level information to be extracted.

In terms of wired and wireless communication and network infrastructure technologies, IoT wired and wireless communication and network devices include conventional wireless personal area networks (WPAN), Wi-Fi, 3G / 4G / LTE, Bluetooth, Ethernet, It refers to all wired and wireless networks that can connect humans, objects, and services such as broadband convergence network (BcN), satellite communication, microware, serial communication, and power line communication (PLC).

The IoT service interface plays a role of integrating the three major components of the IoT (human / object / service) with an application service performing a specific function.

More specifically, the IoT service interface is not a concept of a network interface, but senses, processes / extracts / processes, stores, judges, recognizes, recognizes, secures / privacy, authenticates / authorizes, discovers, object formalizes, and ontos information. It acts as an interface (storage, processing, transformation, etc.) to provide services such as luggage-based semantics, open sensor APIs, virtualization, positioning, processor management, open platform technology, middleware technology, data mining technology, web service technology, and social networks. Perform.

In particular, the wireless calling bell used in restaurants, hospitals, and factories plays a role of calling or transmitting information to convey a user's requirements or whether a specific event has occurred. have.

The user pushes the radio call bell, and the call signal is transmitted to the central control unit to identify the radio call bell and provide an environment for following up on the radio call bell.

On the other hand, in these days, as individualism intensifies and aging has led to a growing number of single people, most single people who use restaurants have their own private space where their private lives are guaranteed, rather than being joined by strangers. I want to be able to take food.

Although restaurants are seeking changes to reflect this food culture, they are not being promoted due to various economic problems.

Restaurant managers have a lot of difficulties in running a restaurant because of the labor costs of employees who work in ordering, delivering, or serving customers' needs, excluding kitchen staff. do.

Existing restaurants are generally divided into two types.

First, the restaurant is a general restaurant (full service) that provides services to all the needs of customers.

The employee takes the order from the customer and delivers it to the kitchen, provides the customer with the ordered food and drink, arranges the dishes after the customer finishes the meal, and takes the bill at the cash register or at the table to receive the payment.

The full service may be attractive to customers because the faithful service is provided to the customer, but the price may be increased and the operation of the restaurant may be inefficient due to the price of the provided service.

Second, it is a self service restaurant.

In self-service restaurants, customers take their own meals from the kitchen and eat them.

That is, the customer orders food at the order table, and after payment, receives a food preparation completion notice from the kitchen, receives the ordered food and drink, and eats.

In addition, after the meal, the customer carries the food bowl, etc. to the dishwasher.

These self-service restaurants can be efficient in terms of operations, but it is very difficult to take food in a private space where their own privacy is guaranteed.

Therefore, in order to meet the trend and demands of the times as described above, not only the interior of the restaurant but also the outside space of the restaurant can be conveniently ordered and settled automatically so that the ordered menu can be immediately ordered into the kitchen, thereby reducing the labor costs of the restaurant. Not only significantly reduce the number of customers, but also reduce the waiting time for customers to eat food. In addition to general menus, multimedia-based customized menu recommendations that can recommend the most preferred or interested menus in the restaurant. The need for this unmanned grocery store system is needed.

(Previous Document 1) Republic of Korea Patent Publication No. 10-1999-0058075

Therefore, the present invention is to solve the above conventional problems,

The first object of the present invention can be performed in real time using a customer terminal equipped with a kiosk or a dedicated app in the store, the order can be performed in real time through a customer terminal equipped with a dedicated app outside the store. It is possible to sell unattended food without employees.

The second object of the present invention is to shoot a customer staring at the menu information for menu selection using a customer terminal equipped with a kiosk or a dedicated app, and obtains the video information of the captured image from the management server, the gender information of the customer, After analyzing the age information and emotion information by menu, we generate the most selected recommendation menu information and the recommendation menu information that showed the most interest, and provide not only general menus but also customized customized menu information at the store. It is to make it possible.

In order to solve the problem of the present invention, the unmanned food store system capable of recommending a multimedia-based customized menu according to an embodiment of the present invention,

It is installed in the store, receives the customer's unique information from the customer, receives the menu information that can be provided in the store from the management server and provides it to the customer, performs the payment and receipt printing for the menu selected by the customer, and the payment is completed. A kiosk 100 for transmitting the information and the customer specific information to the management server;

Dedicated app is installed, the dedicated app receives the menu information that can be provided in the store from the management server to provide to the user, perform the payment for the menu selected by the user, and complete the menu information and the unique information of the store A customer terminal 200 for transmitting to the management server;

When the payment is received from the kiosk 100 menu information and customer specific information is transmitted to the kitchen terminal 400 linked to the kiosk to send the cooking request information for the cooking request of the menu, cooking completion information for the cooking request When receiving from the kitchen terminal 400 transmits a cooking completion notification message to the customer terminal corresponding to the customer unique information,

When the payment is received from the customer terminal 200 menu information and the unique information of the store, and transmits the cooking request information for the cooking request of the menu to the kitchen terminal 400 linked to the store, for the cooking request A management server 300 that transmits a cooking completion notification message to a customer terminal that transmits menu information on which payment is completed and unique information of a store when the cooking completion information is received from the kitchen terminal 400;

And a kitchen terminal 400 receiving cooking request information from the management server 300 and transmitting cooking completion information for the cooking request to the management server 300 when the cooking is completed.

According to the inventor of the present invention, the unmanned food retailer system capable of recommending a customized menu based on a multimedia device can perform an order in real time using a customer terminal equipped with a kiosk or a dedicated app in a store, and a dedicated app is installed outside a store. By allowing customers to place orders in real time through a customer terminal, it is possible to sell unattended food without employees, providing the effect of operating an unmanned food store that can save labor costs, and providing order numbers, payment information, and food pickup as text messages. It provides an effect that can save the user waiting time.

In addition, by using a customer terminal equipped with a kiosk or a dedicated app, the customer stares at the menu information to select a menu, and obtains the video information of the captured image from the management server, and according to the gender information, age information, and menu of the customer. After analyzing the emotion information, the most selected recommendation menu information and the most interested recommendation menu information is generated, and the menu that can be provided in the store can be provided not only the general menu but also customized customized menu information at the same time. It will provide an effect that can provide optimal menu suggestions for each gender and age of customers.

1 is an overall configuration diagram of an unmanned food store system capable of recommending a multimedia-based customized menu according to a first embodiment of the present invention.
2 is a block diagram of a kiosk 100 of an unmanned food store system capable of recommending a customized menu based on multimedia according to the first embodiment of the present invention.
3 is a block diagram of a customer terminal 200 of an unmanned food store system capable of recommending a customized menu based on multimedia according to the first embodiment of the present invention.
4 is a block diagram of a management server 300 of an unmanned food store system capable of recommending a customized menu based on multimedia according to the first embodiment of the present invention.
5 is a block diagram of a kiosk 100, a customer terminal 200, and a management server 300 of an unmanned food store system capable of recommending a customized menu based on multimedia according to another embodiment of the present invention.

The following merely illustrates the principles of the invention. Therefore, those skilled in the art, although not explicitly described or illustrated herein, can embody the principles of the present invention and invent various devices that fall within the spirit and scope of the present invention.

In addition, all conditional terms and embodiments listed herein are in principle clearly intended to be understood only for the purpose of understanding the concept of the invention and are not to be limited to the specifically listed embodiments and states. do.

In describing the present invention, terms such as first and second may be used to describe various components, but the components may not be limited by the terms.

For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

When a component is referred to as being connected or connected to another component, it may be understood that the component may be directly connected to or connected to the other component, but there may be other components in between. .

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention, and singular forms may include plural forms unless the context clearly indicates otherwise.

In this specification, the terms including or including are intended to designate that there exists a feature, a number, a step, an operation, a component, a part, or a combination thereof described in the specification, and one or more other features or numbers, It can be understood that it does not exclude in advance the possibility of the presence or addition of steps, actions, components, parts or combinations thereof.

<First Embodiment>

According to the first embodiment of the present invention, an unmanned food store system capable of recommending a customized menu based on multimedia,

It is installed in the store, receives the customer's unique information from the customer, receives the menu information that can be provided in the store from the management server and provides it to the customer, performs the payment and receipt printing for the menu selected by the customer, and the payment is completed. A kiosk 100 for transmitting the information and the customer specific information to the management server;

Dedicated app is installed, the dedicated app receives the menu information that can be provided in the store from the management server to provide to the user, perform the payment for the menu selected by the user, and complete the menu information and the unique information of the store A customer terminal 200 for transmitting to the management server;

When the payment is received from the kiosk 100 menu information and customer specific information is transmitted to the kitchen terminal 400 linked to the kiosk to send the cooking request information for the cooking request of the menu, cooking completion information for the cooking request When receiving from the kitchen terminal 400 transmits a cooking completion notification message to the customer terminal corresponding to the customer unique information,

When the payment is received from the customer terminal 200 menu information and the unique information of the store, and transmits the cooking request information for the cooking request of the menu to the kitchen terminal 400 linked to the store, for the cooking request A management server 300 that transmits a cooking completion notification message to a customer terminal that transmits menu information on which payment is completed and unique information of a store when the cooking completion information is received from the kitchen terminal 400;

And a kitchen terminal 400 for receiving cooking request information from the management server 300 and transmitting cooking completion information for the cooking request to the management server 300 when the cooking is completed.

In addition, the kiosk 100,

It is characterized by photographing the customer gazing at the menu information for menu selection, extracting the gender information, age information, and emotion information for each menu from the image information of the photographed image and providing the same to the management server 300.

In addition, the dedicated app mounted on the customer terminal 200,

When a user enters a store, the kiosk 100 installed in the store and the customer terminal 200 are connected to a local area network (Bluetooth, Wi-Fi, etc.), and then store unique information (eg, store unique ID) from the kiosk 100. ) Receive and provide to the management server (300),

It is characterized by photographing the customer gazing at the menu information for menu selection, extracting the gender information, age information, and emotion information for each menu from the image information of the photographed image and providing the same to the management server 300.

At this time, the extraction of emotion information for each menu,

From the image information of the photographed image, using the gaze direction information of the customer's eyes staring at the menu information for menu selection and menu information corresponding to the gaze direction, the interest menu information showing the customer's interest is extracted,

When extracting the interest menu information, the emotion information of the customer showing interest is extracted by extracting the emotion information of the customer from the face image of the customer.

In addition, the menu information provided by the management server 300 to the kiosk 100 and the customer terminal 200 includes general menu information and recommended menu information,

The general menu information is information on a menu that can be generally provided in the store,

The recommendation menu information is characterized in that the information on the personalized recommendation menu according to the gender and age of the customer who is concerned about the current menu selection of the general menu information.

At this time, the recommended menu information,

The information that shows the ranking of the menu most selected by the current gender and age group of the current customer staring at the menu information for menu selection,

It is characterized in that the information indicating the menu that showed the most interest to the current customer gender and age group staring at the menu information for menu selection.

In addition, the management server 300,

Using the selection menu information provided by the kiosk 100 and the customer terminal 200, the gender information of the customer and the age information of the customer, generate the recommended menu information selected by the gender and the age group of the customer most,

By using the gender information of the customer provided by the kiosk 100 and the customer terminal 200, the age information of the customer and the emotional information for each menu, characterized in that to generate the recommended menu information that the customer's gender and age group showed the most interest do.

In addition, the dedicated app mounted on the customer terminal 200,

If the user is outside the store, receives the store unique information from the user and provides to the management server 300,

It is characterized by photographing the customer staring at the menu information for menu selection, extracting the gender information, age information, and emotional information for each menu from the image information of the photographed image and providing the same to the management server 300.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the present invention unmanned food retailer system capable of recommending a customized menu based on multimedia.

1 is an overall configuration diagram of an unmanned food store system capable of recommending a customized menu based on multimedia according to a first embodiment of the present invention.

As shown in FIG. 1, the unmanned food store system capable of recommending a customized menu based on multimedia according to the first embodiment of the present invention includes a plurality of kiosks 100 installed in a store and a plurality of customer terminals 200 equipped with dedicated apps. ), Including the management server 300, the kitchen terminal 400.

Specifically, the kiosk 100 is installed in the store, receives the customer unique information (for example, mobile phone number) from the customer, and the menu (general menu, recommended menu) information that can be provided in the store from the management server It receives and provides to the customer, performs payment and receipt printing for the menu selected by the customer, and transmits the completed menu information and customer's unique information (eg, a mobile phone number) to the management server. A plurality of kiosks 100 may be installed in the store.

2 is a block diagram of a kiosk 100 configuration of an unmanned food store system capable of recommending a customized menu based on multimedia according to the first embodiment of the present invention.

That is, as shown in Figure 2, the kiosk 100,

A customer specific information input unit 110 for receiving customer specific information (eg, a mobile number) from a customer who wants to place an order;

When obtaining the customer specific information through the customer specific information input unit 110, request the menu information that can be provided in the store to the management server, the menu information output to receive the menu information available from the management server and output it to the screen Unit 120;

A payment progress unit 130 for providing request information for a payment request for a menu selected by a customer to an external payment institution and performing a receipt output when the payment is approved;

When the payment is completed, the menu information and the customer-specific information transmission unit 140 for transmitting the payment complete menu information and customer specific information to the management server; and configured.

Specifically, the customer specific information input unit 110 displays an input page for receiving customer specific information, for example, a mobile phone number, from the customer on the screen, and acquires the customer unique information (cell phone number) input on the displayed screen. Will be.

When the menu information output unit 120 obtains customer specific information through the customer specific information input unit, the menu information output unit 120 requests menu (general menu, recommendation menu) information that can be provided in the corresponding store to the management server, and provides the information from the management server. The menu (general menu, recommended menu) information is received and displayed on the screen.

The payment process unit 130 provides the payment request information on the menu selected by the customer to the external payment institution 500, and upon receipt of payment, outputs a receipt.

For example, the payment request information for one Gorgonzola pizza and one carbonara is provided to an external payment institution, and a receipt is output when the payment is approved.

When the payment is completed, the menu information and customer specific information transmitting unit 140 transmits the completed payment menu information and customer specific information to the management server.

For example, '# Gorzonzola pizza, ID: 010-1234-5678 #' information is transmitted to the management server.

The customer terminal 200 is equipped with a dedicated app, the dedicated app receives the menu information that can be provided in the store from the management server to provide to the user, perform the payment for the menu selected by the user, the payment is completed menu Information and the unique information of the store is to be transmitted to the management server.

3 is a block diagram illustrating a configuration of a customer terminal 200 of an unmanned food store system capable of recommending a customized menu based on multimedia according to the first embodiment of the present invention.

That is, as shown in Figure 3, the dedicated app mounted on the customer terminal 200,

When the user enters the store, the kiosk 100 installed in the store and the customer terminal 200 is connected to the local area network, and receives the store-specific information from the kiosk 100, and transmits the store-specific information to the management server. At the same time, it requests menu (general menu, recommendation menu) information that can be provided in the corresponding store, and receives menu information (general menu, recommendation menu) information that can be provided from the management server and outputs it to the screen for the customer terminal. 210);

Providing payment request information for the menu selected by the customer (user) to an external payment institution, and when the payment is approved, the customer progress payment unit 220 for generating and storing a mobile receipt;

When the payment is completed, the menu information and the store-specific information transmission unit 230 for transmitting the payment complete menu information and the unique information of the store to the management server; and configured.

Specifically, the user terminal menu information output unit 210 when the user enters the store, after the kiosk 100 and the customer terminal 200 installed in the store is connected to the local area network (Bluetooth, WiFi, etc.) Receives store specific information (eg, store unique ID) from the kiosk 100.

For example, when a customer visits A Pizza Shop ** branch, a customer terminal equipped with a dedicated app and a kiosk (Kiosk installed at A Pizza Shop ** branch) will perform short-range communication from the kiosk to A Pizza Shop ** The customer's terminal equipped with a dedicated app will receive 'A pizza_12', the store's unique information.

Subsequently, while providing the received store-specific information to the management server 300 and requests the menu (general menu, recommended menu) information that can be provided in the store to the management server, the menu available from the management server (general menu, recommended menu) The information is received and displayed on the screen.

The payment terminal 220 for the customer terminal provides the payment request information for the menu selected by the customer to the external payment institution 500, and upon approval of the payment, generates and stores a mobile receipt.

For example, when ordering one Gorgonzola pizza and one carbonara, the payment request information for the corresponding amount is provided to an external payment institution, and upon receipt of payment, a mobile receipt is generated and output to the screen and stored.

When the payment is completed, the menu information and store specific information transmission unit 230 transmits the completed payment menu information and customer specific information to the management server.

For example, '# Gorzonzola pizza, ID: 010-1234-5678 #' information is transmitted to the management server.

The management server 300 provides the kiosk 100 with menu information that can be provided in the store in which the kiosk 100 is installed, or the store corresponding to the store unique information provided by the customer terminal 200 to the customer terminal 200. Providing menu information that can be provided in the, and receives the menu information and customer specific information is completed payment from the kiosk 100 or the customer terminal 200, the kitchen terminal 400 or the customer terminal 200 linked to the kiosk 100. To perform the cooking request of the menu to the kitchen terminal 400 linked to the store corresponding to the unique information provided by the store, and when the cooking is completed, and sends a cooking completion notification message to the customer terminal corresponding to the customer unique information.

More specifically,

First, the management server 300 provides the kiosk 100 with menu information that can be provided in the store in which the kiosk 100 is installed, or corresponds to the store specific information provided by the corresponding customer terminal 200 to the customer terminal 200. Provide menu information available in the store.

Subsequently, upon receipt of the menu information and customer specific information from the kiosk 100, payment is sent to the kitchen terminal 400 linked to the kiosk, and the cooking request information for the cooking request of the corresponding menu, and the cooking for the cooking request. When the completion information is received from the kitchen terminal 400, the cooking completion notification message is transmitted to the customer terminal corresponding to the customer unique information.

In addition, when receiving the payment complete menu information and customer unique information from the customer terminal 200, the cooking request of the menu to the kitchen terminal 400 linked to the store corresponding to the store unique information provided by the customer terminal 200. When the cooking request information is transmitted, and the cooking completion information for the cooking request is received from the kitchen terminal 400, the cooking completion notification message is transmitted to the customer terminal corresponding to the customer's unique information.

4 is a block diagram illustrating a management server 300 of an unmanned food store system capable of recommending a customized menu based on multimedia according to the first embodiment of the present invention.

That is, as shown in Figure 4, the management server 300,

Menu (general menu, recommended menu) information that the kiosk 100 or the customer terminal 200 can provide in the store (store corresponding to the store unique information provided by the store or the customer terminal 200 is installed kiosk 100) information In the case of requesting, the menu (DB, general menu, recommended menu) that can be provided to the store is extracted from the menu information by store DB (320), and the extracted menu (general menu, recommended menu) information is stored in the corresponding kiosk or customer terminal. Shop-specific menu information provider 310 to provide to the 200;

Shop-specific menu information DB (320) for storing shop-specific menu (general menu, recommended menu) information;

Receiving the menu information and customer specific information is completed payment from the kiosk 100 or the customer terminal 200 corresponding to the store specific information provided by the kitchen terminal 400 or the customer terminal 200 linked to the kiosk 100. A cooking request information transmitter 330 for transmitting cooking request information for a cooking request of a corresponding menu to a kitchen terminal 400 linked to a store;

When the cooking completion information for the cooking request is received from the kitchen terminal 400, the cooking completion for transmitting a cooking completion notification message to the customer terminal corresponding to the customer specific information provided by the kiosk 100 or the customer terminal 200. Notification message transmission unit 340; is configured to include.

Specifically, the store-specific menu information providing unit 310 is a store corresponding to the store (Kiosk 100 is installed, store-specific information provided by the customer terminal 200, the store from the kiosk 100 or the customer terminal 200). In the case of requesting menu (general menu, recommendation menu) information that can be provided from the store-specific menu information DB 320, available menu (general menu, recommendation menu) information matching the corresponding store is extracted.

For example, when the menu request information is obtained from the kiosk 100 or the customer terminal 200, a menu that can be provided in the store, for example, 'A pizza specialty store ** branch' from the menu information DB 320 for each store. (General menu, recommended menu) Information will be extracted. Thereafter, the extracted menu (general menu, recommendation menu) information is provided to the corresponding kiosk or customer terminal.

The cooking request information transmitting unit 330 receives the menu information and the customer unique information from the kiosk 100 or the customer terminal 200, the kitchen terminal 400 or the customer terminal 200 linked to the kiosk 100. ) Sends the cooking request information for the cooking request of the menu to the kitchen terminal 400 linked to the store corresponding to the unique information provided by the store.

For example, if the menu information and customer-specific information '# Gorzonzola pizza, ID: 010-1234-5678 #' from the kiosk 100 or the customer terminal 200 is received, the corresponding store (kiosk ( 100) is installed, the store corresponding to the store-specific information provided by the customer terminal 200) transmits the cooking request information for the cooking request of the menu to the kitchen terminal 400 of the 'A pizza specialty store ** branch' Done. Thereafter, in the kitchen where the kitchen terminal is installed, the gorgonzola pizza, which is the order menu, is cooked.

When the cooking completion notification message transmission unit 340 receives the cooking completion information for the cooking request from the kitchen terminal 400, the customer terminal corresponding to the customer unique information provided by the kiosk 100 or the customer terminal 200. The cooking completion notification message will be sent to.

For example, when acquiring the cooking completion information from the kitchen terminal 400, the cooking completion notification message 'custom gorgonzola one customer ordered with the customer terminal' 010-1234-5678 'corresponding to the customer's unique information is cooked You are finished, please receive food. 'Cooking completion notification message is displayed on the screen of the customer terminal.

On the other hand, according to an additional aspect, the dedicated app mounted on the kiosk 100 and the customer terminal 200,

It is characterized by photographing the customer gazing at the menu information for menu selection, extracting the gender information, age information, and emotion information for each menu from the image information of the photographed image and providing the same to the management server 300.

In order to extract the gender information and age information of the customer from the image information, the dedicated app mounted on the kiosk 100 and the customer terminal 200, as shown in Figure 5,

Image capturing units 150 and 240 for capturing customers staring at menu information for menu selection;

Customer information analysis unit (160, 250) for extracting the gender information, age information, emotional information of each menu from the image information of the image taken by the image taking unit to provide to the management server 300; Will be constructed.

In detail, the image capturing unit 150 and 240 photographs a customer who gazes at the menu information to select a menu.

In this case, the customer information analyzer 160 and 250 extracts gender information, age information, and emotion information for each menu from the image information of the image photographed by the image photographing unit.

In order to provide the same function as the customer information analysis unit 160,250,

Gender recognition module (161, 251) for extracting the gender information of the customer by using the reference image data information by gender and age group stored in the information DB (166, 256) and the captured image information;

Age recognition module (162, 252) for extracting the customer's age information by using the reference image data information by gender and age group stored in the information DB (166, 256) and the captured image information,

A gaze direction recognition module 163 and 253 for recognizing a gaze direction of a customer's pupil gazing at the menu information for selecting a menu from the image information;

Interest menu information generation module (164, 254) for generating menu information corresponding to the gaze direction by using the recognized gaze direction and menu information for each gaze direction stored in the information DB (166, 256). ),

When extracting the interest menu information, analyzing emotion information of the customer by using the reference image data information by gender and age group stored in the information DB (166, 256) and the captured image information to generate emotion information for each menu showing the customer's interest. Emotional information extraction module for each menu (165, 255),

And information DBs 166 and 256 for storing reference image data information by gender and age group and menu information by gaze direction.

The gender recognition modules 161 and 251 and the age recognition modules 162 and 252 are components for extracting gender information and age information of the customer from the captured image. For example, the gender recognition modules 161 and 251 and the age recognition modules 162 and 252 may be used. ) Will utilize commercially available programs such as Face Recognition, Animetrics Face Recognition, Skybiometry Face Detection and Recognition, Face ++, and FaceMark. Therefore, it is possible to extract gender information and age information based on the video image.

In addition, the customer information analysis unit 160, 250 is to extract the emotion information for each menu, the extraction of emotion information for each menu,

From the image information of the captured image, using the gaze direction information of the customer's eyes staring at the menu information for menu selection and menu information corresponding to the gaze direction, the user's interest information is extracted and the interest menu information is extracted. Extracting emotional information (e.g., joy, surprise, sadness, anger, no response, etc.) about the menu that the customer showed interest from the face image of the city and extracting emotion information for each menu that the customer shows interest. It features.

Specifically, in order to extract the emotion information for each menu, the customer information analysis unit 160, 250,

A gaze direction recognition module 163 and 253 for recognizing a gaze direction of a customer's pupil gazing at the menu information for selecting a menu from the image information;

Interest menu information generation module (164, 254) for generating menu information corresponding to the gaze direction by using the recognized gaze direction and menu information for each gaze direction stored in the information DB (166, 256). ),

When extracting the interest menu information, analyzing emotion information of the customer by using the reference image data information by gender and age group stored in the information DB (166, 256) and the captured image information to generate emotion information for each menu showing the customer's interest. Emotional information extraction module for each menu (165, 255),

And information DBs 166 and 256 for storing reference image data information for each gender and age group and menu information for each gaze direction.

The gaze direction recognition modules 163 and 253 perform a function of recognizing the gaze direction of the customer's eyes gazing at the menu information for menu selection.

For example, based on the pupil's position, we calculate where the customer is staring at the menu.

Techniques for calculating such a gaze direction are well known and are also disclosed in Korean Patent Laid-Open No. 10-2009-0052413.

Through this known technology, the gaze direction recognition module analyzes the image information acquired from the image capturing unit 150 to motion sensing the contour of the face of the customer and the pupil position of the eyeball, and analyzes the image data corresponding thereto. The customer's gaze direction can be calculated.

As another technique, Korean Patent Publication No. 10-0851571 discloses a method of determining a change amount and a ratio by separating a face area and a hair area, and then calculating a viewing angle based on a change in the change amount and ratio by comparing the front view angle. have.

Therefore, the gaze direction recognition module may determine the gaze direction of the customer in this manner.

The interest menu information generation module 164 or 254 extracts menu information corresponding to the gaze direction by using the recognized gaze direction and menu information for each gaze direction stored in the information DB 166 and 256, and interest menu information showing a customer's interest. Create

For example, the information DB (166, 256) stores menu information for each gaze direction such as front direction-Gorgonzola, 30 degrees downward from the front side-carbonara, and 45 degrees downward from the front side-pork cut.

At this time, the interest menu information generating module 164, 254 extracts the interest menu information by using the recognized gaze direction and the information DB 166, 256. It is to generate and extract carbonara as interest menu information showing interest.

The emotion information extraction module for each menu 165 and 255 analyzes emotion information of the customer when the interest menu information is extracted to generate emotion information for each menu that the customer is interested in.

Specifically, the emotion information of the customer is analyzed by using the reference image data information for each gender and age group and the captured image information stored in the information DBs 166 and 256 to generate emotion information for each menu that the customer is interested in.

For example, by using commercialized programs such as EmoVu by Eyeris, FaceRect, EyeFace, Kairos, etc., the customer's emotion information is analyzed to generate emotional information for each menu that the customer is currently interested in. If the carbonara menu shows the expressionless face, and the pork cut menu shows a grimace, the current customer is able to create a menu-specific emotion information showing a crush on Gorgonzola.

In addition, the information DBs 166 and 256 store reference image data information for each gender and age group and menu information for each gaze direction in advance.

As a result, the dedicated app mounted on the kiosk 100 and the customer terminal 200 will provide the management server 300 with the analyzed result by analyzing gender information, age information, and emotional information for each menu.

Meanwhile, the menu information provided to the kiosk 100 and the customer terminal 200 by the management server 300 of the present invention includes general menu information and recommended menu information.

In this case, the general menu information is information on the menu that can be generally provided in the store, the recommendation menu information is information on a customized recommendation menu suitable for the gender and age of the customer who is currently considering the selection of the general menu information. It features.

In order to provide information on a customized recommendation menu suitable for the gender and age of the customer, the management server 300,

Using the selection menu information provided by the kiosk 100 and the customer terminal 200, the gender information of the customer, and the age information of the customer, the first recommendation menu information selected by the gender and the age group of the customer is most generated, and the kiosk 100 ) And the second recommendation menu information showing the most interest of the gender and age group of the customer by using the gender information of the customer, the age information of the customer, and the emotional information for each menu provided by the customer terminal 200. .

To this end, the management server 300,

Store-recommended menu information DB that receives and stores selection menu information selected by the customer, gender information of the customer, age information of the customer, and emotional information for each menu from the kiosk 100 and the customer terminal 200 each time the user selects the menu. 350);

The gender of the customer who wants to select the current menu using the gender information, age information of the current customer provided by the kiosk or the customer terminal, the selection menu information stored in the recommended menu information DB for each store, the gender information of the customer and the age information of the customer A recommendation menu information generation unit 360 for each selection order for generating first recommendation menu information indicative of menus most frequently selected by the age group;

The gender and age range of the customer is the most using the current customer's gender information, age information provided by the kiosk or the customer terminal, and the customer's gender information stored in the store-specific recommended menu information DB, the customer's age information, and emotional information for each menu. And a recommendation menu information generation unit 370 for each interest ranking for generating second recommendation menu information indicating a menu showing interest.

Specifically,

Shop recommendation menu information DB 350 receives the selection menu information selected by the customer, the gender information of the customer, the age information of the customer and the emotion information for each menu from the kiosk (100) and the customer terminal (200) every time the customer's menu selection Save it.

The recommendation menu information generation unit 360 according to the selection order includes gender information, age information of the current customer provided by the kiosk or the customer terminal, selection menu information stored in the recommendation menu information DB for each store, sex information of the customer, and age information of the customer. By using the first generation menu information indicating the menu that is most selected by the gender and age group of the customer to select the current menu is generated.

For example, if a customer who is currently in their 40s wants to select a menu through a kiosk or a customer terminal, the selected menu information selected by the customers stored in the shop recommendation menu information DB, the gender information of the customer, and the age information of the customer are used. To generate the first recommended menu information that ranks the menu most selected by women in their 40s, for example, the first recommended menu information is 1st-Gorgonzola Pizza, 2nd-Carbonara, 3rd Spaghetti Can be.

The recommendation menu information generation unit 370 for each interest rank may include gender information, age information of the current customer provided by the kiosk or the customer terminal, gender information of the customer stored in the store-specific recommendation menu information DB, customer age information and menu By using the emotion information, the second recommendation menu information indicating the menus that are most interested in the gender and age group of the customer who wants to select the current menu is generated.

For example, if a customer who wants to select a menu through a kiosk or a customer terminal is a woman in her 20s, the customer uses the selected menu information, the gender information of the customer, and the age information of the customer stored in the recommended menu information DB for each store. By creating a second recommendation menu information that ranks the menus that women in their 20s were most interested in, for example, the second recommendation menu information is 1st- spaghetti, 2nd-gorgonzola pizza, 3rd carbona Can be lye.

That is, the first recommendation menu information, which is the recommendation menu information selected by the existing customers, and the second recommendation menu information that the existing customers are most interested in, may be different, and the customer may have the first and second recommendation menus provided by the management server. Among the information, the existing customers select the menu by using the recommended menu information that is selected the most or by using the recommended menu information that shows the most interest.

On the other hand, according to another additional aspect, the dedicated app mounted on the customer terminal 200,

If the user is outside the store, receives the store unique information from the user and provides to the management server 300,

It is characterized by photographing the customer staring at the menu information for menu selection, extracting the gender information, age information, and emotional information for each menu from the image information of the photographed image and providing the same to the management server 300.

That is, as shown in Figure 1, you can perform the order for the desired menu through the kiosk or the customer terminal of the customer in the store, but if necessary to order the menu in real time using the customer terminal of the customer outside the store. Because the customer can select and enter the unique information of the store you want to order directly, the dedicated app provides a store selection page, and if you select the "A pizza specialty store ** branch on the store selection page, store-specific information 'Pizza_12' is provided to the management server 300.

In addition, by taking a face image of the customer through the camera installed in the customer terminal, and extracting the gender information, age information, emotional information of each menu from the image information of the captured image, the description thereof is the first of the present invention Same as described in the embodiment.

Therefore, by extracting the analyzed gender information, age information, emotional information for each menu to the management server 300, as well as the general menu information as described above, the customer of the current menu selection of the general menu information You will also be able to receive information on customized recommendations for your gender and age.

Although the above has been illustrated and described with respect to the preferred embodiments of the present invention, the present invention is not limited to the specific embodiments described above, it is usually in the technical field to which the invention belongs without departing from the spirit of the invention claimed in the claims. Various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

100: kiosk
200: customer terminal
300: management server
400: kitchen terminal
500: external payment institution

Claims (10)

In the unmanned food store system that can recommend multimedia menus based on customer customization,
It is installed in the store, receives the customer's unique information from the customer, receives the menu information that can be provided in the store from the management server and provides it to the customer, performs the payment and receipt printing for the menu selected by the customer, and the payment is completed. A kiosk 100 for transmitting the information and the customer specific information to the management server;

Dedicated app is installed, the dedicated app receives the store-specific information from the kiosk 100 and provides it to the management server, receives the menu information available in the store from the management server and provides it to the user, for the menu selected by the user A customer terminal 200 performing payment and transmitting the completed menu information and customer specific information to the management server;

Providing menu information that can be provided in the kiosk 100 at the store where the kiosk 100 is installed, or providing menu information that can be provided in the store corresponding to the store specific information provided by the customer terminal 200 to the customer terminal 200. And, when receiving the menu information and customer unique information is completed payment from the kiosk 100 or the customer terminal 200 to the store-specific information provided by the kitchen terminal 400 or the customer terminal 200 linked to the kiosk 100. A management server 300 for performing a cooking request of a corresponding menu to a kitchen terminal 400 linked to a corresponding store and transmitting a cooking completion notification message to a customer terminal corresponding to customer specific information when cooking is completed;

And a kitchen terminal 400 for receiving the cooking request information from the management server 300 and transmitting the cooking completion information for the cooking request to the management server 300 when the cooking is completed.

The kiosk 100,
An image capturing unit 150 for capturing a customer gazing at the menu information to select a menu;
And a customer information analyzer 160 for extracting gender information, age information, and emotional information for each menu from the image information of the image photographed by the image photographing unit and providing the same to the management server 300. Unmanned food retailer system that can recommend customized menu based on multimedia.
The method of claim 1,
The kiosk 100,
A customer specific information input unit 110 for receiving customer specific information from a customer who wants to place an order;
When obtaining the customer specific information through the customer specific information input unit 110, request the menu information that can be provided in the store to the management server, the menu information output to receive the menu information available from the management server and output it to the screen Unit 120;
A payment progress unit 130 for providing request information for a payment request for a menu selected by a customer to an external payment institution and performing a receipt output when the payment is approved;
When the payment is completed, the menu information and customer-specific information transmission unit 140 for transmitting the payment complete menu information and customer specific information to the management server; Food Store System.
The method of claim 1,
The dedicated app mounted on the customer terminal 200,
When the user enters the store, the kiosk 100 installed in the store and the customer terminal 200 is connected to the local area network, and receives the store-specific information from the kiosk 100, and transmits the store-specific information to the management server. At the same time, request menu information that can be provided in the store, the menu information output unit for the customer terminal for receiving the menu information that can be provided from the management server and output to the screen;
Providing payment request information for the menu selected by the customer (user) to an external payment institution, and when the payment is approved, the customer progress payment unit 220 for generating and storing a mobile receipt;
When the payment is completed, the menu information and the store-specific information transmission unit 230 for transmitting the payment complete menu information and customer specific information to the management server; Food Store System.
The method of claim 1,
The management server 300,
When the kiosk 100 or the customer terminal 200 requests menu information that can be provided in the corresponding store, the menu information corresponding to the corresponding store is extracted from the store-specific menu information DB 320, and the extracted menu information is applied. Shop-specific menu information providing unit 310 to provide a kiosk or customer terminal 200,
Store-specific menu information DB (320) that stores the menu information for each store,
Receiving the menu information and customer specific information is completed payment from the kiosk 100 or the customer terminal 200 corresponding to the store specific information provided by the kitchen terminal 400 or the customer terminal 200 linked to the kiosk 100. A cooking request information transmitting unit 330 for transmitting cooking request information for a cooking request of a corresponding menu to a kitchen terminal 400 linked to a store;
When the cooking completion information for the cooking request is received from the kitchen terminal 400, the cooking completion for transmitting a cooking completion notification message to the customer terminal corresponding to the customer specific information provided by the kiosk 100 or the customer terminal 200. Notification message transmission unit 340; unmanned food store system capable of recommending a customized menu based on multimedia, characterized in that it comprises a.
The method of claim 3,
The dedicated app mounted on the customer terminal 200,
An image capturing unit 240 for photographing a customer gazing at the menu information to select a menu;
And a customer information analyzer 250 for extracting gender information, age information, and emotional information for each menu from the image information of the image photographed by the image photographing unit and providing the same to the management server 300. Unmanned food store system that can recommend customized menu based on multimedia.
The method of claim 5,
The customer information analysis unit 250,
Gender recognition module 251 for extracting the gender information of the customer by using the reference image data information for each gender and age group stored in the information DB 256 and the captured image information;
An age recognition module 252 for extracting age information of a customer by using reference image data information of each gender and age group stored in the information DB 256 and photographed image information;
A gaze direction recognition module 253 for recognizing a gaze direction of a customer's eyes gazing at the menu information for selecting a menu from the image information;
An interest menu information generation module 254 for extracting menu information corresponding to the gaze direction by using the recognized gaze direction and menu information for each gaze direction stored in the information DB 256 to generate interest menu information showing a customer's interest;
When extracting the interest menu information, analyzing the emotion information of the customer by using the reference image data information by gender and age group and the captured image information stored in the information DB 256 to generate the emotion information for each menu that the customer is interested in. Emotional information extraction module (255) for each menu;
Unmanned food retailer system that can recommend a customized menu based on multimedia, characterized in that it comprises an information DB (256) for storing reference image data information by gender, age group and gaze direction menu information.
The method of claim 1,
The management server 300,
Using the selection menu information provided by the kiosk 100 and the customer terminal 200, the gender information of the customer, the age information of the customer to generate and provide the first recommendation menu information selected by the gender and age group of the customer most,
By using the gender information of the customer, the customer age information, and emotional information for each menu provided by the kiosk 100 and the customer terminal 200 to generate and provide the second recommendation menu information that the customer's gender and age group showed the most interest. Unmanned food retailer system that can recommend a customized menu based on multimedia, characterized in that.
The method of claim 4, wherein
The management server 300,
Store-recommended menu information DB that receives and stores selection menu information selected by the customer, gender information of the customer, age information of the customer, and emotional information for each menu from the kiosk 100 and the customer terminal 200 each time the user selects the menu. 350);
The gender of the customer who wants to select the current menu using the gender information, age information of the current customer provided by the kiosk or the customer terminal, the selection menu information stored in the recommended menu information DB for each store, the gender information of the customer and the age information of the customer A recommendation menu information generation unit 360 for each selection order for generating and providing first recommendation menu information indicative of a menu most frequently selected by the age group;
The gender and age range of the customer is the most using the current customer's gender information, age information provided by the kiosk or the customer terminal, and the customer's gender information stored in the store-specific recommended menu information DB, the customer's age information, and emotional information for each menu. Multimedia-based unmanned food capable of recommending customized menus, including; recommendation menu information generation unit 370 for generating interests for generating and providing second recommendation menu information indicating a menu showing interest. Dealer system.
The method of claim 1,
The dedicated app mounted on the customer terminal 200,
If the user is outside the store, receives the store unique information from the user and provides to the management server 300,
Multimedia-based, characterized in that photographing the customer staring at the menu information for menu selection, extracts the gender information, age information, emotional information of each menu from the image information of the photographed image and provide it to the management server 300 Unmanned grocery store system with customized menu recommendations.
The method of claim 1,
The customer information analysis unit 160,
Gender recognition module 161 for extracting the gender information of the customer by using the reference image data information by gender and age group stored in the information DB 166, and the captured image information,
An age recognition module 162 for extracting age information of the customer by using reference image data information of each gender and age group stored in the information DB 166 and photographed image information;
A gaze direction recognition module 163 for recognizing a gaze direction of a customer's pupil gazing at the menu information for selecting a menu from the image information;
An interest menu information generation module 164 that generates menu information corresponding to the gaze direction by using the recognized gaze direction and menu information for each gaze direction stored in the information DB 166 to generate interest menu information showing a customer's interest;
When extracting the interest menu information, analyzing the emotion information of the customer by using the reference image data information by gender and age group and the captured image information stored in the information DB 166 to generate the emotion information for each menu that the customer is interested in. Emotion information extraction module (165) for each menu;
Unmanned food retailer system that can recommend a customized menu based on multimedia, characterized in that it comprises an information DB (166) for storing reference image data information by gender and age group and menu information by gaze direction.

Images