JP7376272B2 - Customer management systems, programs, devices and methods - Google Patents

Description

TECHNICAL FIELD The present invention relates to a management technique for managing customer information of a store such as a restaurant and information on the contents of offerings such as cooked products.

In managing stores such as restaurants, their customers, and the cooking provided by the stores, images taken of customers and prepared foods are utilized.
Regarding the management of customer information, we take photos of the customer's face and the entire customer group at the store entrance, perform face matching using the facial image, and determine the number of customer groups, customer gender, height category, and customer group composition. It is known to provide a selection menu (for example, Patent Document 1).

Japanese Patent Application Publication No. 2008-262435

By the way, in the management of a restaurant, improving and developing the quality of menus and services provided affects the acquisition of new customers, and the attraction and retention of repeat customers. However, in restaurants, it is difficult to elicit the true intentions of the customers who have received the service, even if staff members directly collect impressions and opinions from customers or set up questionnaires. There are challenges.
The present inventor believes that analyzing information such as facial expressions and eating and drinking times of customers when eating and drinking, and deriving trend information including customer preferences is extremely important for developing menus and services and improving quality. We obtained knowledge that it would be useful information.
Such a problem is not disclosed or suggested in Patent Document 1, and the configuration disclosed in Patent Document 1 cannot solve this problem.

Therefore, the purpose of the present invention is based on the above-mentioned problems and knowledge, and based on the above-mentioned problems and knowledge, the present invention is based on the following: The goal is to obtain information from customers, derive trend information including customer preferences, and use the information to attract customers.

In order to achieve the above object, according to one aspect of the customer management system of the present invention, a food and beverage image including at least a cooked product to be evaluated and a face image of a visitor who consumes the cooked product, a photographing unit that acquires a table image of the customer who is ingesting the food ; and information on changes in facial expressions extracted from a part or the whole of the face image of the customer when the customer ingests the cooked product extracted from the food and drink image; Along with reaction information including information determined based on the change information, either or both of intake time information and remaining amount of cooked food after intake information is analyzed, and from the table image at least other people who visited the store with the customer are analyzed. Analyzing the number of people visiting the store including customers, reading an evaluation table storing correction information for modifying the evaluation of the reaction information according to the number of customers, and correcting the reaction information according to the evaluation table, and a processing unit that calculates trend information including preference trends of the store visitor in accordance with at least the reaction information.

The customer management system may further include an information presentation unit that presents at least the trend information by acquiring the food and beverage images.
In the above customer management system, the storage unit further stores any one or more of the food and drink image, the reaction information, the intake time information, the cooked product remaining amount information, or the trend information in association with attributes of the visitor. may be provided.
In the above customer management system, the processing unit may further calculate the trend information for at least each visitor or each cooked item.
In the above customer management system, the photographing section further acquires a motion image of the visitor before or during the intake of the cooked product, and the processing section analyzes the visitor's evaluation information regarding the cooked product based on the motion image. Then, the trend information may be calculated.
In the above customer management system, the processing unit may further update the trend information regularly or irregularly after acquiring the eating and drinking images .

In order to achieve the above object, according to one aspect of the program of the present invention , a food and drink image including at least a cooked product to be evaluated and a face image of a visitor who consumes the cooked product, A function of acquiring a table image of the visitor who is about to eat the food, using a photographing unit, and information on a change in facial expression extracted from a part or the whole of the face image of the visitor when the customer ingests the prepared food extracted from the food and drink image. , or reaction information including information determined based on the change information, as well as one or both of the intake time information and the remaining amount of cooked product information after intake, and based on the table image, at least the person who visited the store with the visitor Analyzing the number of people visiting the store, including other customers, reading out an evaluation table that stores correction information for modifying the evaluation of the reaction information according to the number of customers, and modifying the reaction information according to the evaluation table. However, at least a function of calculating trend information including the preferences of the store visitor according to the reaction information is realized by a computer .

In order to achieve the above object, according to one aspect of the customer management device of the present invention, a food and beverage image including at least a cooked product to be evaluated and a facial image of a visitor who consumes the cooked product, a photographing unit that acquires a table image of the customer who is ingesting the food ; and information on changes in facial expressions extracted from a part or the whole of the face image of the customer when the customer ingests the cooked product extracted from the food and drink image; Along with reaction information including information determined based on the change information, either or both of intake time information and remaining amount of cooked food after intake information is analyzed, and from the table image at least other people who visited the store with the customer are analyzed. Analyzing the number of people visiting the store including customers, reading an evaluation table storing correction information for modifying the evaluation of the reaction information according to the number of customers, and correcting the reaction information according to the evaluation table, and a processing unit that calculates trend information including preference trends of the store visitor in accordance with at least the reaction information.

In order to achieve the above object, according to one aspect of the customer management method of the present invention, a food and beverage image including at least a cooked product to be evaluated and a facial image of a visitor who consumes the cooked product, a step of acquiring a table image of the customer who is about to consume the food in a photographing unit; and information about changes in facial expressions extracted from a part or the whole of the face image of the customer when the customer consumes the prepared food extracted from the food and drink image. , or reaction information including information determined based on the change information, as well as one or both of the intake time information and the remaining amount of cooked product information after intake, and based on the table image, at least the person who visited the store with the visitor Analyzing the number of people visiting the store, including other customers, reading out an evaluation table that stores correction information for modifying the evaluation of the reaction information according to the number of customers, and modifying the reaction information according to the evaluation table. and a step of calculating trend information including preference trends of the store visitor in accordance with at least the reaction information, using a processing section.

According to the present invention, any of the following effects can be obtained.
(1) By analyzing information that appears unconsciously in store visitors, such as reactions to cooked products, intake time, and amount of leftover cooked food, and by deriving trend information including store visitors' preferences, we can improve the services provided at the store and It is possible to understand the evaluation of cooked products, which can be used for customer attraction information such as developing new customers and retaining customers.
(2) It is possible to provide services such as suggesting cooking products based on information obtained from store visitor trends, and it is possible to improve customer service to store visitors without relying on the customer service experience of employees. can.
(3) It is possible to contribute to the development of menus such as cooked products by referring to trend information.
(4) Trend information can be used to improve store operations.

FIG. 1 is a diagram illustrating a configuration example of a customer management system according to a first embodiment. FIG. 2 is a block diagram showing an example of a functional configuration of a customer management system. FIG. 3 is a diagram illustrating an example of a photographing range by a camera in the audience seating floor. It is a figure showing an example of the photographing state of the eating and drinking image. It is a figure showing an example of a cooked product image. It is a figure which shows an example of the analysis process of a cooked product image. It is a diagram showing an example of the hardware configuration of a management server. FIG. 3 is a diagram showing an example of an integrated information file. It is a figure showing an example of an order information file. It is a figure showing an example of a customer information file. It is a figure which shows an example of a trend information file. It is a figure showing an example of trend information. It is a flowchart which shows an example of customer management processing. It is a figure showing an example of composition of a customer management system concerning a 2nd embodiment. It is a diagram showing an example of a photographing area on a store floor. It is a figure which shows an example of the photography state of a visitor image. It is a flowchart which shows an example of customer management processing. It is a figure showing an example of functional composition of a customer management system concerning a 3rd embodiment. FIG. 3 is a diagram showing an example of an evaluation table for ordered cooked products based on the number of customers. It is a flowchart which shows an example of customer management processing. It is a figure showing the example of composition of the customer management system concerning a 4th embodiment. It is a figure which shows an example of the photography state of the eating-and-drinking image and a user's action image. It is a figure which shows an example of a trend information file. It is a flowchart which shows an example of customer management processing.

[First embodiment]
FIG. 1 shows a customer management system according to a first embodiment of the present invention. The configuration shown in FIG. 1 is an example, and the present invention is not limited to such a configuration.
This customer management system 2 is an example of a system that calculates trend information, including reactions to cooked products ordered by store visitors P, preferences of store visitors P, etc., using information extracted from image information. As shown in FIG. 2, it has a photographing section 4 and a management server 6 that can photograph the inside of the store. The photographing unit 4 is an example of an image acquisition means of the present invention, and may be any device that can acquire at least image information such as the seated position of the customer P and the ordered prepared food ordered by the customer P. This photographing section 4 has a single camera or a plurality of cameras, and each camera is installed on the floor of the store. That is, for example, a plurality of in-store cameras (hereinafter simply referred to as "cameras") are installed on the customer seating floor 8 where visitors P eat and drink. The camera is, for example, a video camera, and in addition to photographing at least the visitor P sitting on the seat 10 arranged on the customer seating floor 8, it also photographs the cooked food 14 on the table 12 and the like.

The management server 6 is an example of a processing section of the present invention, and includes, for example, a server main body 16, a monitor 18, an operation input section 20, and the like. The server main body 16 is, for example, a computer, and can perform a data communication function of transmitting and receiving data to and from the camera of the photographing unit 4, and a data processing function of generating trend information based on the received data. For example, the camera of the photographing section 4, the operation input section 20, the management terminal 22, and the like are linked to the server main body 16 through wired or wireless connections.
The monitor 18 is an example of information presentation means controlled by the server main body 16, and for example, an LCD (Liquid Crystal Display) or the like is used. The monitor 18 displays, for example, image information photographed by the photographing section 4, trend information generated through information processing, and the like.
The operation input unit 20 is used to input information about the visitor P and other information. The operation input section 20 includes, for example, a keyboard, a mouse, and the like, as well as a touch panel for inputting input by touching the display screen of the monitor 18.

The management terminal 22 is an example of a device that is connected to the management server 6 through a wired connection or a wireless connection and can exchange information with the management server 6. The management terminal 22 includes, for example, information processing terminals 22-1 and 22-2 such as a tablet terminal or a smartphone equipped with a communication function. The wireless connection between the management terminal 22 and the management server 6 uses, for example, an Internet connection, a Wi-Fi (registered trademark) connection, or the like. In addition, the management terminal 22 may include, for example, a cashier counter used to manage payment for services provided to the store visitor P, an information terminal of a sales management system such as a POS (Point of Sale), and the like.

<Configuration example of customer management system 2>
FIG. 2 shows an example of a functional configuration of a customer management system using the management server 6. As shown in FIG.
In this customer management system 2, for example, a camera in a photographing section 4 photographs an image of a visitor P eating and drinking, and the image is transmitted to a management server 6. At this time, the photographing unit 4 may receive photographing instruction information from, for example, the management server 6 side.
The management server 6 includes, for example, a processing section 24 and a storage section 26, as shown in FIG.
The processing unit 24 is an example of a means for capturing customer information, food and drinks, and image information of eating and drinking conditions, and analyzing the preference trends of the customer P, evaluations of the provided services, etc. from this image information. The processing unit 24 includes, for example, analysis that analyzes the image information of the acquired eating and drinking images and extracts and analyzes the identification of the visitor P, the reaction information of the visitor when eating and drinking, the proportion of leftover food in the prepared food, etc. It has a processing section 28 and a trend information calculation processing section 30 that calculates trend information including store visitors' preferences from the information analyzed by the analysis processing section 28. The reaction information extracted by the processing unit 24 is an example of information extracted from an image in which a reaction such as a change in a visitor's facial expression when eating or drinking is captured. It includes information that can be used for calculations and information processing, such as the amount and rate of change in the corners of the mouth.
The storage unit 26 is an example of a means for storing image information obtained by photographing the visitor P, personal information of the visitor P, information on cooked products ordered by the visitor P, and the like.
The information processed by the processing unit 24 is stored in the storage unit 26 and is also transmitted to the management terminal 22 side. The management terminal 22 may display the generated visitor information, trend information, and the like. Then, the store clerk can serve customers while checking customer information, trend information, etc. displayed on the management terminal 22.

<Acquisition of visitor images and cooked product images>
FIG. 3 shows an example of the photographing range of the camera within the audience seating floor.
The audience floor 8 includes, for example, a plurality of tables 12-1, 12-2, 12-3, and 12-4, and a plurality of seats 10 are arranged for each table. The position of the seat 10 with respect to the table 12 is an example of address information indicating the position where the visitor P eats and drinks. Further, on the audience floor 8, for example, a plurality of cameras 4-1, 4-2, 4-3, 4-4, . . . are arranged. For these cameras 4-1, 4-2, 4-3, 4-4, etc., photographing ranges are set in directions corresponding to address information assigned to each camera.
For example, address information for seven seats 10-1-1, 10-1-2, 10-1-3, . . . 10-1-7 is set in the table 12-1. The camera 4-4 is disposed facing the table 12-1, for example, and has a focus F4 for narrowing down the address information of the object to be photographed. Photographing directions and focuses F1, F2, F4, and F6 are also set for the other cameras 4-1, 4-2, . . . 4-6, respectively.

<About photographing and analysis processing of food and drink images>
In the photographing process, for example, as shown in A of FIG. 4, the camera 4-X, which has been set in advance for the seating position (address information) of the visitor P, is activated to capture the face of the visitor P who is eating the prepared food. (eating and drinking image) and an image of the cooked product 14 that the visitor P is eating and drinking (cooked product image).
For example, as shown in FIG. 4B, the camera 4-X captures a plurality of eating and drinking images 32 including the face of the customer P just before eating the cooked product 14 and after putting it in the mouth. The camera 4-X may start photographing, for example, at the timing when the cooked food 14 is served, and photograph the eating and drinking image 32 until after the visitor P eats and drinks the cooked food 14. At this time, the camera 4-X may, for example, take images for a certain period of time from just before the visitor P takes the first bite, or may take images continuously until the cooked product 14 is completely eaten. All you have to do is take a picture. Further, the photography may be video photography, or photography may be done periodically at a timing that coincides with eating and drinking activities or at set time intervals.
In the management server 6, for example, as shown in FIG. 4B, the analysis processing unit 28 extracts the face portion 34 of the visitor P shown in the eating and drinking image 32, and analyzes changes in facial expressions when eating and drinking. To extract the face portion 34, for example, a digital camera's face recognition function, smile detection function, etc. may be used.

The analysis processing unit 28 performs data processing on the extracted image data of the face portion 34 to generate an analysis image 36, as shown in FIG. 4C, for example. This analysis image 36 is created by applying grid information of a predetermined size to, for example, the data of the extracted face portion 34. Then, the analysis processing unit 28 analyzes the state of the facial expression of the photographed visitor P for each of one or more small areas 38 of the grid information. At this time, the analysis processing unit 28 extracts changes in the facial expression of the visitor P due to eating and drinking, for example. This change in facial expression is an example of reaction information to the cooked product, and includes changes that indicate that it matches the taste, such as "smile" and "surprise," as well as changes that do not match the taste, such as "disappointment" and "rejection." There are changes that represent this.
The customer management system 2 uses reaction information based on changes in facial expressions, for example, to extract trend information representing the preferences of the customer P regarding the seasoning of cooked products and the contents provided.
In extracting changes in facial expressions, for example, in addition to determining the entire facial expression by comparing it with facial pattern information, the determination may also be made by combining changes in facial parts such as the "corners of the mouth" and the "eyes."

<About photographing and analysis processing of cooked product images>
The camera 4-X photographs a cooked product image 40 at the time of serving or just before eating or drinking, as shown in FIG. 5A, for example. At this time, the camera 4-X may fix the photographing position and focus magnification in order to compare the cooked food 14 as time passes and as eating and drinking progresses. Furthermore, for example, when the visitor P moves the cooked food 14, the camera 4-X may adjust the focus magnification and photographing position to match the shape and size of the cooked food before eating or drinking.
For example, as shown in FIG. 5B, the camera 4-X periodically takes images of the cooked food after eating and drinking starts, and captures the cooked food image 42 when eating and drinking is finished or at a predetermined timing. Take a photo.

Then, as shown in FIG. 6, for example, the analysis processing unit 28 of the management server 6 generates an analysis image 43 to which grid information is applied using the acquired cooked product images 40 and 42. This analysis image 43 is, for example, a combination of an image 40 before eating and drinking and an image 42 after eating and drinking, and a portion 44 of eating and drinking and a portion 46 of leftover food are extracted. The analysis processing unit 28 then performs, for example, a binarization process on this analysis image 43, extracts the leftover portion 46 of the cooked product 14 before eating and drinking, and calculates the amount of leftover food or the ratio of leftover food.
The information on the change in the facial expression of the visitor P and the amount of leftover food extracted by the processing unit 24 is stored in the storage unit 26.

<Example of server configuration>
FIG. 7 shows an example of the hardware configuration of the server main body.
The server body 16 of the management server 6 includes a processor 50, a memory section 52, a communication section 54, a timer 56, an input/output section (I/O) 58, and the like.
The processor 50 is an example of a processing unit that executes information processing, and processes information using various programs such as an OS (Operating System) and a customer management program stored in the memory unit 52.
This information processing includes, for example, a) photography control, b) acquisition of eating and drinking images of visitor P, c) acquisition of cooked product images, d) storage of visitor information, and e) determination of reaction information of visitors due to eating and drinking. , f) Calculation of eating and drinking time of customers, g) Calculation of number of customers, h) Calculation and presentation of trend information, i) Information processing such as control of various functional units.

The memory unit 52 is an example of the storage unit 26 of the management server 6, and stores the OS, customer management program, and customer management information. This memory section 52 uses storage elements such as ROM (Read Only Memory), RAM (Random Access Memory), and EEPROM (Electrically Erasable and Programmable Read Only Memory). The memory unit 52 may be an external storage device such as a hard disk device connected to the processor 50 via a wireless connection or a wired connection.

The communication unit 54 executes communication processing for exchanging information with external devices such as the management terminal 22 (FIG. 1) and the accounting terminal 114 (FIG. 16) under the control of the processor 50.
The timer 56 is an example of a timer, and may be a hardware timer or a software timer operated by the processor 50. In the customer management process, the elapsed time of the visitor P is measured, such as the time the visitor P stays at the store and the time from when the ordered cooked food is served until the visitor P finishes eating and drinking. Furthermore, the timer 56 may measure time including the elapsed time related to the store, such as the time required from the customer P ordering the cooked food until the food is served. In the customer management process, evaluation information about the quality of the service provided by the store is collected in addition to evaluation information about the cooked products from the elapsed time.

The I/O 58 performs photographing operations, image acquisition, and monitoring of the cameras 4-1, 4-2, 4-3, ..., 4-N installed on the audience floor 8 and other floors under the control of the processor 50. It performs input/output related to control such as information presentation by the operation input section 18 and acquisition of operation input information from the operation input section 20.

<Information processing of management server 6>
a) Shooting control The management server 6 activates the cameras 4-1, 4-2, ... 4-N when the store opens or when a visitor P enters the store, and performs the set operations. mode. The management server 6 starts photographing with the set camera, for example, when the visitor P enters the customer seating floor 8 or reaches the seating position (address information) of a predetermined seat. For example, when the cameras 4-1, 4-2, . To detect the entrance and exit of the visitor P, for example, an entrance/exit monitoring camera installed on a part of the customer seating floor 8 may be set and the captured images and videos may be used, or a person may be detected at the entrance of the customer seating floor 8. You can use sensors etc. Images of the visitors P entering and exiting the customer seating floor 8 and the cooked products are taken by any one of the cameras 4-1, 4-2, . . . 4-N, or a combination thereof under the control of the management server 6.

b) Obtaining eating and drinking images of visitor P The management server 6 obtains eating and drinking images of visitor P from each camera 4-1, 4-2, . . . 4-N. Cameras 4-1, 4-2, ... 4-N are captured when a visitor P is seated at the seating position set for each, and for example, when the visitor P starts eating and drinking prepared food, is eating and drinking, or when eating and drinking is completed. A face image of the visitor P is photographed at that time.
For example, the management server 6 may acquire captured images sequentially from the cameras 4-1, 4-2, . . . , 4-N, or may acquire images all at once when a predetermined timing is reached. It's okay.

c) Acquisition of cooked product images The management server 6 acquires cooked product images taken by the cameras 4-1, 4-2, . . . , 4-N. The cooked product image acquired here is stored, for example, in an integrated information file 60 (FIG. 8) formed in the storage unit 26.

d) Storage of visitor information The management server 6 stores an image of the visitor P and a cooked product image of the cooked product ordered by the visitor P in the memory unit 52 as visitor information. This visitor information is stored in the comprehensive information file 60. Further, the management server 6 may perform a process of determining the attributes of the visitor P using the visitor image. The attribute information of the visitor P includes, for example, the result of determining whether the visitor is a new visitor or a repeat visitor using gender information, age information, past visit history, and the like. This determination result is stored in the comprehensive information file 60.

e) Extraction of reaction information of visitor P due to eating and drinking The management server 6 extracts at least the image of the visitor immediately before eating or drinking the cooked food, among the images of the visitor obtained from the cameras 4-1, 4-2, ... 4-N. An image and an image taken immediately after eating and drinking the cooked product, that is, when the person starts eating the cooked product, are extracted. The management server 6 then extracts reaction information such as changes in the facial expressions of the visitor P before and after eating and drinking. To extract this reaction information, for example, a camera's face recognition technology or smile detection technology may be used. Once the management server 6 extracts the reaction information of the visitor P, the management server 6 may determine the visitor's evaluation of the cooked product based on the extraction result.

f) Calculation of visitor's eating and drinking time, etc. The management server 6 stores, for example, the time from when the cooked food ordered by visitor P is served until the end of eating and drinking as the eating and drinking time. Furthermore, the management server 6 stores the time from when the cooked food is ordered until it is served as the service provision time. In addition, the management server 6 may calculate, for example, the average eating and drinking time for the same cooked product from the eating and drinking time of the visitor and the number of orders. This average eating and drinking time may be calculated using, for example, the following equation (1).
Average eating and drinking time = total value of customers' eating and drinking time / number of orders...(1)
For example, this average eating and drinking time was extracted by using the eating and drinking time of all customers who ordered the same prepared item, as well as by narrowing down conditions such as specifying the time of visit and whether the customer was a new visitor or a repeat visitor. It may be calculated using information.
Furthermore, the management server 6 may calculate, for example, the average value of the service provision time from order to serving for the same cooked product for each store or for each time period.

g) Calculation of number of store visitors, etc. The management server 6 counts the number of store visitors P, and calculates the total number of store visitors Npm, the number of new store visitors Nnm, and the number of repeat visitors Nrm. At the same time, the ratio (%) of the number of new visitors Nnm to the total number of visitors Npm and the number Nrm (%) of repeat visitors to the total number of visitors Npm are calculated for a specific period.

h) Calculation of trend information and its presentation The management server 6, for example, uses the acquisition of eating and drinking images of the customer P as an opportunity to search the memory unit 52 for order information and customer information, and calculates trends in the number of orders for each cooked item. , trends in eating and drinking times, changes in service hours, reactions to prepared foods, etc. are calculated as trend information.
In addition, the management server 6 uses the visitor information stored in the memory unit 52, for example, to determine attributes including whether or not the visitor is a new visitor, and to monitor trends in the visitor's length of stay or the number of visitors. It may be calculated as information.
The management server 6 calculates trend information at least for each cooked item, and causes the monitor 18 and the information presentation section of the management terminal 22 to present the trend information.

i) Control of Various Functional Units The management server 6 updates the trend information in the memory unit 52 regularly or irregularly, using the acquisition of food and drink images as an opportunity.

<Database>
The memory unit 52 stores a database that stores customer management information. This database stores, for example, information in a plurality of file formats in which data collected by the customer management system 2 is summarized for each specific item. The database includes the general information file 60 (Figure 8), order information file 70 (Figure 9), visitor information file 80 (Figure 10), trend information file 90 (Figure 11), and monthly trend information (Figure 12). A), annual trend information (B in FIG. 12), and other information are included.

<General information file 60>
FIG. 8 shows an example of the comprehensive information file 60. This comprehensive information file 60 is created, for example, when a visitor P comes to the store, and stores information for each visitor P.
The comprehensive information file 60 includes, for example, a number section 61, a date and time section 62, a food/drink image section 63, a visitor attribute section 64, a time information section 65, an ordered cooked product section 66, and a visitor section 67. Information numbers are assigned to the number section 61, for example, in the order in which visitors visit the store. The date and time section 62 stores date and time information at which the eating and drinking information of the visitor P was acquired. The eating and drinking image section 63 stores eating and drinking images taken of the visitor P. The visitor attribute section 64 includes a new visitor section 64-1 and a repeat visitor section 64-2 as examples of attribute information. The new visitor section 64-1 stores information indicating that the visitor is a new visitor, and the repeat visitor section 64-2 stores information indicating that the visitor is a repeat visitor. The visitor attribute section 64 may store, for example, information such as the gender and age of the visitor P (not shown) as attribute information.

The time information section 65 includes an entry time section 65-1, an exit time section 65-2, a stay time section 65-3, and an eating and drinking time section 65-4. Store entry time section 65-1 stores time information when visitor P enters the store. The leaving time section 65-2 stores information about the time when the visitor P leaves the store. The staying time section 65-3 stores the calculation result of (leaving time - entering time), for example, as the staying time from the time of entering the store to the time of leaving the store. For example, the eating and drinking time section 65-4 stores information for each cooked item ordered by visitor P, from the time when eating and drinking starts or when the cooked item is served, to the time when eating and drinking ends or the cooked item is collected (lowered). To measure time.
The order cooked product section 66 includes a cooked product image section 66-1 at the time of serving, a cooked product image section 66-2 at the end of eating and drinking, a cooked product section 66-3, a number of items section 66-4, and a leftover ratio section 66-5. is included. The cooked product image section 66-1 at the time of serving stores images of cooked products before eating and drinking. The cooked product image section 66-2 at the end of eating and drinking stores images of cooked products including plates and leftover food at the end of eating and drinking. The cooked item section 66-3 stores attribute information such as a name that identifies the ordered cooked item. The number of cooked items is stored in the number of items section 66-4. The leftover food ratio section 66-5 stores information on the ratio of the amount of leftover food calculated using the eating and drinking images. The amount of leftover food and its proportion may be calculated using, for example, images of the cooked product at the time of serving and images of the cooked product at the end of eating and drinking.

The store visitor section 67 includes a total store visitor number section 67-1, a new store visitor number section 67-2, and a returning store visitor number section 67-3. The total number of visitors to the store 67-1 stores the total number of visitors over a certain period of time. The number of new visitors is stored in the new visitor number section 67-2. The number of repeat visitors is stored in the repeat visitor number section 67-3.

<Order information file 70>
FIG. 9 shows an example of the order information file 70. In this order information file 70, individual files 70-1, 70-2, . . . 70-N are created for each cooking item, and information is stored when an order is placed.
The order information file 70 includes, for example, a number section 71, a date and time section 72, a visitor information section 73, an evaluation information section 74, and a time information section 75. Information numbers are assigned to the number section 71 in order of order. The date and time section 72 stores the ordered date and time information.
The visitor information section 73 includes a visitor image section 73-1, an attribute information section 73-2, and a number of visitors section 73-3. The image of the visitor P is stored in the visitor image section 73-1. The attribute information section 73-2 stores, for example, information for determining whether the visitor is a new visitor, information on the visitor's age or gender, or information on a combination thereof. The number of visitors section 73-3 stores information indicating whether a person is visiting the store alone or in a group of multiple people.
The evaluation information section 74 includes a food and drink image section 74-1, an action image section 74-2, and an attribute information section 74-3. The eating and drinking image section 74-1 stores images of the customer P eating and drinking for each ordered cooked item. This eating and drinking image is an image taken, for example, at the timing of the start and end of eating and drinking for each cooked item, and is an example of reaction information indicating a change in the facial expression of the visitor P due to eating and drinking. The motion image section 74-2 stores images of the motions of the visitor P from the time the cooked food is served to the time he or she starts eating and drinking. This action image includes information on, for example, a reaction to the prepared food being served, a reaction upon receiving a meal delivery service, or a reaction to the prepared food due to the use of seasonings. The attribute information section 74-3 stores information indicating whether this is the cooked product ordered by the visitor P for the first time or whether the visitor P is a repeat customer.

The time information section 75 includes an order time section 75-1, a serving time section 75-2, a eating and drinking start time section 75-3, an eating and drinking end time section 75-4, and an eating and drinking time section 75-5. The order time field 75-1 stores information on the time when the cooked item was ordered. The serving time section 75-2 stores information on the time when the ordered cooked food was served. The eating/drinking start time field 75-3 stores information on the time when the visitor P starts eating the prepared food that has been served. The eating/drinking end time section 75-4 stores information on the time when the visitor P has finished eating for each cooked item. The eating and drinking time section 75-5 stores information on the time required to eat and drink the cooked food. This eating and drinking time is calculated from the time information stored in the eating and drinking start time section 75-3 and the eating and drinking end time section 75-4.
In addition, the order information file 70 may also include information such as the number of items ordered within a certain period of time and attribute information of the cooked item, such as whether the cooked item is a staple food or a main dish.

<Visitor information file 80>
FIG. 10 shows an example of the visitor information file 80. In the store visitor information file 80, a plurality of store visitor information files 80-1, 80-2, .
The visitor information file 80 includes a number section 81, a visitor information section 82, a time information section 83, and an ordered cooked product section 84. The number section 81 stores, for example, an information number in the order of visitors or an identification number assigned to a visitor. The visitor information section 82 includes a visitor image section 82-1, an attribute information section 82-2, and a number of visitors section 82-3. A visitor image is stored in the visitor image section 82-1. The attribute information section 82-2 stores, for example, information indicating whether the customer is a new visitor or a repeat visitor, age information, gender information, or a combination thereof. The number of people section 82-3 stores information indicating whether a person is visiting the store alone or in a group of multiple people.
The time information section 83 includes an entry time section 83-1, an exit time section 83-2, and a stay time section 83-3. Store entry time section 83-1 stores time information when visitor P enters the store. Store leaving time section 83-2 stores time information when visitor P leaves the store. The stay time field 83-3 stores the stay time from the time of entering the store to the time of leaving the store. The staying time may be calculated using the time information stored in the store entry time section 83-1 and the store exit time section 83-2.
The ordered cooked product section 84 includes a cooked product image section 84-1, a food/drink image section 84-2, a cooked product section 84-3, a number of items section 84-4, and a leftover ratio section 84-5. Cooked product images are stored in the cooked product image section 84-1. This cooked product image is, for example, a cooked product arranged near the store visitor P, or a cooked product ordered by the store visitor P, and is image data of a photograph of the cooked product eaten by the store visitor P. That is, in the cooked product image section 84-1, an image of a cooked product is associated with a person who has eaten or eaten the cooked product. Thereby, it is possible to prevent the cooked food from being evaluated based on the facial expressions of people who are not actually eating or drinking, due to conditions such as the food being served near a specific customer P, for example.
The eating and drinking image section 84-2 stores images of the customer P eating and drinking for each ordered cooked item. The cooked item section 84-3 stores information identifying the cooked item, such as the name of the cooked item. The number of items ordered is stored in the number of items section 84-4. The leftover food ratio section 84-5 stores the ratio information of the amount of leftover food calculated using the eating and drinking images.

<Trend information file 90>
FIG. 11 shows an example of the trend information file 90. This trend information file 90 is an example of trend information calculated using information acquired from the store visitor P, and includes information on the cooked items ordered by the store related to the time information, reaction information on the cooked items, etc. Stored. Information stored in the general information file 60, order information file 70, and store visitor information file 80 may be used to calculate the trend information.
The trend information file 90 includes a number section 91, a time information section 92, and an ordered cooked product information section 93. The number section 91 stores an information number. The time information section 92 stores date and time information 92-1 when the trend was calculated, and period information 92-2 representing the collection date and period of data used for trend calculation.
The ordered cooked product information section 93 includes a cooked product section 93-1, an order number section 93-2, a leftover food ratio section 93-3, a food and drink time section 93-4, a reaction information section 93-5, and a reorder information section 93. -6, evaluation information section 93-7, etc. The cooked item section 93-1 stores the name of the ordered cooked item, attribute information of the cooked item, and the like. The order number section 93-2 stores the number of orders for cooked items. The eating and drinking time section 93-4 stores, for example, the eating and drinking time for each visitor as well as average eating and drinking time information for the number of orders. The reaction information section 93-5 stores changes in facial expressions extracted from eating and drinking images of visitor P when eating and drinking, and information determined based on the changes in facial expressions. The reorder information 93-6 is information extracted from the order information of the repeat visitor P, and stores information such as the number of orders for the same cooked product in the past. The evaluation information 93-7 stores evaluation information for the cooked product calculated by including the information stored in the reaction information section 93-5, the eating and drinking time section 93-4, the reorder information section 93-6, and other information. Ru. For this evaluation information, the information to be used and the calculation method may be set for each store that provides cooked products. Evaluation information is used, for example, when calculating the number of reorders, when evaluating the amount of food provided based on the percentage of leftovers among customers, and when determining changes in the number of ordered prepared items by time, date, and day of the week. It may be calculated depending on the purpose, such as when

<Trend information>
FIG. 12 shows an example of trend information calculated based on customer P's eating and drinking images. The trend information shown in FIG. 12 is assumed information for representing the transition status of an image of the store visitor P, information ordered by the store visitor P, etc., and the present invention is not limited to such content.
The trend information shown in A of FIG. 12 is an example of monthly trend information for each order-cooked item, for example. This monthly trend information includes, for example, a bar graph showing daily information on the horizontal axis and the number of orders on the vertical axis, and a line graph showing reaction information (Re), information on intake time of cooked products (Ti), and the proportion of leftover food (Rp). It is shown. The number of orders also shows the number of new orders (NOr) and the number of reorders (ROr). That is, monthly trend information is trend information that specifies the transition state of information that changes over a predetermined period of time. By calculating this monthly trend information, it is possible to grasp the state in which the numerical value repeats increases and decreases every few days during January (Month).

The trend information shown in B of FIG. 12 indicates annual trend information. This annual trend information includes, for example, a bar graph with monthly information on the horizontal axis and the number of orders on the vertical axis, and a line graph with reaction information (Re), information on the intake time of cooked products (Ti), and the proportion of leftover food (Rp). It is shown. The number of orders also shows the number of new orders (NOr) and the number of reorders (ROr). The annual trend information is, for example, trend information for a predetermined period of time, and shows the monthly trends in the number of visitors and the number of orders. Using this annual trend information, it is possible to grasp the number of visitors, the number of orders, and the changes in customer evaluations of cooked products, depending on seasonal changes related to the month, for example.

<Customer management processing>
FIG. 13 shows an example of a customer management processing procedure according to the first embodiment. This processing procedure is an example of the program or customer management method of the present invention, and the present invention is not limited to this processing content or processing procedure. S indicates a step, and the number attached to S indicates the order of the steps (the same applies hereinafter). In addition, the processing procedure shown in this flowchart is continuously executed in the customer management system 2, and is not limited to the case where the process is performed according to the process number. Processing takes place.
This processing procedure involves taking images of customers, collecting food and drink images when serving food, during and after eating and drinking, and images of cooked products, and using the captured images to provide information about the food and visitors. This includes extraction and recording, calculation of trend information based on extracted information, and presentation processing of this trend information.

The management server 6 activates the cameras 4-1, 4-2, 4-3, 4-4, . . . , 4-N by activating the customer management system 2. The management server 6 causes the camera 4-X to photograph the visitor P when the visitor P is seated at a predetermined seat (address) on the customer seating floor 8, and obtains an image of the visitor (S101). The management server 6 then stores the acquired visitor image in the visitor information file 80 of the memory unit 52 (S102).
The management server 6 performs authentication using, for example, the acquired store visitor image (S103). In the store visitor authentication process, for example, it is determined whether the acquired store visitor image and the store visitor image stored in the store visitor information file 80 of the memory unit 52 match. Through this authentication process, it is determined whether the visitor P is a new visitor or not as attribute information (S104), and if the visitor is a new visitor (YES in S104), a new visitor information file 80 is stored in the memory unit 52. Create (S105). At this time, the management server 6 not only registers the visitor image of the new visitor P, but also provides time information acquired from the clock function installed in the management server 6, the external function section, etc. to the store entry section 83-1. Record.
Further, if the store visitor P is a repeat visitor (NO in S104), the management server 6 reads the store visitor information file 80 stored in the memory unit 52, for example. At this time, the management server 6 may read out past order information registered with the visitor P, eating and drinking times for the cooked food, reaction information on the cooked food, and the like.

When the management server 6 acquires the order information of the customer P (S106), the management server 6 notifies the information processing terminals 22-1 and 22-2, which are used by the galley and floor staff, that an order has been placed and its contents, for example. do it.
The cameras 4-1, 4-2, . . . , 4-N start capturing food and drink images when the ordered cooked food is served (S107). At this time, the cameras 4-1, 4-2, . . . , 4-N also capture an image including the facial expression of the visitor P just before eating or drinking, and an image of the cooked food. Then, the management server 6 causes the cameras 4-1, 4-2, ... 4-N to photograph the eating and drinking status, confirms that the eating and drinking for each ordered cooked item is finished, and then An image or an image of the cooked product is acquired (S108). At this time, whether or not eating and drinking is finished is determined by, for example, the photographed image showing that the cooked food has been completely eaten, or when a certain amount of time has passed without the customer eating or drinking, or when the floor staff has lowered the cooked food. do it.

The management server 6 stores the acquired visitor attribute information of the visitor P, food and drink images of the visitor P at the beginning and end of eating, intake time information measured by a timer, stay time information, images of ordered cooked products, and remaining quantity information. , reaction information is recorded in each file of the database (S109). As described above, the remaining amount information may be calculated by comparing the images of the cooked product before and after eating and drinking to calculate the remaining amount of the cooked product or its ratio. The reaction information is calculated by evaluation processing using, for example, remaining amount information, eating and drinking time information, and information on changes in facial expressions of visitors extracted from the eating and drinking screen. In addition, the management server 6 uses information recorded in the general information file 60, order information file 70, visitor information file 80, etc. to determine the trend in the number of orders for cooked products, the number of new orders (NOr), or the number of reorders. The ratio of (ROr) and its transition state are calculated. In addition, the management server 6 may calculate reaction information (Re), intake time information (Ti), trends in leftover food ratio (Rp), and the like. In addition, it is also possible to calculate the transition status of the service content provided by the store, such as the transition of time from placing an order until the food is served.

The management server 6 stores the calculated trend information in the memory unit 52 and presents it to the monitor 18 and the information processing terminals 22-1 and 22-2 (S110). At this time, the management server 6 may, for example, display all of the calculated trend information, or may select the information to be displayed depending on the authority information of the store clerk who views this information, the type of information processing terminal that views the information, etc. good. In other words, for example, when presenting trend information to a store employee who serves customer P, the management server 6 uses information that can be used for customer service such as information on cooked products that customer P has ordered in the past, information on the month and day of the week, etc. Information such as the number of cooked items may be selected and displayed. By narrowing down the number of displayed information in this way, viewers of trend information are prevented from determining whether or not the information is unnecessary for their work, and from being confused by information that cannot be used.

<Effects of the first embodiment>
According to the first embodiment, one of the following effects can be obtained.
(1) Using the acquired images of the customer eating and drinking and the images of the cooked products, information on the reaction of the visitor P to the cooked products, intake time, and remaining amount information is extracted, and trends are calculated using this information. By using this information, it is possible to obtain evaluation information on the prepared foods and services offered at the store, and improve customer service to customers.
(2) By using images of food and drinks and images of prepared foods eaten by customers, it is possible to obtain information on reactions of customers to prepared foods and store services more directly than through methods such as questionnaires to customers, and to attract new customers. You can get detailed information on attracting customers, such as opening new stores and retaining customers.
(3) Based on the information obtained from the trend information, it is possible to provide services such as suggesting cooking products, and it is possible to improve customer service to customers without relying on the customer service experience of employees.

[Second embodiment]
<Customer management system 100>
FIG. 14 shows a customer management system according to the second embodiment. The configuration shown in FIG. 14 is an example, and the present invention is not limited to such a configuration. In FIG. 14, the same components as in FIGS. 1 and 2 are denoted by the same reference numerals.
The customer management system 100 according to the second embodiment acquires images of food and drinks of the customer P and images of ordered cooked products in the first embodiment, and also acquires store visit information of the customer P and visits to the store at the time of checkout. The trend information calculation process is performed by adding the user image.
Therefore, in this customer management system 100, for example, as shown in FIG. 14, an image of a visitor visiting the store is acquired by the photographing unit 4 and transmitted to the management server 6 side. The management server 6 stores the store visit image in the order information file 70 and the store visitor information file 80 of the storage unit 26.
For example, the management server 6 may extract past store visit information by using images of visitors acquired when visiting the store, and display this information on the management terminal 22. This allows the floor staff to perform customer service using the displayed visitor information and trend information calculated or extracted from this information.
Other than that, photographing of food and drink images may be performed in the same manner as in the first embodiment.

<About the status of store 102>
FIG. 15 shows an example of the arrangement of shooting ranges and address information within a store.
The store 102 includes, for example, a seating floor 8, an entrance floor 104, an office 108, a cooking room 110, a pantry 112, and the like. A cash register counter 106 is arranged on the entrance/exit floor 104. In the customer seating floor 8, a passageway is provided between the tables 12-1, 12-2, . . . , 12-6 as a flow line area for store visitors P and store staff.
On the audience floor 8, for example, cameras 4-1, 4-2, ..., 4-10 are arranged on the ceiling or wall, and tables 12-1, 12-2, ..., 12- In addition to photographing the visitor P arriving at the restaurant 6 and the cooked food, the travel route L traveled by the visitor P is tracked and photographed.
The management server 6 sets the XY coordinates within the audience floor 8, and from the images taken by the cameras 4-1, 4-2, . . . , 4-10, tables 12-1, 12-2, . ..., the location of 12-6 can be converted into address information and understood. Similarly, the location of the food on the table 12 can also be converted into address information and obtained.
A camera 4-20 is installed on the entrance/exit floor 104, and captures facial images of visitors P when they arrive at the store, when they are waiting to be guided to the customer seating floor 8, or when they pay.
A management server 6 is installed in the office 108. The management terminal 22 may also be installed in the cooking room 110 and the pantry 112.

Further, the camera 4-20 photographs the visitor P who is inside the entrance/exit floor 104, as shown in FIG. 16, for example. The camera 4-20 photographs the state of the visitor P when he or she enters the store, is greeted at the entrance floor 104 of the store 102, and when exits (pays). This allows the camera 4-1 to capture an image of the visitor P including his or her facial expressions at the timing of entering and exiting the store.
The management server 6 can extract information on reactions to prepared foods and services provided at the store based on, for example, changes in the facial expressions of the visitor P when entering and leaving the store, and facial expressions when paying. The management server 6 may store the extracted reaction information in the trend information file 90 of the storage unit 26.
Furthermore, the management server 6 may extract the length of stay from the time of arrival to the time of leaving the store from the image of the visitor, and create a length of stay information file as a database for calculating trend information. The stay time information file includes, for example, information on visitors related to the stay time, ordered prepared items, etc., information on the stay time from the time of entering the store to the time of leaving the store, and whether the visitor is a new visitor or a repeat visitor. Information etc. may also be stored. The information in the visitor's stay time information file may be used, for example, to calculate trend information of new or repeat visitors with respect to the overall number of visitors, based on the stay time. .

<Customer management processing>
FIG. 17 shows an example of a customer management processing procedure according to the second embodiment. This processing procedure is an example of the program or customer management method of the present invention, and the present invention is not limited to this processing content or processing procedure.
This processing procedure includes collecting visitor information including images of visitors upon visiting the store, authenticating whether the visitor is a new visitor or a repeat visitor, obtaining order information of the visitor, obtaining images of food and beverages and cooked products, and so on. This includes calculation of trend information based on store visitor information, and presentation processing of store visitor information and trend information.

The management server 6 activates the cameras 4-1, 4-2, 4-3, 4-4, . . . , 4-N by activating the customer management system 2. When a visitor P enters the entrance floor 104, the camera 4-20 photographs the visitor P, and the cameras 4-1, 4-2, 4-3, 4-4. ... 4-N (S201), and the management server 6 acquires the image of the visitor (S202). At this time, the management server 6 tracks and photographs the moving store visitor P by linking a plurality of cameras 4-1, 4-2, . . . , 4-N, for example. The management server 6 stores, for example, images of the store visitor P taken by the camera 4-20 on the entrance/exit floor 104, and images of the store visitor P taken by the cameras 4-1, 4-2, . . . , on the customer seating floor 8. The degree of matching with the image may be determined, or whether the visitor P is the same visitor P may be determined based on monitoring results such as the travel route of the visitor.
At this time, the management server 6 may always capture images of the store visitors photographed by the cameras 4-1, 4-2, .

The management server 6 stores the acquired visitor image in the general information file 60, the visitor information file 80, etc. of the memory unit 52 (S203). For example, when the management server 6 confirms that a new visitor P enters or exits the store or starts photographing with the camera 4-20, the management server 6 reads the visitor information file 80 from the memory section 52 and stores a new number in the number section 81. Register and generate management data. In addition, the management server 6 transmits time information obtained from, for example, a clock function installed in the management server 6 or an external function unit to the store entry time section 83-1 when registering a visitor image of a new visitor P. input.

In this customer management process, the processes of authenticating the visitor P using the visitor image (S204), determining whether the visitor is a new visitor or a repeat visitor (S205), and creating a visitor information file (S206) are performed in the steps shown in FIG. Processing similar to S103 to S105 may be performed.

When the management server 6 acquires the order information of the customer P (S207), the management server 6 notifies the information processing terminals 22-1 and 22-2, which are used by the galley and floor staff, that an order has been placed and its contents, for example. do. The management server 6 takes an image of the visitor P and the cooked food upon serving the ordered cooked food, and acquires the image (S208). The cameras 4-1, 4-2, . . . , 4-N may always take pictures of the visitor P eating and drinking the cooked food, or may take pictures periodically.
The management server 6 causes the cameras 4-1, 4-2, ... 4-N to photograph the state of eating and drinking, confirms that the eating and drinking of each ordered cooked item has been completed, and displays an image of the visitor P at the time of completion. An image of the cooked food is acquired (S209).
Further, the management server 6 causes the camera 4-20 to take an image of the face and entire body of the visitor P who is paying, for example, at the entrance/exit floor 104 to obtain a visitor image (S210).
The management server 6 then stores visitor attributes, the acquired eating and drinking images of visitor P at the beginning and end of eating, intake time information measured by a timer, stay time information, images of ordered cooked products, remaining amount information, and reaction information. In addition, information such as whether the visitor P is a new visitor or a repeat visitor, age, gender, and other attribute information, the number of visitors, and an image of the visitor at the time of payment is recorded in each file of the database.

After recording each piece of information acquired from the image, the management server 6 uses the information recorded in the general information file 60, order information file 70, visitor information file 80, etc. In addition to the ratio of the number of orders (NOr) or the number of reorders (ROr) and their trends, reaction information (Re), intake time information (Ti), and leftover food ratio (Rp), changes in the length of stay of visitors. Or calculate trend information including changes in the number of store visitors.
In calculating trend information, for example, the total number of store visitors Npm, the number of new store visitors Nnm, the number of returning store visitors Nrm, the ratio (%) of the number of new store visitors Nnm to the total number of store visitors Npm, and the number of repeat store visitors to the total number of store visitors Npm. The ratio (%) of the number of people Nrm may be calculated.

The management server 6 stores the calculated trend information in the memory unit 52 and presents it to the monitor 18 and information processing terminals 22-1 and 22-2 (S212). At this time, the management server 6 may display all or part of the trend information corresponding to the presenting information processing terminals 22-1, 22-2 or the staff in charge.

<Effects of the second embodiment>
According to such a configuration, in addition to the effects of the first embodiment, one of the following effects can be obtained.
(1) Obtain attributes, length of stay, and number of visitors using acquired images of visitors and prepared food images, and use visitor trend information calculated using this information. This will improve customer service to store visitors.
(2) Utilize the preferences of visitors or evaluations from other visitors by using trend information on the visitor's attributes, the type of food prepared, or the number of ordered dishes calculated in relation to the visitor's attributes. You can present the services that you have provided.
(3) By using trend information calculated by linking store visitor attributes, it is possible to understand store visitor trends and expand the range of sales strategies, such as providing menus and services that match these trends.
(4) Transition states such as the number of customers and the number of ordered cooked items can be used for setting the number of store staff and their arrangement, planning and analyzing the procurement of ingredients for cooked items, etc.
(5) Changes in trend information regarding the type and number of ordered prepared items can be used to adjust or change menus, replace seasonal menus, develop new menus, etc.
(6) By extracting facial expressions of visitors from images of them when they enter or leave the store, it is possible to obtain their unconscious reactions to the prepared foods and services provided.

[Third embodiment]
FIG. 18 shows a customer management system according to the third embodiment. The configuration shown in FIG. 18 is an example, and the present invention is not limited to such a configuration. In FIG. 18, the same components as in FIGS. 1 and 14 are designated by the same reference numerals.
In the customer management system 120 according to the third embodiment, for example, as shown in FIG. When a plurality of people visit the store, reaction information is evaluated according to the number of people visiting the store, and trend information is calculated using the corrected reaction information. In other words, reactions extracted from eating and drinking images are influenced by the emotions of visitors. Therefore, even if a visitor eats the same prepared food, their facial expressions will change differently depending on whether they are eating or drinking alone or when they are eating and drinking while talking with a group of people.
Therefore, in this customer management system 120, evaluation correction processing is performed when extracting reaction information according to the dining environment and conditions of the visitor P.

In the customer management system 120, for example, one or more photographing units 4 are used to photograph customer images and eating and drinking images of the customers P1, P2, and P3 to be monitored who are seated at the same table 12. The customer management system 120 also determines whether the visitors P1, P2, and P3 are visiting the store in the same group. This determination may be made using, for example, whether or not the customer is having a conversation while seated, an image taken when the customer enters the store, and other information input by the floor staff who served the customer.
For example, as shown in FIG. 19, the storage unit 26 of the management server 6 stores an evaluation table 122, and when it is determined that visitors P1, P2, and P3 are in the same group, the evaluation table 122 is read out. The management server 6 then corrects the evaluation using the evaluation table 122 for the evaluation information when extracting reaction information from the reactions of the visitors included in the acquired eating and drinking images.
The evaluation table 122 includes, for example, the number of visitors and evaluation weighting information. The number of visitors is divided into, for example, 1 person, 2 to 3 people, and 3 or more people. Then, the weighting information is set to "0" when there are two or three people, and the change in facial expression extracted from the image is evaluated as is, and when there is only one person, a predetermined value is set for the change in facial expression extracted from the image. It is sufficient to evaluate by adding a weight of "+α". Furthermore, if there are three or more people, the changes in facial expressions extracted from the image can be evaluated by adding a predetermined weight of "-β". This weighting can be applied, for example, as the number of people in the group increases. It may be difficult to evaluate changes in facial expressions due to conversation and changes in facial expressions due to eating and drinking. Therefore, if there are many people, reaction information is extracted by lowering the evaluation value. On the other hand, if the person is alone, the evaluation value is increased and reaction information is extracted, assuming that the person's facial expression does not change easily due to eating and drinking.
Note that the weighting determination method and weighting values shown here are merely examples. For example, the weighting for one person may be set to 0, and the weighting of the evaluation may be lowered as the number of people in the group increases.

<Customer management processing>
FIG. 20 shows an example of a customer management processing procedure according to the third embodiment. This processing procedure is an example of the program or customer management method of the present invention, and the present invention is not limited to this processing content or processing procedure.
This processing procedure includes collecting visitor information, authenticating whether the visitor is a new visitor or a repeat visitor, obtaining order information of the visitor, obtaining images of food and beverages and prepared products, correcting reaction information, and processing based on visitor information. This includes calculating trend information and presenting visitor information and trend information.

When the store visitor P1 is seated at a predetermined seat (address) on the customer seating floor 8, the management server 6 photographs the store visitor P1 using the photographing unit 4, and obtains a store visitor image (S301). At this time, the number of people visiting the store together with the customer P1 is confirmed (S302), and the image of the customers is stored in the store visitor information file 80 of the memory unit 52 together with the number of people information (S303). At this time, the management server 6 may acquire the visitor images of the other visitors P2 and P3 who came to the store together with the visitor P1, and perform customer management processing.

In this customer management process, steps S304 to S309 in which authentication and other processes are performed for the visitor P based on the visitor's image may be performed, for example, similar to the processes shown in steps S204 to S209 shown in FIG. 17.

The management server 6 reads the evaluation table 122 in the memory section 52 of the storage section 26 (S310), and performs corrections using the evaluation table 122 in the process of extracting reaction information for the acquired eating and drinking images (S311).
In this modification process, the management server 6 reads weighting information from the evaluation table 122, for example, regarding the customer P1 who is the target of the evaluation process, according to the number of people who visited the store together. As a result, the management server 6 extracts reaction information by applying weighting to the degree of change in the facial expression of the visitor P1 before and after eating and drinking. At this time, for example, if there are multiple store visitors P1 and there are few changes in their facial expressions, the management server 6 responds with a response indicating that the evaluation of store visitor P1 is "low" as a result of correction using the weighting information. Output information.
On the other hand, if there is only one visitor P1, the management server 6 evaluates the change in the facial expression of the visitor P1 as is, or as a result of slight correction, the management server 6 determines whether the evaluation of the visitor P1 is "normal" or not. Outputs reaction information indicating that is "good".
In modifying the reaction information, for example, the judgment level may be varied in response to changes in the overall facial expression, or the number of changed parts of the face, the magnitude of movement, etc. may be individually modified and evaluated. .
Note that the weighting information used in this correction process is not limited to one that makes the evaluation based on reaction information stricter as the number of visitors increases, but on the contrary, it can be used to soften the evaluation according to the number of visitors. Good too.

The management server 6 stores the acquired eating and drinking images of visitor P at the beginning and end of eating, intake time information measured by a timer, stay time information, ordered cooked product images, remaining amount information, and reaction information in each file of the database. Record (S312).
The management server 6 uses the information recorded in the general information file 60, the order information file 70, the customer information file 80, etc. to monitor the trends in the number of orders for cooked products, new orders (NOr), or reorders (ROr). Calculate the ratio and its transition status. In addition, the management server 6 may calculate reaction information (Re), sensation at time of intake (Ti), trends in leftover food ratio (Rp), and the like. In addition, it is also possible to calculate the transition status of the service content provided by the store, such as the transition of time from placing an order until the food is served.

The management server 6 stores the calculated trend information in the memory unit 52 and presents it to the monitor 18 and information processing terminals 22-1 and 22-2 (S313).

<Effects of the third embodiment>
According to such a configuration, in addition to the first implementation, any of the following effects can be obtained.
(1) By correcting reaction information according to the number of visitors, it is possible to improve the accuracy of visitor evaluation information extracted from eating and drinking images.
(2) By storing the information on the number of customers in the store visitor information and creating trend information of cooked food orders that includes the number of customers, it is possible to present prepared food according to the number of customers and to accommodate large numbers of customers. It can be used as information for developing corresponding menus.
(3) Using the information on the number of customers, it is possible to understand trends in the number of customers by day of the week, time of day, or month, which can be used to predict trends in the number of customers, and to ensure appropriate staff allocation. It can be used to secure the number of people.

[Fourth embodiment]
FIG. 21 shows a customer management system according to the fourth embodiment. The configuration shown in FIG. 21 is an example, and the present invention is not limited to such a configuration. In FIG. 21, the same components as in FIG. 1, FIG. 2, etc. are denoted by the same reference numerals.
In the customer management system 130 according to the fourth embodiment, for example, as shown in FIG. A motion image of a visitor P while eating and drinking is acquired. This operation image shows whether or not the visitor P uses the seasonings placed on the table 12 before or during eating and drinking, and monitors the amount used. The customer management system 130 then stores the seasoning usage information as reaction information or evaluation information of visitors to the cooked product.
The storage unit 26 of the management server 6 stores motion information 132 extracted from motion images.

For example, as shown in FIG. 22, the photographing unit 4 photographs an image 134 including the face of the visitor P just before eating the cooked product and the cooked product. The photographing unit 4 may start photographing, for example, at the timing when the cooked food is served, and photograph the image when the visitor P starts eating and drinking the cooked food, or when the visitor P starts eating and drinking the cooked food. That is, in the customer management system 130, eating and drinking images may be used as the action images.
For example, in the analysis processing unit 28, the management server 6 extracts the arms and hands of the visitor P shown in the food and drink image 32, or the image portion 136 that includes seasonings, and determines whether seasonings are used or not. Analyze quantity. The image portion 136 can be extracted by detecting, for example, a movement of an active part other than the face or a movement that identifies the use of a seasoning. Then, the analysis processing unit 28 extracts the amount of seasoning used based on the size of the movement, the number of movements, etc. Thereby, the management server 6 acquires reaction information or evaluation information representing the preferences of the store visitor P, such as "the taste is bland" or "the taste is not enough", for the prepared food provided at the store.

<Trend information>
FIG. 23 shows an example of trend information calculated based on eating and drinking images and movement images of visitor P. The trend information shown in FIG. 23 is assumed information for representing the transition status of an image of the store visitor P, information ordered by the store visitor P, etc., and the present invention is not limited to such content.
The trend information shown in A of FIG. 23 is, for example, monthly trend information for each ordered cooked item. This monthly trend information includes seasoning usage rate information (Su) in addition to the various trend information shown in A of FIG. This seasoning usage rate information (Su) indicates, for example, the ratio of the number of times the seasoning is used to the number of orders for cooked product A, or the ratio of the number of customers P who use a certain amount or more of the seasoning. . This information shows, for example, trends in seasoning usage rates during the month of January, allowing us to understand the differences in customer preferences between weekdays and weekends, as well as the demographics of customers who visit the store at the beginning and end of the month. You can understand the differences in people's tastes and preferences.

The trend information shown in B of FIG. 23 indicates annual trend information. This annual trend information includes seasoning usage rate information (Su) in addition to the various trend information shown in A of FIG. The annual trend information allows, for example, for cooked product A, changes in the number of orders from month to month and season to season, as well as trends in customer preferences. In other words, it is possible to grasp changing trends such as increases and decreases in seasonings during hot or cold seasons.

<Customer management processing>
FIG. 24 shows an example of a customer management processing procedure according to the fourth embodiment. This processing procedure is an example of the program or customer management method of the present invention, and the present invention is not limited to this processing content or processing procedure.
This processing procedure includes taking images of visitors, collecting food and drink images when serving cooked food, while eating and drinking, and at the end of eating and drinking, movement images of visitor P just before eating and while eating, and images of cooked products. This includes extracting and recording information on cooked products and customers using images, calculating trend information based on the extracted information, and processing images for presenting this trend information.

By activating the customer management system 2, the management server 6 activates the cameras 4-1, 4-2, 4-3, 4-4, . Photograph is taken (S401). In this customer management process, steps S402 to S407, which represent processes such as acquiring a store visitor image and authenticating the store visitor P, may be performed in the same manner as steps S101 to S106 in FIG. 13, for example.
The management server 6 acquires an image of the visitor P and the cooked product immediately before the food is served (S408), and also acquires an operational image of the visitor P before eating and drinking (S409).
The management server 6 then stores, for example, visitor attribute information, acquired food and drink images of visitor P at the beginning and end of eating, movement images for cooked products, intake time information measured by a timer, stay time information, and ordered cooked product images. , remaining amount information, and reaction information are recorded in each file of the database (S410).
The management server 6 uses the information recorded in the general information file 60, the order information file 70, the customer information file 80, etc., as well as the operation information 132 stored in the memory section 52 of the storage section 26, to order cooked products. The ratio of the number of new orders (NOr) or the number of reorders (ROr) and the state of their changes are calculated. In addition, the management server 6 may calculate reaction information (Re), intake time information (Ti), trends in leftover food ratio (Rp), seasoning usage rate information (Su), and the like. In addition, it is also possible to calculate the transition status of the service content provided by the store, such as the transition of time from placing an order until the food is served.

The management server 6 stores the calculated trend information in the memory unit 52 and presents it to the monitor 18 and information processing terminals 22-1 and 22-2 (S411).

<Effects of the fourth embodiment>
(1) In the customer management process, by extracting the usage status of seasonings from the movement information of visitor P regarding food and drinks, the customer It is possible to provide services that predict the preferences of people.
(2) Since computers can predict cooking adjustment processes that match trends in the use of seasonings, etc., it is possible to realize cooking adjustments such as seasonings that match customer preferences, regardless of the chef's experience.
(3) In customer management, it is possible to predict purchasing and seasoning management processing in response to seasons and times when seasonings tend to be used more, and it is possible to reduce the burden on store staff.

[Other embodiments]
Modifications of the embodiment described above are listed below.

(1) In this customer management system 2, some of the calculated trend information or Information obtained by processing this trend information may be calculated and presented to the information processing terminals 22-1, 22-2, etc.

(2) In the above embodiment, a case has been described in which information on whether the ordered cooked product is a staple food or a main dish is recorded as attribute information, but the invention is not limited to this. The attribute information may include, for example, information such as whether a specific cooked item was shared and eaten by multiple people. In this case, information on the number of people eating and drinking may be stored as related information to the cooked product intake time information.

(3) In the above embodiment, the camera 4-20, cameras 4-1, 4-2, 4-3, ..., 4-N of the photographing section 4 are used to acquire store visitor images. Although shown, it is not limited to this. In the customer management system 2, for example, an image taken by the store visitor P may be captured and used for authentication of the store visitor information. The customer management system 2 may include a functional unit that receives image information from, for example, a photographing device owned by the store visitor P or a portable information processing device equipped with a photographing function. In this case, to receive the image, for example, in addition to using a communication line such as a telephone line or the Internet line, short-range wireless communication such as infrared communication or Wi-Fi (registered trademark) may be used.

(4) In the above embodiment, a case has been described in which whether or not the visitor P uses seasonings and the amount used are extracted from the motion images taken immediately before eating or drinking or while eating and drinking, but the present invention is not limited to this. In the customer management system 2, sensors may be placed, for example, on each seasoning container placed on the table 12, or on a plate on which the container is placed. As a result, the customer management system 2 can, for example, determine whether or not seasonings are used based on operational images, and extract information detected by sensors to detect changes in the amount of seasonings used in the seasoning containers placed on each table. It may also be used as evaluation information for cooked products.

As described above, the most preferred embodiment of the present invention has been described. The present invention is not limited to the above description. Various modifications and changes can be made by those skilled in the art based on the gist of the invention described in the claims or disclosed in the detailed description. It goes without saying that such modifications and changes are included within the scope of the present invention.

According to the customer management system, program, device, and method of the present invention, information on a visitor's reaction to a cooked product, intake time information, and remaining amount of cooked product after consumption is extracted from images of the visitor eating and drinking and images of the cooked product. However, by calculating trend information including the preferences of store visitors, it is useful because it can be used for customer attraction information such as cultivating new customers and retaining customers.

2, 100, 120, 130 Customer management system 4 Photography department 4-1, 4-2, 4-3,...4-N Camera 6 Management server 8 Audience floor 10 Seats 12, 12-1, 12-2, 12-3, 12-4, 12-5, 12-6 Table 14 Cooked product 16 Server body 18 Monitor 20 Operation input section 22 Management terminal 22-1, 22-2 Information processing terminal 24 Processing section 26 Storage section 28 Analysis processing Part 30 Trend information calculation processing part 32 Eating and drinking image 34 Face part 36 Analysis image 38 Small area 40, 42 Cooked product image 43 Analysis image 44 Eating and drinking part 46 Leftover part 50 Processor 52 Memory part 54 Communication part 56 Timer 58 Input/output part ( I/O)
60 General information file 61 Number section 62 Date and time section 63 Food and beverage image section 64 Visitor attributes section 64-1 New visitor section 64-2 Repeat visitor section 65 Time information section 65-1 Entering section 65-2 When leaving the store Section 65-3 Residence time section 65-4 Eating and drinking time section 66 Ordered food section 66-1 Cooked product image section at the time of serving 66-2 Cooked product image section at the end of eating and drinking 66-3 Cooked product section 66-4 Number of items 66-5 Leftovers ratio 67 Customers 67-1 Total number of customers 67-2 Number of new customers 67-3 Number of repeat customers 70, 70-1, 70-2,... 70-N Order information file 71 Number section 72 Date and time section 73 Visitor information section 73-1 Visitor image section 73-2 Attribute information section 73-3 Number of people section 74 Evaluation information section 74-1 Food and drink image section 74-2 Operation image Section 74-3 Attribute information section 75 Time information section 75-1 Order time section 75-2 Serving time section 75-3 Eating and drinking start time section 75-4 Eating and drinking end time section 75-5 Eating and drinking time section 80, 80-1, 80 -2,...80-N Visitor information file 81 Number section 82 Visitor information section 82-1 Visitor image section 82-2 Attribute information section 82-3 Number of people section 83 Time information section 83-1 Store entry section 83-2 Leaving time section 83-3 Staying time section 84 Ordered cooked products section 84-1 Cooked product image section 84-2 Eating and drinking image section 84-3 Cooked products section 84-4 Number of items section 84-5 Remaining food ratio section 90 Trend information file 91 Number section 92 Time information section 92-1 Date and time information 92-2 Period information 93 Order cooked product information section 93-1 Cooked product section 93-2 Order number section 93-3 Leftover food ratio section 93-4 Eating and drinking time Department 93-5 Reaction information department 93-6 Reorder information department 93-7 Evaluation information department 102 Store 104 Entrance floor 106 Cash register counter 108 Office 110 Galley 112 Pantry 114 Accounting terminal 122 Evaluation table 132 Operation information 134 Image

Claims (9)

a photographing unit that acquires an eating and drinking image including at least a face image of a cooked product to be evaluated and a face image of a visitor who consumes the cooked product, and a table image of the visitor who consumes the cooked product ;
along with reaction information including information on a change in facial expression extracted from a part or the whole of the facial image of the visitor when ingesting the prepared food extracted from the food and drink image, or information determined based on the change information; Analyze either or both of the intake time information and the remaining cooked product information after consumption, and analyze the number of customers including at least other customers who came to the store with the customer from the table image, and calculate the number of customers. reads an evaluation table storing correction information for correcting the evaluation of the reaction information according to the evaluation table, corrects the reaction information according to the evaluation table, and includes at least the preference tendency of the store visitor according to the reaction information . a processing unit that calculates trend information;
A customer management system characterized by comprising: The customer management system according to claim 1, further comprising an information presentation unit that presents at least the trend information by acquiring the eating and drinking images. Further, a storage unit that stores one or more of the food and beverage images, the reaction information, the intake time information, the cooked product remaining amount information, and the trend information in relation to the attributes of visitors;
The customer management system according to claim 1, further comprising: The customer management system according to claim 1, wherein the processing unit calculates the trend information at least for each visitor or for each cooked item. Furthermore, the photographing unit acquires an operational image of the visitor before or during the intake of the prepared food,
The customer management system according to any one of claims 1 to 4, wherein the processing unit calculates the trend information by analyzing customer evaluation information regarding the cooked products based on the operation image. Further, the processing unit updates the trend information regularly or irregularly after acquiring the eating and drinking images, customer management according to any one of claims 1 to 5. system. A function for a photographing unit to acquire a food and drink image including at least a face image of a cooked product to be evaluated and a face image of a visitor who consumes the cooked product, and a table image of the visitor who consumes the cooked product ;
along with reaction information including information on a change in facial expression extracted from a part or the whole of the facial image of the visitor when ingesting the prepared food extracted from the food and drink image, or information determined based on the change information; Analyze either or both of the intake time information and the remaining cooked product information after consumption, and analyze the number of customers including at least other customers who came to the store with the customer from the table image, and calculate the number of customers. reads an evaluation table storing correction information for correcting the evaluation of the reaction information according to the evaluation table, corrects the reaction information according to the evaluation table, and includes at least the preference tendency of the store visitor according to the reaction information . A function to calculate trend information,
A program for realizing this on a computer . a photographing unit that acquires an eating and drinking image including at least a face image of a cooked product to be evaluated and a face image of a visitor who consumes the cooked product, and a table image of the visitor who consumes the cooked product ;
along with reaction information including information on a change in facial expression extracted from a part or the whole of the facial image of the visitor when ingesting the prepared food extracted from the food and drink image, or information determined based on the change information; Analyze either or both of the intake time information and the remaining cooked product information after consumption, and analyze the number of customers including at least other customers who came to the store with the customer from the table image, and calculate the number of customers. reads an evaluation table storing correction information for correcting the evaluation of the reaction information according to the evaluation table, corrects the reaction information according to the evaluation table, and includes at least the preference tendency of the store visitor according to the reaction information . a processing unit that calculates trend information;
A customer management device comprising: a step of acquiring, in a photographing unit, an eating and drinking image including at least a face image of a cooked product to be evaluated and a face image of a visitor who consumes the cooked product, and a table image of the visitor who consumes the cooked product ;
along with reaction information including information on a change in facial expression extracted from a part or the whole of the facial image of the visitor when ingesting the prepared food extracted from the food and drink image, or information determined based on the change information; Analyze either or both of the intake time information and the remaining cooked product information after consumption, and analyze the number of customers including at least other customers who came to the store with the customer from the table image, and calculate the number of customers. reads an evaluation table storing correction information for correcting the evaluation of the reaction information according to the evaluation table, corrects the reaction information according to the evaluation table, and includes at least the preference tendency of the store visitor according to the reaction information . A process of calculating trend information in a processing unit,
A customer management method characterized by comprising:

Images