JP6842037B2 - Information processing device, control method of information processing device, and program - Google Patents

Description

This onset Ming, the information processing apparatus, a method of controlling an information processing apparatus, and a program.

In recent years, with the development of image analysis technology, it has become possible to provide a mechanism for quickly extracting consumables to be replenished from acquired images and appropriately notifying them.

For example, Patent Document 1 proposes a mounting device capable of notifying a user of a decrease in the remaining amount of general-purpose consumables without incurring a cost. Then, by using this method, for example, it becomes possible to identify whether or not the remaining amount of the shampoo bottle is below a predetermined level by image analysis.

If such a mechanism is applied to restaurants, it can be expected that business efficiency and services will be improved. For example, if you install a camera (hereinafter, "imaging device") that can take still images and videos (hereinafter, "images") of your glasses and cups, this image will be analyzed and the remaining amount of drink will be small. You can easily identify the glass that has become. And if you can serve food and drink such as waiters and waitresses with this information, or notify the staff who receive orders from guests (hereinafter, "cast"), you can quickly pour cold substitutes or take drink orders without delay. You can go to. In addition, at Tonkatsu specialty stores that offer services such as "free refills for cabbage, rice, and miso soup," you will be able to quickly find plates that may need refills.

In FIG. 3, two guests 303 "A" and "B" are sitting side by side on one table 302, and three glasses 301 "α" "β" "γ" are lined up on the table 302. It is a figure. In such a case, if the above mechanism is used, it can be quickly detected that the remaining amount of drink in the β glass is low, so that the cast can easily respond quickly.

Japanese Unexamined Patent Publication No. 2015-162164

However, with the above mechanism, it is not possible to specify which of A and B is using the glass 301 "β" in FIG. Therefore, the cast who receives the notification cannot determine which guest 303 should be called, and may call the wrong person.
An object of the present invention is to easily manage the correspondence between a person and tableware.

The information processing device according to the present invention has the following configuration. That is, it is estimated whether the person picked up the tableware by using the acquisition means for acquiring the image data obtained by imaging the person and the tableware and the result of analyzing the acquired image data. It is characterized by having an estimation means for performing the estimation and a registration means for registering information in which the person and the tableware are associated with each other according to the result of the estimation.

According to the present invention, it is possible to easily manage the correspondence between a person and tableware.

It is a block diagram which showed an example of the system structure of this invention. It is a block diagram which showed an example of the structure of the hardware of this invention. It is a block diagram which showed an example of the state which the glass is placed on the table. FIG. 5 is a configuration diagram showing an example of a display screen displayed on the display unit 210 of the information processing device 120, the mobile terminal 130, and the robot 135. It is a block diagram which showed an example of the figure which saw the glass 301 from the upper surface and the side surface. It is a block diagram which showed an example of correspondence of estimated alcohol blood concentration 1323 and alcohol gauge 402. It is a block diagram which showed an example of the display screen displayed on the display part 210 of an information processing apparatus 120 and a mobile terminal 130. It is a block diagram which showed an example of the main flow of this invention. It is a block diagram which showed an example of the subflow of step S806 of this invention. It is a block diagram which showed an example of the subflow of step S807 of this invention of this invention. It is a block diagram which showed an example of the subflow of step S1011 of this invention. It is a block diagram which showed an example of the subflow of step S808 of this invention. It is a block diagram which showed an example of the data table used in the data flow processing of this invention. It is a block diagram which showed an example of a series of flow until the alcohol gauge 402 changes according to the amount of drinking of a guest by associating a guest with a glass. It is a block diagram which showed an example of the mechanism for identifying a glass 301. An example of a table showing the correspondence between the state of the glass and the condition of the history of eating and drinking of the guest, the character display of the alcohol gauge 402, the behavior pattern to be a guest, and the object to be displayed on the display unit 210 to notify it. It is a block diagram shown.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a configuration diagram showing an example of the system configuration of the present invention.

The present invention is an information processing system in which an image pickup device 110, an information processing device 120, a mobile terminal 130, and a robot 135 are communicably connected via a communication network 140.

The image pickup device 110 analyzes images / videos (still images / moving images are not limited) in order to identify the vessel to be observed in the present invention, the user of the vessel, the food / drink contained in the vessel, the amount thereof, and the like. , Hereinafter collectively referred to as an "image"). Therefore, as in the embodiment of FIG. 1, it does not necessarily have to be something like a surveillance camera, and for example, a form such that it is provided as a part of the camera function of the mobile terminal 130 or the robot 135 described later is also available. Conceivable.

The information processing device 120 is a device that acquires and manages an image captured by the image pickup device 110. In addition, the information processing device 120 can analyze the image and identify tableware such as cups and glasses in the image (general container for storing food and drink, hereinafter referred to as "glass" in this embodiment). Further, the information processing device 120 can analyze the image and identify the amount and type (item) of the food and drink (hereinafter referred to as "drink" in this embodiment) and the user of the device. In addition, the information processing device 120 analyzes the image to determine which item and how much drink the glass user drank, the time when he / she started eating and drinking, the time when he / she last put his / her mouth in the glass, and the number and frequency of drink refills. Etc. can be specified and the history can be managed. In addition, the information processing device 120 manages the options for responding to the guest, and can determine the response to the guest based on the usage status of the glass identified through image analysis and the history of the user. .. Further, the information regarding the decision to deal with the guest can be transmitted to the mobile terminal 130 and the robot 135. Further, when the information processing device 120 also serves as the mobile terminal 130 or the robot 135, the information processing device 120 can respond to the glass user based on the decision according to the decision.

Therefore, the information processing device 120 does not necessarily have to be an independent information processing device as in the embodiment of FIG. That is, the information processing device 120 may be in a form that also serves as a mobile terminal 130 or a robot 135, which will be described later.

The mobile terminal 130 is a device for displaying information and instructions based on decisions made by the information processing device 120. Further, the mobile terminal 130 itself may be provided so that the response to the guest can be determined based on the usage status of the glass and the history of the user.

The first form of the mobile terminal 130 is intended to be carried mainly by a person engaged in some service related to the above-mentioned food and drink vessels. Specifically, for example, the cast 150 of a restaurant or a wedding hall. Carry. Specific examples of the mobile terminal 130 include tablet terminals, smartphones, wearable terminals (glass-type, watch-type, and other mobile terminals).

A second form of the mobile terminal 130 is a tablet-type terminal that is placed on a table and accepts guest food and drink orders. A mechanism is conceivable to actively pick up orders based on the guest's eating and drinking history. Specifically, for example, if you know that the guest is ordering a pot and detect that the amount of ingredients in the pot is less than the specified amount, you will see "Uncle, set, and udon set. How about it? ”, You can think of a way to use it, such as setting up a place to talk to guests. (It may be better to call out to a person who frequently orders (= judged to be the secretary).) Such usage is the same for the robot 135, which can serve as a substitute for the cast below.

The robot 135 is a robot capable of performing the work of the cast 150. Specifically, based on the information and instructions based on the decision made by the information processing apparatus 120, a service of accepting an additional order from the user of the vessel and serving food and drink is provided. Further, similarly to the mobile terminal 130, the robot 135 may be provided so that the response to the guest can be determined based on the usage status of the glass and the history of the user. Further, the robot 135 may be provided with a mobile terminal 130 for showing to the guest.

In any case, the robot 135 is expected to have various functions and forms according to its use and environment. For example, it may be installed on a desktop to serve the role of the image pickup device 110 or to accept an additional order from the user of the device, or it may be a humanoid life-size and movable / movable. , It may be something that can handle orders for food and drink and serve. Further, the mobile terminal 130 and the robot 135 may be in a form in which only one of them is used or in a form in which both are used according to the intended use.

The communication network 140 is a network to which the image pickup device 110, the information processing device 120, the mobile terminal 130, and the robot 135 are connected so as to be able to communicate with each other. Various existing technologies can be used as the communication method. It may be wired or wireless, and may be WAN (Wide Area Network) or LAN (Local Area Network).

FIG. 2 is a configuration diagram showing an example of the configuration of the hardware of the present invention. Specifically, the hardware corresponds to the image pickup device 110, the information processing device 120, the mobile terminal 130, and the robot 135.

Reference numeral 201 denotes a CPU, which controls each device connected to the system bus 204 by loading the program stored in the program ROM of the ROM 202 or the external memory 211 into the RAM 203 and executing the program. Further, font data and the like are stored in the font ROM of the ROM 202, and various data are stored in the data ROM of the ROM 202. Reference numeral 203 denotes a RAM, which functions as a main memory, a work area, and the like of the CPU 201. Reference numeral 202 denotes a keyboard controller (input controller), which controls input from 209 or a pointing device (not shown), for example, a touch pad.

The input controller 205 controls the input device 209. The input device 209 also includes a touch panel. In that case, the input device 209 may also serve as a display unit 210 (a display device for displaying a display screen for displaying an image). Reference numeral 206 denotes a video card (VC), which controls the display of the display unit 210.

Reference numeral 207 is a memory controller, which controls access to an external memory 211 (hard disk or the like). Reference numeral 208 denotes a network interface card (NIC), which controls communication on the communication network 140.

The camera controller 220 controls the camera unit 221 that captures an image. The camera controller 220 is particularly provided in the image pickup device 110, the mobile terminal 130, and the robot 135, and may be provided in the information processing device 120. Specifically, the camera unit 221 may be, for example, a camera portion of the image pickup device 110, a camera portion of the mobile terminal 130, or a camera portion of the robot 135.

The motion control controller 230 controls to physically drive the drive unit 231. The motion control controller 230 is particularly provided in the image pickup device 110 and the robot 135, and may be provided in the information processing device 120 and the mobile terminal 130. Specifically, the drive unit 231 may be, for example, a motor or a gear for driving a surveillance camera, various body parts for driving a robot, or a means of transportation. Specifically, it is all body parts of the robot, such as the hands, feet, and wheels of the robot, which are necessary for the operation for ordering and serving food and drink.

The voice control controller 240 controls for physically driving the voice unit 241. The voice control controller 240 is particularly provided in the mobile terminal 130 and the robot 135, and may be provided in the image pickup device 110 and the information processing device 120. Specifically, the voice unit 241 is a unit for collecting and pronouncing sounds such as a microphone and a speaker. Then, it is used for information notification to the cast 150 via voice, acceptance of orders, communication with guest 303 regarding serving of food and drink such as calling out of serve.
FIG. 3 is a configuration diagram showing an example of a state in which a glass is placed on a table.

FIG. 3 shows a state in which three glasses 301 of α, β, and γ are placed on the table 302. In addition, guest 303, who eats two people, A and B, is shown to be seated.

The glass 301 is an example of a container (tableware) for food and drink used by the guest 303. Therefore, the glass 301 is not particularly limited to a glass container, and may have various containers such as a teacup, a cup, a teacup, and a plate.

Guest 303 is a person who becomes a user of the glass 301. In FIG. 3, the guest 303 is a person who sits in the seat 304 and eats and drinks using the glass 301 placed on the table 302. Further, although not shown, the guest 303 includes various organisms such as dogs and cats because this mechanism can be applied not only to the purpose of serving but also to feeding and water supply of pets and livestock.

FIG. 4 is a configuration diagram showing an example of a display screen displayed on the display unit 210 of the information processing device 120, the mobile terminal 130, and the robot 135. In FIG. 4, information on the correspondence of the information processing device 120 to the user of the glass, which is determined based on the analysis of the image, is superimposed on the image taken by the image pickup device 110.

The image of FIG. 4 is drawn by imagining a bird's-eye view of the table 302 from above, but it is sufficient if the image in the table can be appropriately acquired, and the camera angle and the number of cameras are particularly limited. It is not limited to the shape. A fixed type device that can cover the entire table is preferably used, but a mobile type device may also be used. Therefore, for example, as shown in FIG. 7, an image acquired by the mobile terminal 130 or the camera unit 221 of the robot 135 may be used.

The seat 304 is the seat on which the guest 303 sits. The table 302, the guest 303, and the seat 304 are identified by image analysis and managed in association with each other. For example, the seat A001 of the table A and the guest A who is sitting are associated with each other, and the guest of the guest who is sitting with the seat A002 is associated with Mr. B. Further, in the case of FIG. 4, the seats A003 and Mr. C are associated with each other, but since Mr. C is not recognized in the image, the information processing device 120 determines that the seat is "away from the seat", and that is the case. It is displayed so that it can be identified.

In FIG. 4, the information processing apparatus 120 can analyze the image and specify the amount (regardless of the ratio or amount) of the drink contained in each glass 301. The method is not particularly limited, but for example, as shown in FIG. 5, a method of estimating the remaining amount of the drink remaining in the glass can be considered based on the outer peripheral edge of the glass and the contour of the drink. Further, as shown in FIG. 7, a method of estimating the height from the bottom of the glass of the drink to the water surface when viewed from the side can be considered.

FIG. 4 shows an example in which eight seats (A001 to A008) are arranged on one table (table A), and eight guests 303 (Mr. A to Mr. H) are seated and eating and drinking. (As mentioned earlier, Mr. C is away from his seat).

The information processing device 120 grasps who used which glass to eat and drink based on the image, and manages the glass 301 and the guest 303 in association with each other. Further, what and how much the guest eats and drinks is detected (or detected / estimated) and managed by various data tables described later in FIG. Then, the information superimposed on the image of FIG. 4 is displayed based on this information. Specifically, what kind of message or object is displayed in what kind of scene will be described in detail with reference to FIG.

Hereinafter, cases (1) to (8) show an example of displaying various patterns determined by the information processing apparatus 120 based on an image, and are surrounded by broken lines in FIG. 4 (1) to (8). It corresponds to each of 8).
[Case (1)]

Case (1) is a case in which guest 303 is a person who can drink alcohol, two glasses 301 are in use placed on the table, and it is detected and estimated to be in the tipsy period to the early stage of drunkenness from the alcohol acquisition status. Is.

Various methods of obtaining information on whether or not alcohol can be drunk can be considered. For example, it is desirable that the information processing apparatus 120 manages the data by acquiring the data in advance by some method such as confirming the age or self-reporting by the guest 303 with high accuracy. However, the method is not limited to such a method, and for example, a method of estimating the age of the guest from the image by using a face recognition technique or the like can be considered.

Various methods can be considered for associating the glass 301 with the guest 303. For example, a method of analyzing which guest picked up which glass, which glass he ate and drank, and the like from the image and associating them with each other can be considered (FIG. 14). Further, for example, an image of tableware used by each guest is registered and managed in the information processing device 120 in advance, and there is also a method of associating with the image, such as pattern matching. Alternatively, for example, if the distance between the seats is sufficiently wide, the glass 301 can be associated with the guest 303 (or the seat of the guest 303) depending on whether or not the glass 301 is within a certain distance or range. Conceivable. In any case, the most preferable response method should be taken depending on the situation. A specific example of the association between the vessel and the guest will be described later with reference to FIG.

The alcohol intake status of the guest 303 can be identified by the information processing apparatus 120 by analyzing the image. Specifically, for example, first, the capacity of the glass 301 and the alcohol content per quantity of each drink item are registered and managed in advance by the information processing apparatus 120. Then, based on this information, if the history of what and how much the guest 303 drank is known from the image, the alcohol blood concentration of the guest can be calculated. For further accuracy, the resolution of alcohol over time may be reflected or the permissible amount may be corrected for individual differences depending on the body weight estimated from the physique of the guest.

As an example of calculation of alcohol blood concentration, the following calculation formula and approximate estimation method are introduced on the website of Kirin Company Limited (registered trademark) as of November 10, 2016.
= Quote from here =
Blood alcohol concentration (%) = (drinking amount [ml] x alcohol content [%]) ÷ (833 x body weight [kg])
(Omitted)
Basis for calculation of coefficient 833: Alcohol blood concentration is the weight of alcohol consumed divided by the weight of body fluid.
= Quote up to here =

Further, as a method of estimating the status of the guest 303 from the alcohol intake status from the above-mentioned alcohol blood concentration, for example, a method by the Alcohol Health Medical Association "Thinking about Alcohol and Health" can be considered. According to this method, as shown in FIG. 6A, the state of sickness can be classified as follows from the blood alcohol concentration.
(B) Refreshing period (alcohol blood concentration is 0.02 to 0.04%)
(B) Tipsy period (alcohol blood concentration is 0.04 to 0.1%)
(C) Early stage of drunkenness (alcohol blood concentration is 0.10 to 0.15%)
(2) Intoxication period (alcohol blood concentration is 0.15 to 0.30%)
(E) Drunk period (alcohol blood concentration is 0.30 to 0.40%)
(F) Coma period (alcohol blood concentration is 0.40 to 0.50%)

The association line 401 is a line (an object superimposed on the image) for displaying the association between the glass 301 and the guest 303 in an identifiable manner. In case (1), guest 303 is associated with two glasses 301.

The alcohol gauge 402 is an object that displays an estimated alcohol blood concentration of guest 303 based on image analysis. As shown in FIG. 6, the alcohol gauge 402 has specifications corresponding to the estimated blood alcohol concentration of guest 303. In the alcohol gauge 402, when the value of the concentration gauge 601 exceeds the caution boundary line 602 indicating the alcohol blood concentration of 0.15% and the intoxication period is reached, "Caution" is displayed as shown in FIG. 6 (c). .. Further, as shown in FIG. 6D, when the warning boundary line 603 indicating the alcohol blood concentration of 0.30% is exceeded, a “warning” is displayed. Further, as shown in FIG. 6E, the guest 303, which has been registered in advance in the information processing device 120 to the effect that he / she cannot drink alcohol, is displayed as “Shimodo” so that he / she can know that he / she is a lower door. Will be done.

The guest name 403 is the name of the guest managed by the information processing device 120. The method of assigning the name is not particularly limited, but for example, the name may be assigned based on a predetermined number (A001 to A008) assigned to the table 302 or the seat 304, or a pre-registered guest face image or the like. It may be mentioned based on the name.

In the case (1), the information processing apparatus 120 detects and estimates that the beverage of about "B" in FIG. 5 remains in each of the two glasses 301 based on the image analysis. Therefore, the information processing apparatus 120 has determined that the response to the guest 303 is "not to speak". For this reason, the display unit 210 does not display information prompting the user to go for a new drink order.
[Case (2)]

Case (2) is a case where the guest 303 is a person who can drink alcohol, and one glass 301 in use placed on the table is in the early stage of drunkenness due to the acquisition status of alcohol. Based on the image analysis, the information processing device 120 detects and estimates that the beverage of about "A" in FIG. 5 remains in the glass 301, and determines that the guest 303 does not respond to the guest 303. There is. For this reason, the display unit 210 does not contain any information prompting the user to go for a new drink order.
[Case (3)]

Case (3) is a case in which guest 303 is away from his / her seat. Based on the image analysis, the information processing apparatus 120 detects and estimates that the beverage of about "C" in FIG. 5 remains in the glass 301. However, since the information processing device 120 also detects that the guest is away from the seat, it has decided not to speak to the guest 303. For this reason, the display unit 210 has an object (replacement stop instruction sign 404) that urges the guest 303 not to go to pick up a new drink order.
[Case (4)]

Case (4) is a case in which the guest 303 is a person who can drink alcohol, and one glass 301 in use placed on the table is in a tipsy period due to the acquisition status of alcohol. Based on the analysis of the image, the information processing device 120 estimates that the beverage of about "C" in FIG. 5 remains (almost empty) in the glass 301, and the response to the guest 303 is "new drink". I'm going to get an order. " For this reason, the display unit 210 is displayed to urge the guest 303 to pick up a new drink order.

The same item serve instruction sign 405 is an object displayed to tell the guest to ask for a replacement (of the same item of drink). This function is, for example, an all-you-can-drink service, where the unit price of the previously poured beverage is low, and the store operating the system of the present invention wants to recommend another drink in order to keep the beverage cost for guests low.・ It is expected that it will be used in such cases. Or conversely, in a service that makes a profit only for the ordered amount, it may be applied to drinks with a high unit price or drinks with a high profit margin.

Alternatively, in order to control that a specific type of drink does not run out of stock, it is a drink with a large inventory, so it is assumed that a "replacement" of the same item is recommended. Alternatively, it is assumed that a drink whose expiration date is approaching is poured into a glass, and a "replacement" of the same item is recommended to use up the drink.

As described above, various cases are assumed. The information processing apparatus 120 determines that the user (guest) of the glass is to serve a drink of the same item as the drink poured into the glass, based on the item poured into the glass. Then, based on this determination, the display unit 210 displays the same item serve instruction sign 405. In case (4), "Wine A" is displayed on the glass 301 at the bottom of the serve instruction sign 405 of the same item, and the cast who received this instruction asked the guest, "Would you like another glass of this wine?" ? ”Will be recommended.
[Case (5)]

Case (5) is a case in which the guest 303 is a person who can drink alcohol, and one glass 301 in use placed on the table is in a tipsy period due to the acquisition status of alcohol. In case (5), the cast 150 (robot 135 or mobile terminal 130 may be used) once asks guest 303 to serve a drink, but the guest says "it's fine now". I'm assuming a case like a drink. Therefore, as in the case (3), the substitute stop instruction sign 404 is displayed on the display unit 210.

These decisions are made by the information processing apparatus 120 based on the image. Specifically, for example, when a substitute instruction was given and the drink was not poured into the glass after the cast 150 approached, it was detected that there was a decline in the beverage serve saying "It's fine now".・ Can be estimated. However, it is not always necessary to rely on the image to detect and estimate that there was a decline. For example, the robot 135 may collect voice information from the voice unit 241 and store and manage the fact that the robot 135 has declined based on the voice information. In addition, the cast 150 may accept the information input to the effect of declining received from the guest through the mobile terminal 130.

In addition, the information that the refill has been accepted can be canceled by subsequently identifying that the beverage has been poured into the glass 301 by image analysis or receiving a notification from the guest 303 that the refill is desired. It may be provided as such. Then, by updating the information in this way, it is desirable to provide so that the state of the "sir block" that displays the substitute stop instruction sign 404 can be released.
[Case (6)]

Case (6) is a case in which the guest 303 is a person who can drink alcohol, and one glass 301 in use placed on the table is in the early stage of drunkenness due to the acquisition status of alcohol. Based on the image analysis, the information processing device 120 estimates and detects that the beverage of about "C" in FIG. 5 remains (almost empty) in the glass 301, and "takes a new drink order". I have decided to go. However, unlike case (4), it is not a decision to recommend a substitute for the same variety. The decision was to present the menu and ask, "Why don't you have a drink?" For this reason, the display unit 210 is displayed to urge the guest 303 to pick up a new drink order.

The different item serve instruction sign 406 is an object that is displayed to tell the guest to ask for a drink. This sign is for all-you-can-drink services when the drinks currently in the glass are expensive, low in stock, or have other stocks that you want to lose early. It is effective if it is provided so that it can be displayed. In the case (6), wine B was poured into the glass 301 first, and "wine B" is displayed at the bottom of the different item serve instruction sign 406 so that it can be understood. Therefore, the cast 150 and the robot 135 will avoid this wine B and take an approach that recommends other drinks to the guest 303. However, when presenting the menu, the menu including wine B is presented to the guest 303, so it is not necessarily the case that orders for the same item are not accepted.
[Case (7)]

In case (7), the guest 303 is a person who cannot drink alcohol, and the glass 301 in use placed on the table is one case. Based on the image analysis, the information processing device 120 detects and estimates that the beverage of about "C" in FIG. 5 remains in the glass 301, and determines that "go to get a new drink order". ing. For this reason, the display unit 210 is displayed to urge the guest 303 to pick up a new drink order.

The object displayed here is a sign of instruction to serve the same kind of drink as in case 4, but it is possible to identify that alcohol cannot be drunk, which is the same item non-alcoholic serve instruction sign 407. Is. Further, as shown in FIG. 6E, the alcohol gauge 402 has a "Shimodo" sign indicating that the person cannot drink. In the case (6), juice A was poured into the glass 301 first, and "juice A" is displayed at the bottom of the non-alcoholic serve instruction sign 407 of the same item so that it can be understood.
[Case (8)]

Case (8) is a case in which the guest 303 is a person who can drink alcohol, and one glass 301 in use placed on the table is in a drunken state due to the acquisition status of alcohol. Based on the image analysis, the information processing device 120 decides to call out to the guest 303, "How about a soft drink?" For this reason, the display unit 210 has a display urging the guest 303 to go to pick up an order for a soft drink.

Alternatively, the decision may be to serve cold first. This is a decision aimed at lowering the alcohol blood concentration of guest 303, and is a decision not to serve alcohol to the guest any more. In order to discriminately display such a situation, the alcohol gauge 402 may fall into an extremely unfavorable state such as a drunken period or a coma period when further alcohol intake is performed, as shown in FIG. 6 (c). The word "Caution" is displayed to indicate that there is. In addition, a hydration instruction sign 408 for notifying the cast 150 to recommend the guest 303 for a soft drink (or water) for lowering the alcohol blood concentration of the guest 303 is displayed.

Further, although not shown in this figure, based on image analysis, it is possible to manage when the guest 303 used the glass 301, identify the glass 301 that has not been used for a predetermined time or longer, and remove the glass 301. A mechanism for making a decision to ask the guest 303 if it is okay to do so is also conceivable. Then, a notification based on this decision may be given to the cast mobile terminal 130 or the robot 135, and a mechanism may be provided that leads to an action of asking the user's guest whether the vessel may be removed based on the decision.

FIG. 5 is a configuration diagram showing an example of a view of the glass 301 viewed from the top surface and the side surface. The view shown by the circle in the upper half is the top view, and the view in the lower half is the side view. In FIG. 5, A represents a state in which the glass 301 is filled with a beverage, B represents a state in which the glass 301 contains about half of the beverage, and C represents a state in which the beverage in the glass 301 has been drunk. The information processing device 120 analyzes the amount of drink in the glass detected from the image, and estimates the amount and the item. For example, assuming that the capacity of this wine glass is 120 ml, in the state of A, nearly 120 ml (100%) still remains, and in the state of B, half a drink remains, and nearly 60 ml (50%) remains. ... So I haven't spoken to him yet. Can be detected, estimated, and determined. In addition, in the state of C, since the drink is almost completely consumed, the history of eating and drinking can be increased by 120 ml, and detection / estimation / determination such as "calling out" to the guest can be performed.

It is not always necessary to specify the item of the drink poured into the glass only by the color of the drink in the glass. For example, it is possible to improve the accuracy by linking with the drink order information received from the guest. Also, if it is something like wine, a method of identifying the item by analyzing the shape and label of the bottle in the image when the drink is poured into the glass can be considered.
FIG. 6 is a configuration diagram showing an example of correspondence between the estimated alcohol blood concentration 1323 and the alcohol gauge 402.
FIG. 6A is a diagram showing the estimated blood alcohol concentration 1323 and the names of the respective states.

Alcohol blood concentration 0.02 to 0.04% is called "refreshing period", and guest 303 feels "refreshing", "skin becomes red", "becomes cheerful", and "judgment is slightly impaired". It is said that it will be in a state such as.

Alcohol blood concentration of 0.04 to 0.10% is called "tips", and guest 303 "feels tipsy", "becomes active in hand movements", and "can be suppressed (loss of reason)". It is said that "body temperature rises" and "pulse becomes faster".

Alcohol blood concentration of 0.10 to 0.15% is called "early drunkenness", and guest 303 is "getting upset", "loudly screaming", "getting angry", and "staggering when standing". It is said that it will be in such a state.

In this example, the blood alcohol concentration of 0.00 to 0.15% (from the refreshing period to the early stage of drunkenness) is assumed to be the amount that the guest 303 can preferably ingest alcohol, and during this period, cautions and warning messages regarding the serve of alcohol are given. It is in the range that does not come out. However, this is just an example, and the standard value (predetermined value) of alcohol blood concentration for issuing notifications and messages may be changed depending on the physique and constitution of each guest 303, and is not necessarily 0.15%. It doesn't have to be.

Alcohol blood concentration of 0.15 to 0.30% is called "drunkenness", and guest 303 is "staggered", "speaks the same thing many times", "breathing faster", "nausea / vomiting". It is said that it will be in an unfavorable state such as "it will happen".

Therefore, in this example, the estimated blood alcohol concentration of 0.15 to 0.30% (drunkenness period) is defined as an amount that requires caution in the alcohol intake of guest 303. During this period, the guest 303 is instructed to refrain from ordering or serving alcohol, urging orders for soft drinks, and encouraging cold serve. The guideline for the blood alcohol concentration for issuing such a notification or message is only an example, and the value may be changed depending on the physique and constitution of each guest 303.

Alcohol blood concentration of 0.30 to 0.40% is called "drunk period", and guest 303 is very unfavorable, such as "not standing properly", "unclear consciousness", and "language is messed up". It is said to be in a state. When the alcohol blood concentration exceeds 0.40%, a state called "coma period" occurs, and the guest 303 is said to be in a very unfavorable state with a risk of death in addition to the above.

In this example, the state in which the estimated blood alcohol concentration of the guest 303 exceeds 0.30% (drunkenness period) is regarded as the amount that causes the guest 303 to fall into a very unfavorable state. Then, in the case where the guest 303 is determined to have fallen into such a situation, it is provided to make a decision to take further necessary measures such as calling an ambulance. The guideline for the blood alcohol concentration for sending, instructing, and processing such notifications and messages is just an example, and the value may be changed depending on the physique and constitution of each guest 303.

Supplement each condition and the standard of alcohol. It is said that the standard for the refreshing period is up to 1 beer (medium bottle), 1 go of sake, and 2 whiskey singles. It is said that the guideline for the tipsy period is 1-2 bottles of beer (medium bottle), 1-2 go of sake, and 3 glasses of whiskey single. It is said that the initial guideline for drunkenness is 3 bottles of beer (medium bottle), 3 cups of sake, and 3 glasses of whiskey double. It is said that the standard for drunkenness is 4 to 6 bottles of beer (medium bottle), 4 to 6 cups of sake, and 5 glasses of whiskey double. It is said that the standard for the drunken period is 7 to 10 bottles of beer (medium bottle), 7 to 1 sho of sake, and 1 bottle of whiskey. The coma period is greater than or equal to the above. In this way, by grasping what kind of alcohol and how much the guest 303 ingested by using the image, the guest 303 is based on the above-mentioned guideline without using the calculation formula shown in FIG. It is also possible to detect and estimate the state of.

FIG. 6B is a configuration diagram showing an example of a state of the alcohol gauge 402 without a caution or warning message. The concentration gauge 601 shows the alcohol blood concentration estimated from the image as a bar graph. In the case of this figure, it is about 0.10%, which indicates that the state is between the tipsy period and the early stage of drunkenness.

FIG. 6C is a configuration diagram showing an example of the state of the alcohol gauge 402 in which the estimated state of the guest 303 is displayed during the intoxication period. The caution boundary line 602 is a line indicating a predetermined alcohol blood concentration that serves as a guide for transmitting a caution message. In this embodiment, the estimated blood alcohol concentration of guest 303 is 0.15 to 0.30% as a guideline value for transmitting a caution message, and when the concentration gauge 601 indicates between these values. A caution message is displayed.

FIG. 6D is a configuration diagram showing an example of the state of the alcohol gauge 402 in which the estimated state of the guest 303 is displayed during the drunken period or the coma period. The warning boundary line 603 is a line indicating a predetermined alcohol blood concentration that serves as a guide for transmitting a warning message. In this embodiment, the estimated blood alcohol concentration of guest 303 is 0.15 to 0.30% as a guideline value for transmitting a warning message (display of "warning"), and the concentration gauge 601 sets the concentration gauge 601. A warning message is displayed when indicating between these values.

FIG. 6E is a configuration diagram showing an example of the state of the displayed alcohol gauge 402 in order to identify the guest 303 who cannot drink alcohol. Although the description in this figure is "Shimodo", the reason why you cannot drink alcohol is not particularly limited, and there are various reasons such as being unable to drink alcohol for religious reasons, being a minor, or having an allergic constitution. It is assumed, and it may be written separately so that it can be identified.

FIG. 7 is a configuration diagram showing an example of a display screen displayed on the display unit 210 of the information processing device 120 or the mobile terminal 130. FIG. 7 shows an example in which three seats (B001 to B003) are arranged on one table (table B), and three guests (Mr. I, Mr. J, and Mr. K) are seated and eating and drinking. Is shown. Further, a robot 135 (may be a mobile terminal 130) is provided on the table.
The image is an image acquired by the mobile terminal 130 or the camera unit 221 of the robot 135. The display method conforms to FIG.

The emergency response command mark 701 is based on the fact that the information processing device 120 detects and estimates that the corresponding guest 303 is in a very unfavorable state based on the image, and decides to take emergency measures. , An object displayed to indicate that. Emergency treatment is, for example, if there is a high possibility of acute alcohol intoxication and you need to call an ambulance, you are in a state of cardiopulmonary arrest, and you need to use an AED (automated external defibrillator) or give artificial respiration. If there is, and so on. In this way, the display of the emergency response command mark 701 may be further subdivided according to the necessary measures, and in addition to the display, processing such as an ambulance call and a guidance display of the AED storage location can be performed. , May be provided.

In addition, in situations where an emergency response command mark 701 or a warning display is required as shown in this figure, other displays are controlled, such as a faint display that interrupts other displays so that the attention of the cast 150 is immediately directed to the warning message. You may perform such a process. Further, the robot 135 may be provided to notify and instruct the robot 135 to give the highest priority to the guest who is presumed to be in a very unfavorable state, and to be controlled as such.

As described above, it is desirable that the decisions based on the image and history of the information processing apparatus 120 are prioritized and processed in that order. For example, it is conceivable that the alcohol gauge 402 gives the highest priority to responding to the guest 303 in the "warning" state, and the second is "caution". In addition, it is possible to consider operations such as giving priority to guest 303, who has a high profit margin and gives a lot of refills. In this way, when there are a plurality of guests 303 to be called out, it is desirable to establish a predetermined rule of how to prioritize the calls.
FIG. 8 is a configuration diagram showing an example of the main flow of the present invention.

In the description of the flow of each figure of the present embodiment (mainly FIGS. 8 to 12), specifically, the CPU 201 loads the control program stored in the external memory 211 into the RAM 203 and executes it. It is something that is said. However, for the sake of simple expression, the image pickup device 110, the information processing device 120, the mobile terminal 130, the robot 135, and the like will be mainly described.
This flow is the main flow of the present invention, and shows the following movements of the information processing apparatus 120.
(1) Acquisition of images taken by the image pickup device 110 (2) Update of the eating and drinking history of the glasses 301 and guest 303 based on (1) and the current detection / estimation guest (3) To guest 303 based on (2) Decision of response and instruction of response based on (4) and (3)

Further, as shown above, there may be a case where the information processing device 120 also serves as an image pickup device 110, or a case where the information processing device 120 also serves as a mobile terminal 130 or a robot 135. The communication (transmission / reception) step may be omitted.
In step S801, the information processing device 120 requests the image pickup device 110 to transmit an image.
In step S802, the image pickup apparatus 110 receives the image transmission request transmitted from the information processing apparatus 120.
In step S803, the image pickup apparatus 110 acquires an image through the camera unit 221.
In step S804, the image pickup apparatus 110 transmits the image acquired in step S803 to the information processing apparatus 120.
In step S805, the information processing device 120 receives the image transmitted from the image pickup device 110.

In step S806, the information processing device 120 detects and estimates the glass 301, the guest 303, and the current state based on the image, and updates the history of eating and drinking of the guest. The detailed processing of this step will be described in the subflow of FIG.

In step S807, the information processing apparatus 120 determines the correspondence to the guest 303 from among several options based on the information detected, estimated, and updated in step S806. The detailed processing of this step will be described in the subflows of FIGS. 10 and 11.

In step S808, the information processing device 120 notifies the mobile terminal 130 and the robot 135 of the information obtained from the image in step S806 and the information regarding the decision in step S807 in order to implement the correspondence to the guest 303. Alternatively, the information processing apparatus 120 responds to the guest 303 based on the determination in step S807.
The detailed processing of this step will be described in the subflow of FIG.
In step S809, the information processing apparatus 120 receives an instruction from the user to end the flow process.

In step S810, the information processing apparatus 120 determines whether or not the instruction to end the flow process has been received from the user in step S809. Then, if it is determined that the acceptance has been made, the process proceeds to step S810. If it is determined that the message has not been accepted, the process proceeds to step S801.
In step S811, the information processing device 120 transmits an instruction to end the flow process received in step S809 to the image pickup device 110.
In step S812, the image pickup apparatus 110 receives the instruction to end the flow processing transmitted from the information processing apparatus 120 in step S811.

In step S813, the image pickup apparatus 110 determines in step S812 whether or not it has received the instruction to end the flow process. Then, when it is determined that the acceptance has been received, the flow is terminated. If it is determined that the message has not been accepted, the process proceeds to step S802.
FIG. 9 is a configuration diagram showing an example of the subflow of step S806 of the present invention.
This flow is a flow in which the information processing device 120 detects, estimates, and updates various information based on the image acquired from the image pickup device.

In step S901, the information processing device 120 identifies the glass 301, the beverage (item and quantity), the person, and the seat in the glass 301 from the image. Also, identify if there is a person tied to the glass or seat.

Various methods can be considered for identifying (detecting or estimating) each glass 301, and the method is not particularly limited. For example, there is a method described later in FIG. 15 for details, but here, as an example, a number or an identification code is assigned to a part of the glass, and the glass can be identified by analyzing the image. I recommend. Then, the information detected / estimated from the image is managed by the glass state management table 1300. Here, it is assumed that the glasses 301 of G001 to G009 are placed on the table 302 (table A) of FIG. 4, and the glasses 301 of G010 to G012 are placed on the table 302 (table B) of FIG.
Various methods can be considered for specifying the amount of the glass, and the method is not particularly limited. For example, the method described with reference to FIG. 5 can be considered.

Various methods can be considered for identifying the item in the glass 301, and the method is not particularly limited. For example, it may be estimated from the color of the drink poured into the glass and the characteristics of the bottle when the drink is poured into the glass, or information input from whom the order was placed is separately accepted and managed. It may be associated with the information.

Various methods can be considered for identifying a person, and the method is not particularly limited, but a form in which features such as clothes and face are extracted and estimated is preferable. The guest's face may be separately registered and managed in advance and collated with this information, or the guest extracted from the image may be assigned a number and managed. Alternatively, a non-contact tag or the like may be worn in advance and detected in combination with a device capable of detecting the tag. In any case, a method that can accurately identify the person corresponding to the image may be used.

Various methods can be considered for identifying the seat, and the method is not particularly limited. However, in FIG. 4, the backrests of the seats are numbered A001 to A008, and the seats are identified based on these numbers. ..

In addition to the information described above, this step is provided to provide any information obtained by analyzing the image that will be needed to determine the subsequent response to the guest. The specific information on which to decide the response to the guest will be described in detail in the explanation of the flow that follows.

Steps S902 to S907 are loop processes for performing processing for each of the specified glasses 301. For example, in the case of FIG. 4, since there are nine glasses 301 of G001 to G009, the loop processing is performed nine times.

In step S902, the information processing apparatus 120 determines whether or not there is a user in the glass 301 of interest based on the image. Then, when it is determined that there is a user, the process proceeds to step S906, and when it is determined that there is no user, the process proceeds to step S903.

In step S903, the information processing apparatus 120 determines whether or not the glass 301 of interest is possessed based on the image. If it is determined that the product is possessed, the process proceeds to step S904, while if it is determined that the product is not possessed, the process proceeds to step S907.

In step S904, the information processing apparatus 120 determines whether or not the person holding the glass 301 is the cast 150 based on the image. Then, if it is determined that the cast is 150, the process proceeds to step S907, while if it is determined that the cast is not 150, the process proceeds to step S905.

Various methods can be considered for determining whether or not the cast is cast, but it is preferable to use a cast face image, an identification code worn by the cast, or the like. Specifically, as shown in the cast management table 1310, a cast image and a reading code including information such as a cast name, clothes and a face image are registered and managed in advance, and match the person and the reading code of the image. A method of determining whether or not the cast is performed can be considered.

In step S905, the information processing device 120 associates the person holding the glass 301 of interest as a glass user (guest 303). That is, in the glass state management table 1300, the glass 301 and its user are managed in association with each other.

In step S906, the information processing apparatus 120 updates the data table such as the food and drink history of the guest 303 associated with the glass 301 and the inventory information of drinks based on the information obtained based on the image in step S901. Will be done.

Specifically, for example, the food and drink history 1322 of the guest history management table 1320 is updated according to changes in the item and quantity of drinks in the glass. Specifically, for example, suppose that it was first recorded that 120 ml of red wine A was contained in the previous loop processing, and that this time, 60 ml of red wine A was detected and estimated. In this case, the food and drink history 1322 increases the history of red wine A corresponding to the user (guest 303) by 60 ml.

In addition, the estimated alcohol blood concentration 1323 is also updated in accordance with the update of the food and drink history 1322. If it can be detected from the image that a new drink has been served, the final serving time 1325 and the number of servings 1327 are updated. Further, when the guest 303 can detect the state of holding the glass or the state of putting it on the mouth, the final eating and drinking time 1326 is updated.

In step S907, the information processing apparatus 120 determines that there is no user in the glass 301 of interest, and proceeds to the next step (next route processing or end of flow).
10 and 11 are block diagrams showing an example of the subflow of step S807 of the present invention of the present invention.

This flow is based on the information detected, estimated, and updated in step S806 based on the image acquired by the information processing device 120 from the image pickup device, and is selected from several options provided in the flow to the guest. This is the flow for determining the correspondence.
Then, this process is repeated for each glass 301.

However, as in the case (1) of FIG. 4, when a plurality of glasses are determined for one guest 303, further provision may be provided so as not to make a determination corresponding to the number of glasses. Good. For example, it is conceivable to preferentially adopt the decision regarding the glass 301, which is the latest time for the final eating and drinking time 1326. With this arrangement, it is not necessary to call out to the same guest multiple times, and a glass of drink whose final eating time 1326 is not the latest can be regarded as tired of drinking in the middle and acted.

In step S1001, the information processing apparatus 120 determines whether or not the user (guest 303) of the glass 301 of interest is present with reference to the glass state management table 1300 updated in step S806. To do. (The determination may be made by analyzing the image to specify whether or not there is a guest 303 corresponding to the glass 301.) Then, if it is determined that the guest is present, the process proceeds to step S1002, while the presence is present. If it is determined that the image is not present (absent), the process proceeds to step S1014.

In step S1002, the information processing apparatus 120 determines whether or not the alcohol blood concentration of the user (guest 303) of the glass 301 of interest is less than 0.15%. The determination is made with reference to the guest history management table 1320 updated in step S806 based on the image. Then, if it is determined that it is less than 0.15%, the process proceeds to step S1004, while if it is determined that it is not less than 0.15%, the process proceeds to step S1003.

In step S1003, the information processing apparatus 120 determines whether or not the alcohol blood concentration of the user (guest 303) of the glass 301 of interest is 0.15% or more and less than 0.30%. The determination is made with reference to the guest history management table 1320 updated in step S806 based on the image. Then, if it is determined that it is less than 0.15%, the process proceeds to step S1004, while if it is determined that it is not less than 0.15%, the process proceeds to step S1003.

In step S1004, the information processing device 120 determines whether or not the user (guest 303) of the glass 301 of interest is the lower door. The determination is made with reference to the guest history management table 1320 or the guest profile data table 1330. Then, if it is determined that the door is a lower door, the process proceeds to step S1007, while if it is determined that the door is not a lower door, the process proceeds to step S1005.

In step S1005, the information processing device 120 determines whether or not the user (guest 303) of the glass 301 of interest is a minor. The determination is made with reference to the guest profile data table 1330. Then, if it is determined that the person is a minor, the process proceeds to step S1007, while if it is determined that the person is not a lower door, the process proceeds to step S1006.

In step S1006, the information processing apparatus 120 determines that "to accept or serve an order including an alcoholic beverage" as a response to the user (guest 303) of the glass 301 of interest.

In step S1007, the information processing device 120 determines that "to accept or serve only soft drinks" as a response to the user (guest 303) of the glass 301 of interest.

In step S1008, the information processing apparatus 120 determines that "emergency treatment is to be performed" as a response to the user (guest 303) of the glass 301 of interest. First aid is specifically for guests with a blood alcohol concentration of 0.3% or higher (drunken or coma), depending on the situation, hydration, laying in an appropriate place, or an ambulance. It is a decision to take measures such as calling.

As described above, the correspondence to the guest 303 determined in any one of step S1006, step S1007, and step S1008 is managed by the order serving policy 1342 of the decision correspondence management table 1340. Further, when it is determined that emergency treatment is necessary, "action 4 (emergency treatment)" is also inserted and managed in the column of action 1343.

In step S1009, the information processing device 120 determines whether or not the glass 301 of interest has not been used for a predetermined time. Judgment is made by guest history management table 1320,

The removal time management table 1350 and the current time based on the clock built in the information processing device 120 (time information may be acquired separately) are referred to. Then, if it is determined that the product has not been used for a predetermined time, the process proceeds to step S1015, while if it is determined that the product has not been used for a predetermined time, the process proceeds to step S1010. This "predetermined time" is arbitrarily determined by the user and is managed by the removal time management table 1350. (In FIG. 13, it is set to 1 hour)

In step S1010, the information processing device 120 determines whether or not the amount of drink in the glass 301 of interest is insufficient. The determination is made with reference to the glass state management table 1300 and the voice determination amount management table 1360 updated in step S806 based on the image. Then, if it is determined that there is a shortage, the process proceeds to step S1011. On the other hand, if it is determined that there is no shortage, the process proceeds to step S1014. The criteria for whether or not the amount of drink is insufficient is arbitrarily determined by the user, and is managed by the voice-decision amount management table 1360. (In FIG. 13, it is set to less than 5%)

In step S1011, the information processing apparatus 120 determines whether or not the glass 301 of interest has a "not lowered flag". The determination is made with reference to the guest history management table 1320 updated in step S806 based on the image. Then, if it is determined that there is no "not lowering flag", the process proceeds to step S1012, while if it is determined that there is a "not lowering flag", the process proceeds to step S1014.

Here, the "don't lower flag" will be described. This is a flag that is set when the cast 150 or the robot 135 approaches the guest 303 to lower the empty glass 301, and the guest requests "Please do not lower the glass yet". The flag is managed in the guest history management table 1320, but the flag input may be obtained from an image or may be set by analyzing the voice obtained from the voice unit. In addition, the cast corresponding to the guest may input / transmit through the mobile terminal and accept it.

In step S1012, the information processing apparatus 120 determines whether or not the glass 301 of interest has the “serve abandonment flag”. The determination is made with reference to the guest history management table 1320 updated in step S806 based on the image. Then, if it is determined that there is no "serve abandonment flag", the process proceeds to step S1013, while if it is determined that there is a "serve abandonment flag", the process proceeds to step S1014.

Here, the "serve abandonment flag" will be described. This is a flag that stands when the cast 150 or robot 135 approaches the user of the glass 301 to accept a drink order, and when the guest offers that "refills are still fine". is there. The flag is managed in the guest history management table 1320, but the flag input may be obtained from an image or may be set by analyzing the voice obtained from the voice unit. In addition, the cast corresponding to the guest may input / transmit through the mobile terminal and accept it.

In step S1013, the information processing apparatus 120 determines to "call out" as a response to the user (guest 303) of the glass 301 of interest. The detailed determination of what kind of voice is to be made will be described in the flow of FIG.

In step S1014, the information processing apparatus 120 determines that "do not speak" as a response to the user (guest 303) of the glass 301 of interest. Then, in response to this decision, "action 5 (no voice)" is inserted and managed in the action 1343 column of the decision response management table 1340.

In step S1015, the information processing apparatus 120 makes a decision to "ask whether or not the glass 301 can be removed" as a response to the user (guest) of the glass of interest. Then, in response to this decision, "action 6 (asking whether or not to remove)" is inserted and managed in the column of action 1343 in the decision response management table 1340.

In step S1101, the information processing device 120 determines whether or not the drink (was) poured into the glass of interest is at least a predetermined unit price. The determination is made with reference to the glass state management table 1300, the correspondence determination level management table 1370, and the inventory management table 1380 updated in step S806 based on the image. Then, if it is determined that the unit price is equal to or higher than the predetermined unit price, the process proceeds to step S1102, while if it is determined that the unit price is not equal to or higher than the predetermined unit price, the process proceeds to step S1106. In addition, this step is a judgment branch assuming that customers who are drinking high-priced drinks in a service that is charged only for what they drink will be encouraged to refill. Therefore, if we assume a situation where we recommend low-priced drinks to customers who are drinking high-priced drinks in the all-you-can-drink service, the opposite decision is made. That is, the information processing device 120 determines whether or not the drink in the glass of interest is at least a predetermined unit price. Then, if it is determined that the unit price is not less than the predetermined unit price, the process proceeds to step S1102, while if it is determined that the unit price is not less than the predetermined unit price, the process proceeds to step S1106. This determination is made with reference to the glass state management table 1300, the correspondence determination level management table 1370, and the inventory management table 1380 updated in step S806 based on the image.

This "predetermined unit price" used for this step determination is arbitrarily determined by the user, and is managed by the unit price level 1372 of the correspondence determination level management table 1370. Further, the unit price of each item used for this step determination is managed by the unit price 1382 of the inventory management table 1380.

In step S1102, the information processing device 120 determines whether or not the drink in the glass of interest has a profit margin or higher. The determination is made with reference to the glass state management table 1300, the correspondence determination level management table 1370, and the inventory management table 1380 updated in step S806 based on the image. Then, if it is determined that the profit margin is equal to or higher than the predetermined profit margin, the process proceeds to step S1103, while if it is determined that the profit margin is not equal to or higher than the predetermined profit margin, the process proceeds to step S1106. This determination is made with reference to the glass state management table 1300, the correspondence determination level management table 1370, and the inventory management table 1380 updated in step S806 based on the image. The "predetermined profit margin" used for this step determination is arbitrarily determined by the user, and is managed by the profit margin level 1373 of the correspondence decision level management table 1370. Further, the profit margin of each item used for this step determination is managed by the profit margin 1383 of the inventory management table 1380.

In step S1103, the information processing apparatus 120 determines whether or not the inventory of the drink in the glass of interest is within a predetermined expiration date or expiration date. The determination is made with reference to the glass state management table 1300, the correspondence determination level management table 1370, and the inventory management table 1380 updated in step S806 based on the image. Then, if it is determined that the profit margin is equal to or higher than the predetermined profit margin, the process proceeds to step S1103, while if it is determined that the profit margin is not equal to or higher than the predetermined profit margin, the process proceeds to step S1106. This determination is made with reference to the glass state management table 1300, the correspondence determination level management table 1370, and the inventory management table 1380 updated in step S806 based on the image. The "predetermined profit margin" used for this step determination is arbitrarily determined by the user, and is managed by the profit margin level 1373 of the correspondence decision level management table 1370. Further, the expiration date and the expiration date of each item used for this step determination are managed by the expiration date 1385 and the expiration date 1386 of the inventory management table 1380.

In step S1104, the information processing apparatus 120 determines whether or not the inventory amount of the drink in the glass of interest is equal to or higher than a predetermined level. The determination is made with reference to the glass state management table 1300, the correspondence determination level management table 1370, and the inventory management table 1380 updated in step S806 based on the image. Then, when it is determined that the level is equal to or higher than the predetermined level, the process proceeds to step S1105, and when it is determined that the level is not higher than the predetermined level, the process proceeds to step S1106. This determination is made with reference to the glass state management table 1300, the correspondence determination level management table 1370, and the inventory management table 1380 updated in step S806 based on the image. The "predetermined inventory level" used for this step determination is arbitrarily determined by the user, and is managed by the inventory level 1375 of the correspondence determination level management table 1370. Further, the inventory amount of each item used for this step determination is managed by the inventory amount 1384 of the inventory management table 1380.

In step S1105, the information processing apparatus 120 recommends a drink of the same item as the item poured into the glass of interest as a response to the owner (guest 303) of the glass of interest. decide. Then, in response to this decision, "action 1 (same type)" is inserted and managed in the action 1343 column of the decision correspondence management table 1340.

In step S1106, the information processing apparatus 120 recommends a drink of an item different from the item poured into the glass of interest as a response to the owner (guest 303) of the glass of interest. decide. Then, in response to this decision, "action 1 (same type)" is inserted and managed in the action 1343 column of the decision correspondence management table 1340. In addition, I wrote that it was decided to recommend drinks of different items, but in reality, in response to the decision of this step, present the menu to guest 303 and ask "How about ordering drinks?" become. Therefore, when recommending a different item of drink, the options will include the same item as the drink being poured into the glass of interest. It should be added that the expression "decision to recommend drinks of different items" is included, including the occurrence of such a situation.

As described above, the flow of FIGS. 10 and 11 determines the correspondence to the user (guest 303) of the glass 301 corresponding to each glass 301. The determined action is managed by the order serving policy 1342 and the action 1343 of the decision response management table 1340. Correspondence to guest 303 is decided from several options as follows. Specifically, as shown above, it is determined in one or a plurality of steps of step S1006, step S1007, step S1008, step S1013, step S1014, step S1015, step S1105, and step S1116. These decision options are managed by the information processing apparatus 120, and the management form may be in the form of a data table such as the action / response management table 1600, or is incorporated in the flow. It does not matter if it is in the form of.
FIG. 12 is a configuration diagram showing an example of the subflow of step S808 of the present invention.

The point of this flow is that the mobile terminal 130 or the robot 135 has various information based on the information identified (detected / estimated) / updated based on the image in step S806 and the response to the guest determined in step S807. It is the point of performing actions and information processing.

Further, when the information processing device 120 also serves as the mobile terminal 130 or the robot 135, the step of communication (transmission / reception) between the processing entities that also serve may be omitted.
In step S1201, the information processing device 120 transmits the following information to the mobile terminal 130 / robot 135.
(B) Data table that manages the information specified / updated in step S806 (b) Data table that manages the response to guests determined in step S807.

(C) Other data table including information required in steps S1202 to S1209 managed by the information processing apparatus 120 among the other information taken up in FIGS. 13 and 16.

In step S1202, the mobile terminal 130 or the robot 135 receives the information (or the data table including the information) transmitted from the information processing apparatus 120 in step S1201.

In step S1203, the response to the guest 303 is prioritized. In addition, the processing of this step is performed in advance by the information processing device 120 before step S1201 in order to reduce the processing addition of the mobile terminal 130 or the robot 135, and the information including the priority is transmitted in step S1201. It doesn't matter if there is.

In step S1204, the mobile terminal 130 or the robot 135 notifies the cast of various instruction contents based on the information received in step S1202 and the action / response management table 1600. Specifically, for example, as shown in FIGS. 4 and 7, a method of displaying a display screen on which an object showing the state of the guest 303 and the action to be taken is superimposed on the image is displayed on the display unit 210. Alternatively, a method of notifying each guest 303 of what to do through the voice unit 241 can be considered.

In step S1205, the mobile terminal 130 or the robot 135 determines whether or not the guest 303 needs emergency treatment. The determination may be made by referring to the decision correspondence management table 1340, or by using the action / correspondence management table 1600 and the related data tables of FIGS. 13 and 16. Then, when it is determined that the necessary guest 303 is present, the process proceeds to step S1206. On the other hand, if it is determined that the required guest 303 is not present, the process proceeds to step S1207.

In step S1206, the mobile terminal 130 or the robot 135 executes emergency measures for the guest 303 who needs emergency response based on the information received in step S1202 and the action / response management table 1600. The specific emergency response is as described above, but it is a measure such as hydration to lower the blood alcohol concentration, care of the guest 303, and an act of calling an ambulance. Then, when the treatment is completed, the process proceeds to step S1203, and the response to the guest is prioritized again.
Step S1207 and step S1208 are loop processes performed for each guest 303.

In step S1207, the mobile terminal 130 or the robot 135 calls out to the target guest 303 for ordering and then responds to the target guest 303 based on the information received in step S1202 and the action / response management table 1600. Specifically, for example, Mr. A, Mr. B, and Mr. C in FIG. 4 are not called out, and the processing of the next step is not performed. Also, ask Mr. D in Fig. 4, "Would you like another glass of wine A?", And if Mr. D wants a refill, serve wine A in the next step. become. In addition, if the serve is abandoned saying "Refills are fine now", the serve abandonment flag 1329 of the guest history management table 1320 is set, and the serve will not be performed in the next step. In addition, if a replacement for another drink is desired, such as "I want to drink another drink", the next step is to bring the menu and accept the order. As described above, in the next step, various measures are taken depending on the result of the voice call processing in this step.

The processing of this step is written on the assumption that the processing is performed mainly by the mobile terminal 130 or the robot 135 equipped with software capable of interacting with the guest 303. However, if such software cannot be prepared, the cast 150 will be responsible for calling out and responding to orders (such as serving based on orders). Then, the mobile terminal 130 or the robot 135 may accept the input of the result of the cast exchange, or may provide a mechanism for detecting and estimating the result of the cast exchange and process the flow.

In step S1208, the mobile terminal 130 or the robot 135 performs drink serving processing and order processing according to the result of the voice of the target guest in step S1207. A specific example of the correspondence is as described in step S1207.

In step S1209, the mobile terminal 130 or the robot 135 transmits information regarding the processing results of steps S1202 to S1209 to the information processing device 120.
In step S1210, the information processing apparatus 120 receives the information regarding the processing result transmitted in step S1209.
FIG. 13 is a configuration diagram showing an example of a data table used in the data flow processing of the present invention.

The glass state management table 1300 is a data table managed by the information processing apparatus 120. As shown in the explanation of the flow of FIG. 9, this table is a table for the information processing device 120 to analyze the image obtained from the image pickup device 110 and manage the information specified (detected or estimated) from the image. is there. The glasses 301 detected by the information processing device 120 are managed by the rows of the detection glasses 1301. In addition, the drinks poured into each glass are managed by 1302. Then, how much the drink remains in the glass is managed by 1303, the user (guest) of the glass is 1304, the seat of the user is 1305, and the table with the seat of the user is set. It is managed by 1306 respectively. Then, the presence status of whether or not the glass user is enrolled is managed by 1307.

The cast management table 1310 is a data table managed by the information processing apparatus 120. As shown in the flow description of FIG. 9, this table contains the cast name (managed by columns 1311), the image of the cast (managed by columns 1312), and the reading code information for identifying the cast (1313). It is a data table for managing (managed by columns).

The guest history management table 1320 is a data table managed by the information processing apparatus 120. In this table, the guest history management table 1320 is estimated from the history of eating and drinking for each user (guest 303) of each glass 301 based on the information obtained by image analysis by the information processing device 120. It is a data table for managing the state. Specifically, the history of eating and drinking of drinks (managed by column 1322), the alcohol blood concentration estimated from them (managed by column 1323), the start time of eating and drinking (managed by column 1324), and finally the drink to the glass. The time served (managed in 1325 columns) is managed. In addition, the time when the glass used by the guest was last used (managed in the 1326 column) and the number of times the drink was served to the guest 303 (managed in the 1327 column) are managed. In addition, a flag indicating that the guest has received the request not to lower the tableware (managed in the 1328 column), and a flag indicating that the guest has received the request that no refill is required (managed in the 1329 column). ) Is managed.

The guest profile data table 1330 is a data table managed by the information processing apparatus 120. This table is a data table for managing information about the guest's profile. Specifically, the guest's face image (managed in the 1341 column, the guest's characteristics need not be limited to the face image), the guest's height (managed in the 1342 column), and the guest's weight (managed in the 1342 column). (Managed in 1343 columns) and so on. In addition, the gender of the guest (managed in the 1344 column) and whether or not the guest can drink (managed in the 1345 column, in FIG. 13, "OK" if drinkable, "Shimodo", "lactating", "not yet" if not drinkable. It is managed with reasons such as "adult"), etc.

The decision correspondence management table 1340 is a data table managed by the information processing apparatus 120. As shown in FIGS. 10 and 11, the information processing apparatus 120 is a data table for managing information regarding the correspondence to the guest 303 determined in step S808 based on the information obtained from the image. Specifically, the guest 303 should be ordered for serve and drinks containing alcohol, the guest should be ordered for serve and drink only for soft drinks, or emergency measures should be taken for the guest. (Managed in 1342 columns) and so on. In addition, among the responses to the guest 303 as managed in the action / response management table 1600 described later, which response should be taken is managed.

The removal time management table 1350 is a data table managed by the information processing apparatus 120. As described in step S1009, it is a data table for managing the time as a judgment criterion when deciding whether or not to ask the guest whether or not to remove the glass.

The call determination amount management table 1360 is a data table managed by the information processing apparatus 120. As explained in step S1010, the value related to the remaining amount of drink in the glass, which is a criterion for judging whether or not the drink in the glass is reduced to the extent that the guest needs to speak, is managed. It is a data table for.

The correspondence determination level management table 1370 is a data table managed by the information processing apparatus 120. As explained in FIG. 11, data for managing the criteria for deciding whether to recommend the same drink as the drink in the glass or a different drink to the guest for each genre of the drink item. It's a table. The genres are classified into, for example, the genres of wine A and wine B = "wine". Specifically, the unit price standard per unit quantity (managed in the 1372 column), the profit margin standard per unit quantity (managed in the 1373 column), the expiration date or expiration date standard of the inventory (managed in the 1374 column). Management), inventory criteria (managed in 1375 columns), etc. are managed.

The inventory management table 1380 is a data table managed by the information processing apparatus 120. As explained with reference to FIG. 11, the information necessary for determining whether to recommend the same drink as the drink in the glass or a different drink to the guest is managed for each item. Specifically, the unit price per unit quantity (managed in 1382 columns), the profit margin per unit quantity (managed in 1383 columns), the inventory amount (managed in 1384 columns), the expiration date or expiration date of inventory (managed in 1384 columns). It is managed in columns 1385 and 1386, respectively). In addition, the rate of increase / decrease in the alcohol blood concentration of the guest per unit amount for each item (management etc. is managed in the column of 1387.

FIG. 14 is a configuration diagram showing an example of a series of flows until the alcohol gauge 402 changes according to the amount of alcohol consumed by the guest by associating the guest 303 with the glass 301. The flow will be described in the order of FIGS. 14 (a) to 14 (d).

First, FIG. 14A shows a state in which the information processing device 120 identifies the glass 301, the amount of food and drink in the glass, and a person through steps S901 and S902 of FIG. An example is shown.

In FIG. 14B, the information processing apparatus 120 detects and estimates that a person who is not a cast is holding the glass 301 through steps S903 and S904, and the glass 301 and the guest 303 come together through step S905. This is an example of the associated state. Here, the correspondence between the glass 301 and the guest 303 is shown by the relation line 401. In addition, in FIG. 14B, the information processing apparatus 120 identifies and associates the guest 303 A with the glass 301β and the guest 303 B with the glass 301γ, respectively.

Then, in FIG. 14C, an example of a state in which the information processing device 120 associates the guest 303 instep with the glass 301α through step S903 and step S904 after a further time has passed from FIG. 14B. is there. (Since the guest 303 instep picked up the glass 301α, it is associated with it). In this figure, as shown by the association line 401, it can be identified that not only the glass 301β but also the glass 301α is associated with the guest 303 instep.

FIG. 14D shows an example of a state in which the result of updating the data table is reflected after a further time has passed from FIG. 14C and step S906. Specifically, the value of the alcohol gauge is increasing or the different item serve instruction sign 406 is issued according to the amount of alcohol consumed by Guest 303 B and the remaining amount of drink in Glass 301. Various methods can be considered for updating the history of food and drink. For example, in addition to the method described with reference to FIG. 5, there is a method for analyzing changes in the water level of the drink in the image as shown in FIG. Conceivable. (For example, the difference between the water level X in FIG. 14 (c) and the water level X'in FIG. 14 (d) is obtained to estimate the amount of drink.) Also, when the drink is poured, it is poured. It is also conceivable to count the amount of beverages in a batch first.
FIG. 15 is a configuration diagram showing an example of a mechanism for identifying the glass 301.

The reading code 1501 is information attached to each glass in order to identify the individual of the glass 301 based on the image by the information processing device 120. Specifically, it does not matter what kind of code it is, but it may be a reading code of a specific standard such as a character or a bar code, and a feature amount that can be identified for each glass 301. If there is, the pattern may be used as an identification material. In any case, the individual of the glass 301 may be provided so as to be identifiable by reading the image.

If a method such as providing a reading device for an electronic tag on the table 302 is possible, a tag such as an electronic tag 1502 in which information that can identify an individual is written is attached and the reading device is used for reading. At the same time, a method of identifying the glass 301 by associating it with the image is also conceivable.

FIG. 16 shows the correspondence between the state of the glass 301 and the condition of the history of eating and drinking of the guest, the character display of the alcohol gauge 402, the action pattern to be a guest, and the object to be displayed on the display unit 210 to notify the behavior pattern. It is a block diagram which showed an example of the table.

The action / response management table 1600 is a data table managed by the information processing device 120. As described above, based on the information in the data table shown in FIG. 13, it is determined what kind of response the guest should take, and what kind of object should be superimposed and displayed on the display unit 210 is managed. It is a data table.

For example, it is assumed that the guest 303 of the glass 301 of interest is present and the alcohol blood concentration is estimated to be 0.15% or more and less than 0.3%. In this case, the word "Caution" is displayed on the alcohol gauge 402, the hydration instruction sign 408 is displayed, and the cast 150 is asked to say "How about a soft drink?" Through the mobile terminal 130. Instructions will be given. Furthermore, in the case of the second form of the robot 135 or the mobile terminal 130, based on the decision to deal with the guest, the guest 303 will be asked "How about a soft drink?" Become.

The behavior / response management sub-table 1610 is a data table for supplementing a part of the data table managed by the information processing apparatus 120. In the embodiment of FIG. 16, in the case where the guest of the glass 301 of interest is present and the alcohol blood concentration is estimated to be less than 0.15%, the response to the guest and the display on the display unit 210 are displayed. It is provided to determine which object to do.

For example, consider a state in which the guest 303 uses the glass 301 within a predetermined time specified in the removal time management table 1350, and the guest history management table 1320 does not have the flags of 1328 and 1329. Furthermore, the drink in the glass 301 is insufficient (less than the level of the voice-decision determination amount management table 1360), and the unit price, profit margin, and inventory amount above the level of the response decision level management table 1370 are within the predetermined expiration date. Assume that guest 303 is not a lower door or a minor. In this case, on the display unit 210 of the mobile terminal 130, no characters are displayed on the alcohol gauge, and the same item serve instruction sign 405 is displayed. Then, through the display unit 210 and the voice unit 241 to the cast, take a drink of the same item as the drink in the glass and call out to the guest 303 who is the user of the glass 301. The decision to that effect is identifiablely notified. In the case of the robot 135, the cast is asked "How about another drink?" With the drink in the glass and asks the guest 303, who is the user of the glass 301, to accept the order. It will be.

The different item non-alcoholic serve instruction sign 1611 is a kind of the object displayed in FIGS. 4 and 7. The cast 150 identifiablely displays on the display unit 210 that the guest 303, who cannot drink alcohol, is serving a drink of a different item.

The glass removal instruction sign 1612 is a type of object displayed in FIGS. 4 and 7. Cast 150 can now be identified to inform guest 303, who is the user of the glass 301 of interest, that he has not used the glass for a given period of time and asks guest 303 if it is okay to remove it. It is displayed on the display unit 210.

As described above, in the present embodiment, the information processing device 120 manages all the information tables and decides the correspondence to the guest, and the mobile terminal 130 and the robot 135 receive the decision and perform the processing. It has become. However, a part of the data table may be provided so as to be managed by the mobile terminal 130 or the robot 135. In this case, the information processing device 120 is provided with only a function of identifying the state based on the image information and updating the history of eating and drinking of the guest, and a function of determining how to act is provided by the mobile terminal 130 or the robot. It will be in the form of having 135. If provided in this way, it is suitable for reducing the load on the information processing apparatus 120.

As described above, the present invention is intended to search for a vessel in which the amount of food and drink contained is below a certain level, and to provide a mechanism for determining an appropriate approach according to the situation.

Then, as an example of the solution, an image including the glass 301 is acquired (acquisition means), the user and the amount of the drink poured into the glass are specified (specific means), and the amount that needs to be refilled together with the image. We proposed a mechanism to display an object that can identify the owner of the glass.

Furthermore, the above options for dealing with guests are managed (management means), and it is possible to decide whether or not a refill is necessary or not (decision means) according to the amount of food and drink. ..

In addition, the above decision is set to manage factors such as drink items, unit price per quantity for each item, profit margin per quantity, inventory amount, expiration date, expiration date, etc., and based on these factors, It was set up to decide how to respond to guests.

In addition, the history of eating and drinking of the glass user is also provided so that the user's alcohol blood concentration, the number of times the drink is provided, the total customer unit price and profit margin, the time since the start of eating and drinking, and the last eating and drinking It was set up to decide how to respond to guests based on the time that has passed since the event was completed.

In addition, based on the decided response, we asked guests if they needed an order, provided drinks based on the order, and provided instructions to execute these.
In addition, when the glass user is absent, the guest is not asked if an order is required.
In addition, the glasses that were associated and determined by the user were provided so that even if another person picked them up, they would not be associated again.
In addition, when the cast was picked up, the cast was provided so as not to be associated with the user of the glass.
In addition, when a user identifies a glass that has not been used for a predetermined time, a decision is made to ask the user if the glass can be cleaned up.
In addition, when a glass user offers to refill the glass, it is decided not to call for an order.

By establishing the above mechanism, it is possible to identify guests who may need refills and appropriately determine and implement what kind of response (calling, etc.) should be given to those guests. It will be possible to provide a mechanism that can be done.

Further, a recording medium on which a program that realizes the functions of the above-described embodiment is recorded is supplied to the system or device, and a computer (or CPU or MPU) of the system or device reads and executes the program stored in the recording medium. Needless to say, the object of the present invention is achieved by this.

In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium storing the program constitutes the present invention.

Recording media for supplying programs include, for example, flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, and silicon. Disks, solid state drives, etc. can be used.

Further, by executing the program read by the computer, not only the function of the above-described embodiment is realized, but also the OS (operating system) or the like running on the computer is actually operated based on the instruction of the program. Needless to say, there are cases where a part or all of the processing is performed and the processing realizes the functions of the above-described embodiment.

Further, after the program read from the recording medium is written in the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the function expansion board is based on the instruction of the program code. It goes without saying that there are cases where the CPU or the like provided in the function expansion unit performs a part or all of the actual processing, and the processing realizes the functions of the above-described embodiment.

Further, by downloading and reading a program for achieving the present invention from a server, database, or the like on the network by a communication program, the system or device can enjoy the effect of the present invention. It should be noted that all the configurations in which each of the above-described embodiments and modifications thereof are combined are also included in the present invention.

110 Imaging device 120 Information processing device 130 Mobile terminal 135 Robot 201 CPU
202 ROM
203 RAM
210 Display section (display screen, display terminal)
211 External memory 221 Camera unit 231 Drive unit 241 Voice unit 301 Glass 302 Table 303 Guest 304 Seat 401 Related line 402 Alcohol gauge 403 Guest name 404 Replacement stop instruction sign 405 Same item Serve instruction sign 406 Different item Serve instruction sign 407 Same item Non-alcohol serve instruction sign 408 Hydration instruction sign 601 Concentration gauge 602 Attention boundary line 603 Warning boundary line 701 Emergency response command mark 1300 Glass condition management table 1310 Cast management table 1320 Guest history management table 1330 Guest profile data table 1340 Decision response management table 1350 Removal time management table 1360 Calling decision quantity management table 1370 Response decision level management table 1380 Inventory management table 1600 Action / response management table 1610 Action / response management sub-table 1611 Different item non-alcoholic serve instruction sign 1612 Glass removal instruction sign

Claims (8)

An acquisition method for acquiring image data obtained by imaging a person and tableware,
Using the result of analyzing the acquired image data, an estimation means for estimating whether the person picked up the tableware, and an estimation means.
A registration means for registering information in which the person and the tableware are associated with each other according to the estimation result.
An information processing device characterized by having. Further having an authentication means for authenticating the person using the acquired image data,
The information processing apparatus according to claim 1, wherein the registration means registers the person and the tableware in association with each other according to the result of the authentication. The information according to claim 2, wherein the registration means registers the person and the tableware in association with each other when the result of the authentication indicates that the person is not a predetermined person. Processing equipment. Said estimating means, by analyzing the acquired image data, and estimates the attributes of the person,
The information processing apparatus according to any one of claims 1 to 3, wherein the registration means further registers the information associated with the estimated attributes of the person. Said estimating means, by analyzing the acquired image data, and estimates the food of the dishes,
The information processing apparatus according to any one of claims 1 to 4, wherein the registration means further registers the information associated with the estimated food and drink. The information processing device according to any one of claims 1 to 5, further comprising a notification means for notifying the device of the clerk based on the registered information. The acquisition process to acquire the image data obtained by imaging a person and tableware,
An estimation step of estimating whether the person picked up the tableware using the result of analyzing the acquired image data, and an estimation step.
A registration process for registering information in which the person and the tableware are associated with each other according to the estimation result.
A method for controlling an information processing device, which comprises. A program characterized in that a computer functions as the information processing device according to any one of claims 1 to 6.

Images