JP7155295B2 - Image recognition device, image recognition program, and image recognition method - Google Patents

Description

The present invention relates to an image recognition device, an image recognition program, and an image recognition method.

In the business of providing food and drink to customers, it is important to provide services to customers in a timely manner from the viewpoint of improving customer satisfaction and efficient business management.

Therefore, it is necessary for the employees to keep track of the progress of the meal at each table by repeating visual confirmation while going around each table in the store.

On the other hand, there is also known a method in which a customer-operated call button or ordering tablet is placed on each table, and the timing of service instructions is left to the customer's discretion.

In addition, Patent Document 1 discloses that “a container used for a meal and an image of the food contained in the container are photographed, the area of the remaining food is measured from the image, and it is determined whether or not the meal is finished, and the meal is finished. A technique to the effect that "information indicating is output" is disclosed.

JP 2015-138452 A

In addition to serving customers, employees also have jobs such as ordering ingredients, preparing food, cooking, cleaning up afterward, clearing cash registers, and cleaning the store. For this reason, there is a problem that a method in which an employee carefully visually confirms the progress of meals at each table and constantly judges the timing of service instructions increases the burden on the employee.

On the other hand, the method of using a call button or ordering tablet has the problem that it is difficult to further improve customer satisfaction because employees respond passively according to customer requests.

In addition, the technique of Patent Document 1 only discloses a method for measuring the area of the remaining amount of food, and does not consider flexible determination of the progress of various meals.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a technique for providing services with high customer satisfaction.

In order to solve the above problems, one of the typical image recognition devices of the present invention includes an image acquisition unit that acquires an image of an eating and drinking space, and a situation related to eating and drinking for a group of images of the eating and drinking space (hereinafter referred to as "eating situation"). ”) is machine-learned, the image recognition unit recognizes the eating situation in the image acquired by the image acquisition unit, and the progress status of the customer's eating and drinking is determined based on the image-recognized eating situation. and a state determination unit.

The image recognition technology of the present invention makes it possible to provide services with high customer satisfaction.
Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

It is a figure which shows the structure of a business support system and an image recognition apparatus. 4 is a diagram showing the configuration of an image recognition section and a state determination section; FIG. FIG. 4 is an explanatory diagram showing an example of learning data; It is a flowchart explaining operation|movement of an image recognition apparatus. FIG. 4 is a diagram showing an image of a dining space acquired from an imaging device; It is a figure which shows a detection area. It is a figure which shows the detection area which performed block division. FIG. 10 is a diagram showing a detection area for each customer; FIG. 10 is a diagram showing inference results by a learning model for each detection area; It is a figure explaining a 3D convolutional neural network. It is a figure which shows the result of the object detection by YOLO, SSD, etc. FIG. It is a figure which shows the class classification output from a learning model. It is a figure which shows a state transition model. FIG. 10 is a diagram showing an example of transition conditions of a state transition model; It is a figure which shows an example of the service instruction|indication corresponding to a progress state. It is a figure which shows the example of a display of a service instruction|indication.

Embodiments related to the present invention will be described with reference to the drawings.
<Configuration of Example>
FIG. 1 is a block diagram showing the overall configuration of a business support system 1 and an image recognition device 100. As shown in FIG.
The image recognition apparatus 100 may be configured by a computer (including an information processing system, etc.) having a CPU (Central Processing Unit), a memory unit, a communication unit, and the like.

By executing an image recognition program in this hardware, various functions of the image recognition apparatus 100, which will be described later, are realized.

Part or all of the hardware may be replaced by a DSP (Digital Signal Processor), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), or the like. Also, part or all of the hardware may be centrally or distributedly arranged in a server on the network to form a cloud, and shared use by multiple people via the network.

As shown in FIG. 1 , the business support system 1 includes an image recognition device 100 , an imaging device 101 , a display/instruction terminal 102 and a display output device 103 . Furthermore, the image recognition device 100 includes a video acquisition unit 110 , an image recognition unit 120 , a state determination unit 130 , a data communication unit 140 , a recording control unit 150 , a display control unit 170 and a recording device 160 .

The imaging device 101 is composed of one or more cameras, and is arranged on a ceiling, a wall, a tabletop, or the like so as to photograph the dining space. For example, the imaging device 101 can be a CCTV camera, a camera built into a robot installed on a desk, a fisheye lens camera, a web camera, or the like.

The video acquisition unit 110 acquires the video of the dining space from the imaging device 101 as image data in a one-dimensional array or a two-dimensional array via wireless, wired cables, or a network. The video acquisition unit 110 also acquires video groups of the eating and drinking space from an external video database and the recording device 160 .

The image recognition unit 120 performs image recognition on the image acquired by the image acquisition unit 110 regarding the eating situation, the menu situation, and the presence or absence of a person.

In the present disclosure, "meal status" indicates the progress of eating and drinking, and indicates, for example, "meal not provided", "meal provided", "meal completed", and the like.

In addition, in the present disclosure, "menu status" indicates the status of the menu viewed by the customer, such as "no menu", "menu present (closed state)", "menu present (open state)", etc. point to

Further, "with menu (closed state)" may include not only the situation where the menu book is closed and the inside pages cannot be browsed, but also the situation where the menu cannot be browsed because the menu is housed in a partition or the like. In "with menu (open state)", in addition to the situation where the middle page can be viewed with the menu book open, the situation where the menu can be viewed after the menu is taken out from the screen, etc., and the situation where the customer opens the menu. You can also include the situation you are in.

In addition, in the present disclosure, the “state of the presence or absence of a person” indicates the state of the presence or absence of a person, and indicates, for example, “presence” or “empty seat”.
Furthermore, "at a seat" may include not only the state of sitting on a chair, but also the state of standing in a stand-up dining space.

The status determination unit 130 determines the progress status of the customer's eating and drinking based on the result of image recognition, and notifies the timing of the service instruction corresponding to the progress status.

The data communication unit 140 performs data communication among the state determination unit 130, the image recognition unit 120, the display/instruction terminal 102, a data center (not shown), and the like via wireless, wired cable, or network.

The display/instruction terminal 102 is an information processing device such as a POS terminal (point-of-sale information management terminal), and receives the timing of the service instruction notified by the state determination unit 130 via the data communication unit 140 to receive an event or an instruction. The employee in charge of the customer is notified by image, vibration, voice, etc.

Customer information (order data, payment data, etc.) input by the employee to the display/instruction terminal 102 is transmitted to the state determination section 130 via the data communication section 140 . Furthermore, the information input by the employee to the display/instruction terminal 102 (such as notification confirmation or notification stop operation, information as to whether or not a service instruction has been implemented, etc.) is also transmitted to the state determination unit 130 via the data communication unit 140. be done.

The recording control unit 150 performs recording control (on/off of recording, frame rate control, compression ratio control, recording interval control, etc.) for the recording device 160 based on the determination by the state determination unit 130 .

The recording device 160 records the image acquired by the image acquisition unit 110 according to the recording control by the recording control unit 150 . Furthermore, the recording device 160 also records the results of image recognition and state determination by the image recognition device 100 .

The display control unit 170 displays the results of image recognition, state determination, and service instruction by the image recognition device 100 on the display output device 103 together with the image acquired by the image acquisition unit 110 .

FIG. 2 is a block diagram showing the configuration of the image recognition section 120 and the state determination section 130. As shown in FIG.
In the figure, the image recognition unit 120 includes a parameter setting unit 121 , a preprocessing unit 122 , a feature extraction unit 123 , a recognition unit 124 and a learning model 125 . State determination section 130 includes state transition section 131 and instruction output section 132 .

The parameter setting unit 121 has a function of setting, changing, reading, and saving parameters necessary for image processing.

The preprocessing unit 122 performs trimming processing and masking processing of the detection region on the image acquired by the image acquiring unit 110 according to the parameters of the parameter setting unit 121 . Furthermore, the preprocessing unit 122 may perform preprocessing such as a smoothing filter, an edge enhancement filter, and density conversion in order to reduce noise, flicker, and the like on the video. Furthermore, color conversion processing such as RGB color or monochrome may be performed depending on the purpose of image recognition. Furthermore, in order to reduce the processing load, the image data may be reduced.

The feature extraction unit 123 performs image processing for enhancing or extracting image features related to image recognition in order to improve the accuracy of subsequent image recognition on the preprocessed video of the detection region.

The recognition unit 124 performs image recognition using the learning model 125 on the image of the detection region for which the feature extraction has been performed by the feature extraction unit 123 .

The learning model 125 is configured by a convolutional neural network or the like. The learning data of this learning model 125 is created by collecting a group of images of the detection area in advance, and giving class classifications of presence/absence of people, eating situations, and menu situations as teacher values.

For example, the following class classification is the teacher value.
<Meal status>
・Meal not served ・Meal served ・Meal finished <menu status>
・With menu (closed state)
・With menu (open state)
・No menu
-Seats present -Seats vacant FIG. 3 is an explanatory diagram showing an example of learning data to which such teacher values are assigned.

Group A in the figure is a video group to which teacher values such as vacant seats, no meals provided, and no menus are given.

Group B is a video group to which teacher values such as presence, meal not provided, and menu available (open state) are assigned.

Group C is a video group to which a teacher value of being seated, having a meal served, and having no menu is assigned.

Group D is a video group to which teacher values such as being seated, having finished eating, and having no menu were assigned.

Machine learning (for example, back propagation method) is performed so as to minimize the error from the teacher value in the output layer of the learning model 125 while giving such a video group of learning data to the input layer of the learning model 125 .

By such machine learning, when the image of the detection area is given to the input layer of the learning model 125, the output layer of the learning model 125 is "class classification of person presence", "class classification of eating situation", and "class classification of menu situation". Inference results such as "class classification" will be output.

The state transition unit 131 has a state transition model that models the state transition regarding customer eating and drinking from entering the store to leaving the store. The state transition unit 131 applies the result of image recognition by the recognition unit 124 (class classification of presence/absence of person, class classification of eating situation, class classification of menu situation, etc.) to the state transition model. result) is applied as a transition condition, and the state of the customer is changed to determine the current state of progress regarding eating and drinking of the customer.

In addition, the state transition unit 131 predicts the customer's state transition based on the passage of time, and transitions to the next progress state.

Furthermore, the state transition unit 131 acquires customer information such as the customer's order and payment that are input to the display/instruction terminal 102 via the data communication unit 140 . The state transition unit 131 preferentially determines the progress state of the customer based on this customer information.

A group of information causing such a state transition is backed up in the history 160 a in the recording device 160 via the recording control section 150 .

The instruction output unit 132 notifies the display/instruction terminal 102 of the service instruction via the data communication unit 140 according to the customer's progress.

<Operation of Example>
FIG. 4 is a flowchart for explaining the operation of the image recognition device 100. As shown in FIG.
The operation of the embodiment will be described below along the step numbers shown in FIG.

Step S<b>101 : The image acquisition unit 110 acquires an image of the dining space from the imaging device 101 . FIG. 5 is a diagram showing an acquired image 301 of the dining space. An image 301 includes a plurality of detection areas 302 to 305 for each table (table and seat set).

Step S102: After correcting the distortion of the image 301, the parameter setting unit 121 sets divisions for trimming the detection areas 302 to 305 in table units. Such division setting may be set by the user specifying a boundary line between tables. Alternatively, the parameter setting unit 121 may recognize the color and shape of the table from the image 301 to automatically set the partitions for each table.

In addition, the parameter setting unit 121 may set image quality correction (luminance correction, color correction, gamma correction, etc.) for each of the detection areas 302 to 305 for each table based on differences in shooting conditions such as illumination of each table. good.

Furthermore, the parameter setting unit 121 may perform mask setting for excluding fixed areas unrelated to meals and menus, such as decorations on the table.

The preprocessing unit 122 trims the detection areas 302 to 305 in table units from the image 301 based on the above-described series of settings by the parameter setting unit 121 . FIG. 6 is a diagram showing a cropped sensing region 302. As shown in FIG.

To simplify the explanation, only the processing of the detection area 302 will be described from here on, but the remaining detection areas 303 to 305 are also subjected to similar processing in parallel or sequentially.

Step S103: The preprocessing unit 122 further divides the detection area 302 for each table.
FIG. 7 is a diagram showing how the table-based detection area 302 is divided into block-based detection areas 501 of m columns×n rows. By subdividing the processing unit in this way, it is possible to reduce the processing unit of the feature detection and image recognition in the latter stage and increase the speed by parallelization.
FIG. 8 is a diagram showing how detection areas 601 to 603 are determined for each customer. The preprocessing unit 122 detects the presence or absence of a customer by image processing such as a background subtraction method, face detection, and learning model. The preprocessing unit 122 determines the detection areas 601 to 603 so as to cover the area of the detected customer and the area of food and drink and menus provided to the customer. By dividing the detection areas 601 to 603 for each customer's activity area in this way, image recognition for each customer becomes possible.

Furthermore, in the case of a chair where one person sits instead of a chaise longue where a plurality of people can freely sit as shown in FIG.

The types of detection regions described above are selectively used according to the design and processing load of the feature extraction unit 123 and the recognition unit 124 in the latter stage and circumstances specific to the eating and drinking space.

Step S104: The feature extracting unit 123 extracts, for example, the amount of change from the previous image by the background subtraction method or the like, the motion vector based on the optical flow, and the edge information contained in the image as the image feature, for the detection area, as the motion feature. , color information, luminance information, and so on. The extracted features are output to the recognition unit 124 as video data of the detection area.

Note that the feature extraction unit 123 may attach a feature amount and a rule base focusing on the shape of a person or a plate, such as the number of circles or straight lines detected by the Hough transform, to the detection region. Further, the feature extraction unit 123 may extract feature information of human body parts such as hands and feet by skeleton recognition (for example, OpenPose) of a person's hands and feet, and attach the feature information to the detection area.

The recognition unit 124 inputs the multidimensional array data of the detection region input from the feature extraction unit 123 to the input layer of the learning model 125 .

Inside the learning model 125, the multi-dimensional array data of the detection region is subjected to array operations by convolution layers, pooling layers, and fully connected layers, and non-linear operations by activation functions. Feature information other than the image attached to the detection region may be input to the fully connected layer.

In this array operation, the coefficient values and bias values of each layer created by the machine learning described above are used, so that the output layer of the learning model 125 includes the above-described “classification of presence/absence of people” and “meal situation”. Inference results such as "class classification of menu" and "class classification of menu situation" will be output.

FIG. 9 is a diagram showing inference results by the learning model 125 for each detection area 601, 602, 603. FIG.

In addition, in order to correctly estimate the eating situation, since the actions of using tools such as forks are also useful information, the learning model 125 is a 3D convolutional neural network that simultaneously convolves multiple frames in the past (See FIG. 10) or the like may be used to perform inference processing including motion. In this case, the learning model 125 can detect the customer's movements such as "the customer's movement during the meal", "the customer's movement to read the menu", and "the customer's movement to look at the smartphone during the meal".

In addition, the learning model 125 is applied not only to the detection regions for each customer, such as the detection regions 601, 602, and 603, but also to the uniformly divided block-shaped detection region 501 as shown in FIG. can do. In this case, for example, even if only one person is shown, a plurality of blocks will indicate "presence". Therefore, by integrating adjacent blocks indicating the same "presence", the video range for one person can be determined. Further, by determining the majority of the results of class classification for the block groups included in the video range for one person, it is possible to classify the video range for one person.

In addition, if it is not necessary to divide the state of each table for each individual, that is, if you want to manage the state of the entire table, input the entire division of each desk into the learning model 125 like the detection area 302 shown in FIG. It is also possible to perform inference processing by doing so.

Furthermore, if there is a customer need to prevent forgetting to take out the cup of water or drink, for example, object detection such as You Only Look Once (YOLO) and Single Shot Multibox Detector (SSD) By applying the processing to the learning model 125 and defining classes that can exist around the desk, such as "drinks", "meal", and "persons", and making them learn, the position and presence/absence of objects as shown in FIG. may be recognized.

FIG. 12 is a diagram showing the classification performed by the learning model 125. As shown in FIG.
In the figure, the classification consists of the following classes.
<Meal status>
・Meal not served ・Meal served ・Meal finished <menu status>
・With menu (closed state)
・With menu (open state)
・No menu
・Available ・Vacant

Step S105: FIG. 13 is a diagram showing a state transition model used by the state transition section 131. FIG. In the figure, the state transition model has the following progress states (1) to (6). (1) Enter (2) Menu selection (3) Wait (4) During meal (5) After meal (6) Exit

These progress states undergo state transitions (solid line arrows shown in FIG. 13) in ascending order of (1) to (6). This ascending state transition is a standard state transition from when a customer enters the store to when he/she leaves the store. In addition, the state transition model has exceptional state transitions such as the dotted line arrows shown in FIG.

FIG. 14 is a diagram showing an example of transition conditions of this state transition model.
The state transition unit 131 applies the image recognition results of the recognition unit 124 to the state transition model (inference results such as “person presence/absence class classification”, “eating situation class classification”, and “menu situation class classification”). ) are combined to meet the transition condition, the progress state of the customer is changed. This state transition determines the current state of progress regarding the customer's eating and drinking.

For example, when the class classification of the presence of a person changes from vacant to seated in a detection region that was previously in the progress state of "exit", the progress state is initially set to "enter".

Furthermore, in the progress state of "entering", when the transition condition that the menu is opened in the presence state is satisfied, the progress state is changed from "entering" to "menu selection".

In addition, in the progress state of "menu selection", when the transition condition that the opened menu is closed is satisfied, the progress state is changed from "menu selection" to "waiting".

Furthermore, in the progress state of "waiting", when the transition condition that the meal status becomes "meal provided" is satisfied, the progress state is changed from "waiting" to "eating".

In the progress state of "during meal", when the transition condition that the meal status is the end of meal is satisfied, the progress state is changed from "during meal" to "after meal".

Furthermore, when the transition condition that the menu is opened is satisfied in the progress state of "after meal", the progress state is changed from "after meal" to "menu selection".

In addition, in the progress state of "after meal", when the transition condition that the seat is changed from seated to vacant is satisfied, the progress state is changed from "after meal" to "exited".

Step S106: The state transition unit 131 measures the elapsed time after the state transition. Based on this passage of time, the possibility of the next state transition is predicted. With this prediction, the state transition unit 131 can advance the progress state to the next even if the result of image recognition does not change.

Step S107: When the state transition unit 131 acquires customer information from an information processing device such as the display/instruction terminal 102 that collects customer information such as orders through the data communication unit 140, the state transition unit 131 preferentially updates the progress state of the state transition model. (forced).

For example, as shown in FIG. 14, when the customer information of "order completed" is obtained in the progress state of "entering" or "menu selection", the progress state is forcibly changed to "waiting".

Step S108: The instruction output unit 132 determines a service instruction (service content: timing of service provision) corresponding to the progress state for each detection area.
FIG. 15 is a diagram showing an example of a service instruction corresponding to progress.

For example, a service instruction (provide drinking water cup: after entering the store) is determined in accordance with the progress of "entering the store".

Further, a service instruction (confirmation of order: after a predetermined time has elapsed since menu selection) is determined in accordance with the progress of "menu selection".

In addition, service instructions (meal provision: after completion of cooking after waiting) are determined in accordance with the progress state of "waiting".

Further, the service instruction (collection of tableware: after eating) is determined in accordance with the progress of "after eating".

Also, a service instruction (tidying up instruction: after leaving the store) is determined in accordance with the progress of "leaving the store".

Further, the service instruction (delivery of dessert after meal: when there is an order for dessert after meal and after meal) is determined in accordance with the progress of "after meal".

Step S109: Since service instructions are generated for each customer or table, there are cases where multiple service instructions are concentrated in a short period of time. Therefore, the instruction output unit 132 sets a priority level for service instructions generated between customers and tables.

In general, services for customers (such as "order confirmation," "delivery of meals," and "delivery of desserts") are prioritized over services for restaurants (such as "collecting tableware"). becomes higher.

Step S110: The instruction output unit 132 plans a customer service schedule for each employee by changing the timing of service instructions according to the priority level.

Step S111: The instruction output unit 132 determines whether or not the customer service schedule for each employee has a vacant time equal to or longer than a predetermined time. If there is vacant time, the instruction output unit 132 moves the operation to step S112. Otherwise, the instruction output unit 132 moves the operation to step S113.

Step S112: The instruction output unit 132 informs the employee via the data communication unit 140 of work other than customer service (ordering of ingredients, preparation, cooking, cleanup, checkout, cleaning of the store, etc.) and the timing of breaks, according to the vacant time. The display/instruction terminal 102 of the member is notified.

Step S113: The instruction output unit 132 notifies the employee's display/instruction terminal 102 via the data communication unit 140 of the service instruction and the location information of the table to be serviced or the customer at the scheduled timing of the customer service schedule. do.

The service instruction may be displayed on the display output device 103 in the backyard as map information 700 as shown in FIG. For example, when the progress status of "Exit" occurs, "Waiting for checkout" is displayed at the checkout location on the map. By placing such a display output device 103 in a place where employees can easily see it, the entire operation can be grasped, and even employees who do not have an instruction terminal can easily see the work to be processed by each employee. .

Step S114: The recognition unit 124 performs image recognition of the replenishment status regarding the remaining amount of drinking water in the cup, together with the image recognition of the meal status. The state transition unit 131 determines whether the drinking water in the cup is in a shortage state according to the replenishment state.

The instruction output unit 132 sends a service instruction to replenish the drinking water and the location information of the table to be serviced or the customer to the employee's display/instruction terminal via the data communication unit 140 in response to the determination of the shortage state. 102.

Note that the replenishment is not limited to drinking water, and may be drinks, tea, seasonings, garnishes, rice, bread, chopsticks, forks, knives, etc. in the desktop case.

After completing the above-described series of operations (steps S101 to 114), the image recognition apparatus 100 returns the operation to step S101, so that the employee's customer service support is repeatedly performed continuously.
<Effect of Example>

(1) In the embodiment, a machine-learning model is used to image-recognize the customer's eating situation from the video of the eating and drinking space. Therefore, unlike the fixed image recognition such as area measurement of remaining amount of food and drink in Patent Document 1, it is possible to flexibly recognize the customer's meal status by image recognition.

(2) There is no way to determine whether or not the customer has the intention of "ordering" in the area measurement of the remaining amount of food and drink in Patent Document 1. On the other hand, in the embodiment, image recognition is performed on the menu status in addition to the meal status. As a result, it is possible to comprehensively determine whether the customer has an intention to place an order by detecting when the menu is open, and to accurately grasp the customer's intention to place an order. Therefore, by employing the embodiment, it is possible for the employee to ask the customer for the order before the customer calls the employee for the order. Therefore, it becomes possible to improve customer satisfaction.

(3) In addition, the meal situation changes immediately after "meal provision", whereas the menu situation changes after "entering the restaurant". In this way, by adding the menu status to the meal status, it becomes possible to notify service instructions over a wider range of time periods.

(4) In addition, in the embodiment, the progress of the customer's eating and drinking is represented by a state transition model. Therefore, the state of progress is changed based on the current state of progress and the result of image recognition (transition condition). For example, by using image recognition to detect that a menu has been opened in the "after meal" progress, it is possible to accurately capture more detailed order intentions, such as the customer's intention to place an "additional order." Become. In general, "additional orders" are not always generated compared to "first orders", so in order to accurately grasp the intention of the order, the employees need to have a long experience in customer service. However, by adopting the embodiment, even an inexperienced employee can ask the customer for an additional order at an appropriate time. Therefore, it is possible to further improve customer satisfaction.

(5) Furthermore, in the embodiment, the state transition model is changed by prediction based on the passage of time. Therefore, even if the image recognition result cannot be obtained for some reason, the customer's progress progresses and the employee can receive service instructions such as ordering and serving. Therefore, in the embodiment, it is possible to prevent the customer from being inconvenienced by the service instruction being stopped.

(6) In the embodiment, the state transition model preferentially (forcibly) transitions based on the customer information collected from the POS terminal or the like. Therefore, even if the image recognition result cannot be obtained for some reason, the customer's progress progresses and the employee can receive service instructions. Therefore, in the embodiment, it is possible to prevent the customer from being inconvenienced by the service instruction being stopped.

(7) Furthermore, in the embodiment, the detection area for image recognition is determined according to the presence or absence of a customer. Therefore, image recognition is not performed wastefully for regions where there are no customers, and efficient processing becomes possible.

(8) In addition, in the embodiment, a customer service schedule is created in consideration of the priority levels of service instructions for a plurality of customers or customer groups. Therefore, it is possible to alleviate the situation in which a large number of service instructions are generated intensively and cause confusion. Conventionally, in order to avoid such confusion, it is necessary for employees to have deep and long experience in customer service. However, by adopting the embodiment, even an inexperienced employee can perform service orders in order from the highest priority level. Therefore, it is possible to further improve customer satisfaction.

(9) Furthermore, in the embodiment, free time is predicted based on the customer service schedule. Therefore, it is possible to instruct the employee not only about the service but also about the free time. Therefore, by adopting the embodiment, it is possible to prevent idle time from being generated in the behavior of employees, and to make efficient use of idle time while being conscious of it.

(10) In addition, in the embodiment, it is possible to perform image recognition on the replenishment status of tea, drinking water, etc., and to notify the replenishment instruction in a timely manner. Therefore, the adoption of the embodiment eliminates the need for the employee to visually confirm the lack of replenishment many times, and it is possible to reduce the labor of the employee.

(11) As described above, in the embodiment, the employees and managers of the store can receive service instructions at the necessary timing regarding the next work to be done based on the customer's meal situation. Become.

(12) Furthermore, in the embodiment, since the employee receives service instructions at the necessary timing, there is little need to pay attention such as visually confirming the customer's condition, and other tasks such as cash register, cooking/cleaning, cleaning, etc. It will be possible to concentrate on the work, leading to improvement of work efficiency.

(13) As a result of this improvement in work efficiency, the waiting time until the meal is served is shortened in the embodiment. Therefore, efficient and speedy business execution with high customer satisfaction can be expected.

<Supplement to Examples>
In addition, in the embodiment, the learning model 125 of the neural network has been described. However, the invention is not so limited. The present invention may be any learning model capable of classifying, and can also use a learning model such as decision tree learning.

In addition, the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations.

Also, it is possible to replace part of a configuration with another configuration.

Furthermore, it is possible to add, delete, or replace a part of the configuration of the embodiment with another configuration.

REFERENCE SIGNS LIST 1 business support system 100 image recognition device 101 imaging device 102 display/instruction terminal 103 display output device 110 video acquisition unit 120 image recognition unit 121 parameter setting unit 122 Preprocessing unit 123 Feature extraction unit 124 Recognition unit 125 Learning model 130 State determination unit 131 State transition unit 132 Instruction output unit 140 Data communication unit 150 Recording control unit 160... Recording device, 160a... History, 170... Display control unit

Claims (12)

an image acquisition unit that acquires an image of the eating and drinking space;
Using a learning model obtained by machine-learning the classification of the presence/absence of people, the classification of eating situations, and the classification of menu situations for a group of images of the eating and drinking space, the classification of the presence/absence of people for the images acquired by the image acquisition unit. , an image recognition unit for image recognition of the class classification of the meal situation and the class classification of the menu situation;
a state determination unit that determines a progress state regarding the customer's eating and drinking by a state transition combining the image-recognized person presence/absence class classification, the meal situation class classification, and the menu situation class classification;
An image recognition device comprising: The image recognition device according to claim 1,
The state determination unit
Having a state transition model that models the state transition of the progress state, and determining the progress state by applying the progress state to the state transition model using the recognition result of the image recognition unit as a transition condition to cause the state transition. An image recognition device characterized by: In the image recognition device according to claim 2,
The state determination unit
An image recognition device that predicts the state transition regarding the customer's eating and drinking based on the passage of time. In the image recognition device according to any one of claims 1 to 3,
The state determination unit
An image recognition apparatus that acquires the customer information from an information processing apparatus that collects customer information regarding the customer's order or settlement , and determines the progress state by prioritizing determination based on the customer information. In the image recognition device according to any one of claims 1 to 4,
The image recognition device, wherein the image recognition unit detects the presence or absence of the customer in the image, and determines a detection area for image recognition in the image according to the presence or absence of the customer. In the image recognition device according to any one of claims 1 to 5,
The state determination unit
An image recognition apparatus, characterized in that a service instruction corresponding to the progress state is notified. In the image recognition device according to any one of claims 1 to 6,
The state determination unit manages the priority level of the service instruction corresponding to the progress state for each customer or for each customer group, and notifies the service instruction corresponding to the progress state according to the priority level. An image recognition device characterized by: In the image recognition device according to any one of claims 1 to 7,
The image recognition device, wherein the state determination unit determines a vacant time related to customer service based on the progress state, and notifies timing of work other than customer service or break according to the vacant time. In the image recognition device according to any one of claims 1 to 8,
The image recognition unit
image recognition of the situation regarding replenishment to be replaced or replenished in the eating and drinking space (hereinafter referred to as "replenishment status");
The state determination unit
An image recognition apparatus, comprising: judging the shortage of the replenishment based on the status of the replenishment, and notifying a service instruction corresponding to the shortage of the replenishment. An image recognition program that causes a computer to function as the image recognition unit and the state determination unit according to any one of claims 1 to 9. an image acquisition step of acquiring an image of the eating and drinking space;
Using a learning model obtained by machine-learning classification of presence/absence of a person, classification of eating situation, and classification of menu situation for a group of images of an eating and drinking space, classifying the presence/absence of a person for the video acquired in the video acquisition step. , an image recognition step of image-recognizing the class classification of the meal situation and the class classification of the menu situation;
a state determination step of determining a progress state of the customer's eating and drinking by a state transition combining the image-recognized person presence/absence class classification, the meal status class classification, and the menu status class classification;
An image recognition method comprising: In the image recognition method according to claim 11,
The state determination step includes:
Having a state transition model that models the state transition of the progress state,
The image recognition method, wherein the progress state is determined by applying the progress state to the state transition model using a recognition result of the image recognition step as a transition condition to cause a state transition.

Images