JP2005070908A - Method and system for measuring meal intake for hospital - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method for measuring meal intake in a hospital by comparing images before and after meals, and calculating a caloric intake from a preliminarily prepared food component table to automatically measure and record meal intake accurately and quickly by a simple operation. <P>SOLUTION: After the end of a meal, a dining table A of each person is successively stored in an imaging apparatus 2, and the dining table A is automatically photographed from the upper part by a digital camera 8 (a step 1). Only the part of tableware B is extracted from the image of the dining table A (a step S2). Information relating to the tableware B is acquired from a tableware database, and the type of the tableware B is specified. Furthermore, food C remaining on each tableware B is extracted (a step 3). Then, the area of each residual food C is calculated (a step 4). Finally, the area of the food C before a meal on each tableware B is compared with the area of the food remaining after a meal so that the caloric intake and nutrient of the food C can be calculated from the difference of the both areas. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a method and a measurement system for measuring an intake amount when an inpatient or an elderly person ingests a meal in a hospital or a nursing facility (hereinafter simply referred to as “in-hospital”).

At present, for example, the general method for dietary management of hospitalized patients is that a dietician visually confirms the pre-meal state of the meal and the post-meal state when the patient has finished eating for each individual patient, and compares the meal with the food composition table. The amount of intake is measured. Therefore, the standard of measurement is the amount of scale, which varies depending on the judge (usually a dietitian) and cannot be said to be constant. Considering the number of inpatients and the number of meal menus, as well as the amount of meal intake that varies from patient to patient, and the work of keeping them as records, enormous effort and time are spent. For this reason, there is a strong demand from hospital staff who want to measure and record these operations on machines instead of dieticians.
There is prior art related to a neural network (see, for example, Patent Document 1).
JP-A-11-96431

  As described above, the diet management method for in-patients in the hospital visually confirms two states, the state before the dietitian distributes the meal to the patient and the state after the patient finishes the meal. In addition, the intake for each individual patient is measured.

  The problem with the current dietary intake measurement is that the dietician visually determines the patient's dietary intake from the remaining amount of food when measuring the amount of meal consumed by the patient. For this reason, the measured value is ambiguous and the measurement standard varies depending on the dietitian. Further, as shown in FIG. 20, since all diets of individual patients are measured with the eyes of a nutritionist, the work efficiency is low. Furthermore, there are various problems such as not being digitized in the management of individual patient measurement data.

  The present invention was made to solve the above-mentioned problems in conventional diet management, and its basic concept is to compare pre-meal and post-meal images and calculate caloric intake from a pre-prepared food composition table. Therefore, it is an object of the present invention to provide a method and a measurement system for measuring in-hospital meal intake that can automatically and accurately measure and record meal intake by a simple operation.

  The basic concept of the measurement method and measurement system according to the present invention is to compare and observe pre-meal and post-meal images and calculate the calories consumed by patients and the like. As shown in FIG. 4, the processing flow of the present measurement system is to extract the tableware portion and the food portion from the tray after photographing the meal with the imaging device, and then the change amount of the image of the food portion before and after the meal as the area. Measure with In this measurement system, a neural network (hereinafter also referred to as NN) is applied and used to identify individual dishes from a tray image instantaneously (2 to 3 seconds). In addition, labeling and image processing such as 8-neighbor contraction processing are used in combination for extracting food images from tableware images. Then, the change amount of the food image before and after the meal is evaluated by the area, and the calorie intake is calculated by linking with the food database.

In order to achieve the above object, a measurement method according to claim 1 of the present invention includes:
1) Using an imaging device, take a picture of a single meal containing food on each tableware on the tray for the pre-meal state and the post-meal state,
2) Extract each tableware on the tray in the pre-meal state image, and determine the type of tableware by inputting it into the neural network in light of the tableware database input in advance.
3) Extract only the food part on each tableware image, calculate the amount of change in the food part before and after the meal by area, and then calculate the calorie intake in light of the food database prepared in advance. It is characterized by having.

  According to the method for measuring in-hospital meal intake according to claim 1, each person's meal intake can be accurately measured under a certain standard, and each person's meal intake can be automatically and quickly measured. Accurate measurement. Furthermore, it is possible to easily manage each person's meal intake data by digitizing it.

As claimed in claim 2,
The type of each tableware is input to the neural network and discriminated (a predetermined image is cut out from the food image at regular intervals suitable for each tableware registered in the database, and the shape extracted from the cut-out image is input to the neural network. If the response value when the information obtained from the tableware database system is input to the neural network and the response value of the neural network are ignited in the same way as when registering the tableware, it is determined that the cut-out image contains the target tableware. That is, the tableware corresponding to the pattern that outputs the highest value among the values output by the neural network is determined as the input tableware, whereby the type of tableware is determined and its position is also specified. The concentration of food and tableware on each tableware. The tableware image is corrected so that the concentration value of the food portion temporarily matches the concentration value of the tableware portion, and the type of tableware is discriminated based on the image data of only the tableware taken in advance with this corrected image. To do.

  According to the method for measuring in-hospital meal intake according to claim 2, it is possible to calculate the measurement result relating to the meal intake per person within a very short time of about 2 to 3 seconds, and the photographed image is slightly unclear. Or, even if a part of the tableware overlaps, the tableware can be accurately identified including its position, so a large number of meals can be instantly photographed with a digital camera, and the amount of food intake for each tableware can be measured.

As claimed in claim 3,
Each of the tableware images can be included in the dining table image and cut out at predetermined intervals in a predetermined shape such as a rectangle or a circle with the smallest area (overlapping of some cutout amounts is allowed). The food portion on each tableware is corrected to an image that matches the concentration value of the tableware and input to the neural network, and the response value (output value) when the output value of the output unit is registered in the tableware database. The tableware can be extracted when it substantially matches.
According to the method for measuring in-hospital meal intake according to claim 3, since the tableware is specified by correcting the image with no food on it, the type of tableware is surely specified in a short time of 2 to 3 seconds. it can.

As claimed in claim 4,
The radius, vertical and horizontal length (number of pixels) of each tableware input to the neural network,
1) Set a threshold value for binarization. The pixel is divided into “1” pixel and “0” pixel depending on whether it is larger or smaller than the threshold value.
2) Perform 8-neighbor contraction processing to remove noise from the binarized image.
3) A label number is assigned to each portion where “1” pixels are aggregated (there are “1” pixels in the vicinity of 8).
4) Find the maximum and minimum values of the x-axis and y-axis of the label area, respectively.
5) Obtain the vertical and horizontal lengths of the tableware from the maximum and minimum values obtained in 4) above.
-It is preferable to carry out the above steps.

  According to the method for measuring in-hospital meal intake according to claim 4, when the tableware moves on the tray and deviates from the position before the meal (for example, when it protrudes from the tray or some other tableware In other words, the cutout part used when extracting the food even if some of the other tableware entered the cutout part of the image and affected the target tableware image as noise. It is possible to erase a small isolated area that is noise using several times of shrinkage processing on the image of the image, and the discrimination of the tableware is performed by a neural network having a non-linear identification capability, so the position of the tableware can be determined It can be identified accurately. Thereby, the area of the food on the subsequent tableware can be calculated accurately.

As claimed in claim 5,
Extraction of food from each tableware,
1) After extracting the food image from the 256-color bitmap file, for example, extract only a blue image out of RGB (red, blue, green).
2) The blue image is binarized using a threshold value. That is, a) Pixels equal to or greater than the threshold value are set to “1” pixels (food portion). b) Pixels smaller than the threshold are set to “0” pixels (background portion).
3) Eight neighborhood contraction processing is performed to remove noise in the blue image. The number of contractions is 1 to 100 (the optimum value is 4 in the experiment).
4) Perform label processing. That is, a) “1” pixel is searched. b) The current label number is assigned to the searched “1” pixel. c) If there is a “1” pixel near 8 of the current label number, the same label number is set for that pixel.
5) The center position of each label area is obtained, and the maximum value and the minimum value of the area position are obtained for each obtained label area.
-It can be carried out by a plurality of steps as described above.

  According to the method for measuring in-hospital meal intake according to claim 5, the area of the food can be accurately specified not only when the food is solid, but also when it is a fluid, particularly the remaining amount of food is specified. To demonstrate its effectiveness.

As claimed in claim 6,
Calculation of the intake amount
1) Calorie Acal and area (pixel) S when the remaining amount of food is 100% are calculated.
2) The rest of the food after the meal is extracted for each tableware.
3) Calculate the area (pixel) s of the remaining portion of the food.
4) The calorie intake a for each tableware is calculated by the following formula.
acal = {1-s / S} × Acal
-It can be carried out by a plurality of steps as described above.

  According to the method for measuring in-hospital meal intake according to claim 6, since the remaining amount of food can be calculated instantaneously, the intake of meals for a large number of people can be accurately calculated in a short time.

As claimed in claim 7,
The calculated food intake can be digitized and recorded in a database for each patient and used for short-term or long-term meal management of the patient.

  According to the method for measuring in-hospital meal intake according to claim 7, it is possible to easily and quickly carry out nutritional management for a large number of people in a hospital or nursing home for the elderly, and can greatly reduce the burden on administrators such as nutritionists. .

The in-hospital meal intake measuring system according to claim 8 is connected to the same imaging device as a case-like imaging device capable of taking in and out a table with a plurality of dishes arranged on a tray having a predetermined shape such as a square or a rectangle. A measurement system including a personal computer,
One or more digital cameras and a lighting device are provided in the imaging device, and images of a plurality of meals captured by the digital camera are automatically captured by the personal computer. Tableware shape acquisition means by binarization, tableware determination means by neural network, food extraction means from tableware, and food intake calculation means, the tableware determination means in the tableware database and the food intake The calculating means is configured to be connectable to a food database such as a meal menu.

  According to the measurement system of the eighth aspect, the above measurement method can be surely implemented, and the system can be achieved relatively easily and at low cost.

  The in-hospital meal intake measuring method and measuring system according to the present invention have the following effects.

・ Each person's dietary intake can be accurately measured.
・ Each person's food intake data can be easily managed.
・ Each person's food intake can be measured automatically and accurately in a short time (time allowed for removal of the meal in the hospital within 5 seconds).
・ Meal intake can be measured under certain criteria.
・ Long term management of inpatient nutrition is possible.
-Meals can be given to remote residents such as elderly people living alone (by using communication via LAN or telephone line).

  Hereinafter, with respect to the in-hospital meal intake measurement system according to the present invention, an embodiment applied to the meal intake measurement system in an in-patient meal in a hospital will be described based on the drawings, and the measurement method will also be described. explain.

  FIG. 1 is a front view showing an embodiment of a measurement system according to the present invention. As shown in FIG. 1, the measurement system 1 is composed of an imaging device 2 for a meal A and a personal computer (hereinafter referred to as a personal computer) 3. As shown in FIG. 2, the image pickup apparatus 2 includes a housing 4 having a front surface opened, and a lid plate 5 is provided at an open front portion of the housing 4 so as to be opened and closed. The lid plate 5 is slid horizontally and accommodated in the upper end space portion 4a in the housing 4 by sliding upward with a handle 5a at the center of the lower end portion.

  At the bottom of the housing 4, the dining table A can be stored together with the tray T. The dining table A usually includes a substantially square tray T and a plurality of types of tableware B placed on the tray T, and each tableware B contains or contains cooked food C. Also, white cloudy light bulbs (25W) 6 are attached to the four corners of the lower surface of the ceiling 4b in the housing 4 respectively. Furthermore, in order to minimize the influence of reflected light due to the irradiation of the light bulb 6 when photographing the dining table A, a translucent acrylic diffusion plate 7 as shown in FIG. As shown, the four light bulbs 6 are mounted so as to cover the lower side. As a result, the reflected light as shown in FIG. 3C was minimized as shown in FIG.

  Near the center of the ceiling 4b (slightly rearward), a color digital camera for photography (resolution: 320 × 240 pixels) 8 is attached downward, and the connection to the personal computer 3 storing the software portion is USB. A port (not shown) is used. The lid plate 5 is attracted to the magnets 9 on both sides of the lower end of the front surface of the housing 2 and is kept closed.

  FIG. 4 is a flowchart showing a processing procedure in the overall configuration of the main part (software part) of the measurement system 1 of the present embodiment.

  In the measurement system 1 of the present example, as will be described later, first, by eating a single meal A before meal in the imaging device 2, the meal A is automatically photographed from above by a plurality of digital cameras 8. (Step 1 '). Only the portion of the tableware B is extracted from the image of the table A (step 2 '). For tableware B, information about the tableware is acquired from the tableware database, and the type of tableware is specified. Subsequently, the food C on each tableware B is extracted (step 3 '), and the area of each food C is calculated in advance as data (step 4').

  After the meal is finished, each person's meal A is stored in the imaging device 2 one after another, and the meal A is automatically photographed from above by the digital camera 8 (step 1). Only the portion of tableware B is extracted from the image of the table A (step 2). For tableware B, information about tableware (horizontal / vertical dimensions (width, height), radius, binarization threshold, etc.) is acquired from the tableware database, and the type of tableware is specified. Further, the food C remaining on each tableware B is extracted (step 3). Then, the area of the remaining amount of each food C is calculated (step 4). Up to this point, the steps 1 'to 4' are completely the same. Finally, the area of the pre-meal food C on each tableware B is compared with the area of the food remaining after the meal, and the amount of food C ingested from the difference between the two areas is calculated in calories and output (Step 5). At the time of calorie calculation, data on the type and number of foods C on each tableware B, calories, etc. are acquired from a pre-input food database. In the previous menu registration, the type and number of tableware used, the type and number of food placed on each tableware, calories, etc. are acquired in advance as various database information for the meal A before meal. The database exists in the personal computer 3 and is managed by the system as a hierarchical structure. In the flowchart of FIG. 4, step 1 ′ to step 4 ′ in menu registration are omitted.

  The main part of the measurement system 1 includes tableware shape acquisition means (subsystem) 11 by binarization, tableware determination means (subsystem) 12 by a neural network, food extraction means (subsystem) 13 from the tableware, food intake It is roughly classified into quantity calculation means (subsystem) 14. These subsystems 11 to 14 are linked with the main system and stored in the personal computer 3 as programs. A tableware database 15 is connected to the tableware determination means 12, and a food (meal menu in this example) database 16 is connected to the intake amount calculation means 14 as an attached system. These subsystems are stored in the hard disk of the personal computer 3 as a program linked with the main system, and the process from registering the menu of meals to determining the measurement of meal intake is made possible by online processing by the personal computer 3. The calorie intake is calculated by comparing the image of food remaining for each tableware with the pre-meal image on the area, not the image of the entire meal tray.

  Hereinafter, each of the subsystems 11 to 14 will be described in detail.

◎ Food removal process, which is a pre-treatment for determining tableware When a tableware extraction process was performed with food on the tableware, there was a problem that the specific success rate of the tableware position was low. This is because the food part becomes noise when the food image is binarized. Therefore, in order to cut out the correct tableware image from the binarized image of the food bowl and input it to the neural network, it is necessary to remove the food portion of the cutout image. Below, the flow of a food removal process is demonstrated. FIG. 16 is an explanatory diagram illustrating a process for removing a food part from a binarized image showing a food removal algorithm, and is a flowchart of FIG. 17. FIG. 18A shows a correct input image of tableware to the neural network, and FIG. 18B shows an input image of tableware in which food is noise.

  As shown in FIG. 16, the food removal processing method is such that when searching for all pixels of the binarized binarized image that is the source of the NN input image, The minimum value and the maximum value of the Y coordinate are detected (FIG. 16A), and the portion corresponding to the minimum value from the minimum value is filled with black pixels (FIG. 16B). Specifically, while scanning the cut-out image from top to bottom, detecting the coordinates of the top and bottom of the food part and filling in the black pixels between those coordinates while performing the work for all X coordinates , An NN input image is created (FIG. 16B). The above processing flow is as shown in the flowchart of FIG. Here, the tableware is black (0 pixel), and the tray and the food part are white (1 pixel).

  Note that the input image to the neural network is changed from a monochrome grayscale image to a color image, and threshold values for binarization are also changed from threshold settings based on monochrome grayscale values to threshold values based on brightness and hue values. Changed to value setting. By these changes, the binarized image of the above-mentioned food image has become clear.

  Then, the tableware position determination test was performed using the program that introduced the food removal process and the program before the introduction. The results are as shown in FIG. 19. FIG. 19A shows the result after the program change, and FIG. 19B shows the result before the program change. In addition, the circle of FIG. 19 represents the part determined as the present measurement system tableware position. As can be seen from FIG. 19A, the tableware position is accurately determined even when food is placed on the tableware. The reason for this is that when extracting dishes, the food part of the cut-out image input to the neural network is temporarily removed from the dishes by the above-mentioned food removal algorithm. Because it becomes. Therefore, the tableware image in the cut-out image is NN recognized as an image containing no noise.

◎ Determination of Tableware from the Tray Here, the basic configuration of the tableware determination subsystem 11 will be described. The subsystem 11 uses the tableware information obtained from the tableware database system 15 to determine the tableware on the tray from the food image. As shown in FIG. 8, the information on tableware B is usually the number of tableware on the tray, the type of tableware, the shape of the tableware, and the binarized threshold value.

  The structure of the NN for tableware determination is a three-layered hierarchy as shown in FIG. In this NN input layer, the threshold value used when taking out the outer edge of the tableware, the radius of the tableware in the table image, the vertical and horizontal length (number of pixels), and the grayscale image of the tableware are input from the tableware database, respectively. Entered as a value. The number of output units corresponds to the number of registered tableware types (the output unit is associated with tableware 1, tableware 2,..., Tableware N in advance). The learning algorithm is an improved BP method expressed by the following equation (1). The weight is corrected by the probabilistic descent method so that the output pattern, which is the result of the given input pattern, approaches the correct result (teacher value), and the error is reduced. Learning algorithm). Of the values output by the NN, the tableware corresponding to the pattern that outputs the highest value is determined as the input tableware and extracted.

In the processing flow, first, a tableware image is captured by a digital camera, and binarization processing of the image is performed. Next, information regarding the type of tableware is acquired between the processing portion and the tableware database system. The tableware information at this time indicates the number of tableware on the tray, the shape and size of the tableware, the threshold value for binarization, and the like. FIG. 6 shows a main screen on which a photographed image of the meal A is displayed on the display of the personal computer 3. FIG. 9 shows an image of tableware on the tray and an image obtained by binarizing the image.
Here, ΔW is a weight correction amount, ε, α, β are constants, δ is a generalization error, and o is an output.

  FIG. 7 is a flowchart sequentially showing the flow of the process of extracting tableware from the tray described above. Finally, the position of the tableware on the tray is also specified by the following cutout image. FIG. 8 shows a screen displaying data related to one tableware in the tableware database.

  As shown in FIG. 11, an image is cut out from a whole image of the dining table by shifting the position at regular intervals with a frame having a predetermined shape (rectangular in this example) having a minimum area that can surround tableware. The shape and image extracted from the cut-out image are input to the neural network. When the response at the time of registering the tableware substantially matches the response when the cutout image is input, it is determined that the cutout image includes the tableware. In this way, the type of tableware and the like are determined together with the position.

  By the way, in the measurement system of the present invention, a neural network is adopted for determining tableware for the following reason. That is, a method of searching for all pixels of the photographed image by image processing and determining tableware can be considered. However, in this measurement system 1, it is necessary to calculate and display a measurement result relating to the amount of meal intake per patient within a very short time of about 2 to 3 seconds. In addition, since a large number of meals are photographed instantaneously with a digital camera, the photographed image may be slightly blurred. However, if NN is used, even if the image is somewhat blurred, it can be sufficiently handled. Therefore, instead of image processing requiring a lot of computation time, a tableware type determination method using a neural network using partially cut-out images was adopted.

  Further, the radius, vertical and horizontal length (number of pixels) of the tableware input to the NN can be obtained by the following algorithm.

(a) Set a threshold for binarization. The pixel is divided into “1” pixel and “0” pixel depending on whether it is larger or smaller than the threshold value.
(b) Perform 8-neighbor contraction processing to remove noise from the binarized image.
(c) A label number is assigned to each portion where “1” pixels are aggregated (there are “1” pixels in the vicinity of 8).
(d) Find the maximum and minimum values of the x-axis and y-axis of the label area, respectively.
(e) Obtain the tableware radius, vertical and horizontal lengths from the maximum and minimum values.
In addition, Fig.12 (a) is a top view which shows the 10 types of tableware used in a present Example, FIG.12 (b) is a top view which shows the example of arrangement | positioning of 6 types of tableware on a tray.
An extraction test was performed by inputting to the NN using images composed of 59 patterns arranged on the tray by combining the above 10 types of tableware, but the tableware could be specified for all 59 configurations.

Extraction of food from tableware In the present measurement system 1, a process for extracting food from tableware is required as a pre-processing when measuring food. The algorithm of the food extraction process is shown below.

  A food image is extracted from a 256-color bitmap file, and only a blue image, for example, of RGB (red, green, and blue) is extracted. Next, the image is binarized using a threshold value.

(A) A pixel that is equal to or greater than the threshold is set to “1”.
(B) Pixels smaller than the threshold are set to “0” pixels.

  Here, the “1” pixel corresponds to a food portion, and the “0” pixel is another background portion. Next, in order to remove noise in the image, 8-neighbor contraction processing is performed. With respect to this contraction process, the number of contractions can be arbitrarily determined from 1 to 100, and the optimum number of contractions is set to 4 through trial and error in experiments. Then, the following label processing is performed after the 8-neighbor contraction processing.

(a) Search for “1” pixel.
(b) The current label number is assigned to the searched “1” pixel.
(c) If there is a “1” pixel near 8 of the current label number, the same label number is set for that pixel.

  Thereafter, the center position of each label area is obtained. Finally, the maximum value and the minimum value of the area position are obtained for each obtained label area.

  FIG. 13 is a flowchart sequentially showing the flow of the above-described food extraction process from the tableware.

◎ Measure food intake Judgment of food intake by area
Although food measurement by NN was examined, it is presumed that it is not possible to deal with images of the rest of various foods in practice. For example, for food rotation, it is necessary to apply rotation correction to the extracted image or convert it into rotation-invariant information (moment, area, etc.), which is considered to be a rather complicated processing step. Therefore, in the measurement system 1 of the present invention, a method for measuring the area (number of pixels) of the remaining food is adopted as the main body of the measurement subsystem 14 of the measurement system 1. The algorithm is as follows.

(a) Calculate the calorie Acal and the area (number of pixels) S for the remaining amount of food 100%.
(b) Extract the food left after the meal from the dishes
(c) Calculate the area (number of pixels) s of the remaining food.
(d) Calorie intake a for each tableware is calculated by the following formula.
acal = (1-s / S) × A cal

◎ Confirmation test Here, the determination test of the tableware image using the measurement system 1 of the present invention, the extraction test of the food from the tableware, and the measurement test of the meal intake amount were carried out. consider.

1) Tableware confirmation test Here, the tableware confirmation test by the NN tableware determination method described above is performed. Tableware information is obtained by interprocess communication with the tableware database system. FIG. 8 shows a menu screen of the tableware determination subsystem. Table 1 shows NN learning and evaluation conditions.

In the same test, 59 types of images of 10 types of tableware were used, and a definite test on tableware was conducted. An example of the tableware image on the tray is shown in FIG. Table 2 shows an example of tableware information input to the NN.

As a result of the test, it was confirmed that it was possible to determine the types and positions of tableware for all 59 patterns. However, there have been cases where it is not possible to specify the exact tableware position even if the type of tableware is specified successfully. The cause of this is presumed that a part of other tableware may invade into the cut-out portion of the image, and this influences noise on the target tableware image. On the other hand, in the measurement system 1 of the present invention, a small isolated area that is noise can be erased by using several times of shrinkage processing on the image of the cutout portion used at the time of food extraction. The position could be identified accurately.

  FIG. 10A shows an example in which tableware can be extracted at an accurate position, and FIG. 10B shows an example in which tableware cannot be extracted at an accurate position. However, these successful cases and unsuccessful cases are tests conducted without food on the tableware.

  Then, when the extraction processing of the tableware was performed with food on the tableware, as described above, there was a problem that the specific success rate of the tableware position was low, so the correct tableware image from the binarized image of the tableware was generated. In order to cut out and input to the neural network, the program was modified to incorporate an algorithm that removes food parts on tableware from the cutout image. A comparative test was conducted after the program change and before the program change. In the successful example (FIG. 19 (a)), the food on the tableware is temporarily removed by the food removal algorithm and the noise is eliminated. Therefore, the tableware position is accurately set under the same conditions as the tableware registration state in the NN. However, in the failure example (FIG. 19B), the food portion is noisy, so the tableware position is not accurately recognized when it is input to the NN.

2) Food extraction test Fig. 14 (a) shows the result of extracting the fish fillets divided into five parts. Moreover, the extraction process result with respect to a liquid food is shown in FIG.14 (b). Thus, the effectiveness of the algorithm was confirmed about the extraction of the residual food by 8 neighborhood contraction processing and labeling.

3) Meal intake measurement test Here, an intake measurement by NN and an intake measurement by area were performed. However, the measurement test by NN here is positioned as a comparison target of the measurement test by area.

(1) Ingestion determination by NN First, the method of determining the image of the remaining food with the NN configuration similar to the tableware determination and calculating the corresponding calories. The algorithm is shown below.
(a) A typical dietary intake is classified into levels (eg Level-1 = 0%,
Level-2 = 20%, Level-3 = 40%,..., Level-6 = 100%), and calorie Acal for 100% of the remaining amount of food is calculated.
(b) The food remaining after the meal is extracted for each tableware, and the residual food level is determined.
(c) The calorie intake a for each tableware is calculated by the formula (2).
a cal = {1 − (% of final level) / 100} × A cal

  In Test 1, dummy tonkatsu (manufactured with foamed styrene and colored in beige) classified into four levels with remaining amounts of 0%, 20%, 60% and 100% as shown in FIG. 15 (a) is used. This image is used to perform learning and identification experiments. In the measurement unit using NN, as a method for extracting the feature value of the input image, the entire input image is divided into a grid with a mesh, scanned from the vertical, horizontal, and diagonal directions, and the total value of the shade information in the mesh is used as the feature value of the image Enter into NN as In the experiment, 5 learning images of NN were used at each level, and 10 images of unlearned food were used. An unlearned image is added with a position shift and a slight rotation shift. Due to the characteristics of the algorithm of food extraction, the response value of NN was not affected by the position shift of food, but the effect of NN response value decreased by rotation was observed. However, although some NN units have a reaction value of less than 0.5, the overall recognition rate was 100%.

  Next, in Test 2, experiments were performed using real cooked rice classified into the four levels shown in FIG. The experimental result of test 2 was 97.5%. However, in the evaluation of cooked rice, rather than the influence of rotational deviation, the influence of reflected light from the bottom of the tableware when there was no cooked rice was large, and this was identified as white cooked rice despite the absence of cooked rice. However, in this experiment, the subsystem is not linked to the food database programmatically, so the calorie calculation using Equation (2) is not calculated, and the level of remaining food is limited.

(2) Judgment of intake by area The result of operating as a subsystem alone was almost the same performance (97%) as the level classification of the remaining food by NN. The result is not the final calorie intake measurement as in the case of (1), but the number of remaining food pixels. The experimental conditions are the same as (1).

It is a front view which shows the Example of the measuring system of this invention. FIG. 2A is a perspective view showing the imaging apparatus, and shows a fully opened state of the lid plate. FIG. 2B is a perspective view showing the imaging apparatus, and shows a state in the middle of closing the cover plate. FIG. 3A is a plan view of a diffuser plate for reducing reflected light (filter), FIG. 3B is a perspective view of the reflected light mitigating diffuser plate mounted on the imaging apparatus 2, and FIG. Fig. 3 (d) is an image of a food shoot taken without a light reducing diffuser plate (with reflection from the tray), and Fig. 3 (d) is a photo of the food shoot taken with the diffused light reducing diffuser attached (reduction in reflection from the tray). ). It is a flowchart which shows the procedure of the process in the whole structure of the principal part (software part) of the measurement system 1 which concerns on the Example of this invention. It is a block diagram which shows the structure of the principal part of the tableware determination means by NN applied to the measurement system of this invention. In the measurement system of the present invention, a photographed image of the meal A is a main screen displayed on the display of the personal computer 3. It is a flowchart which shows in order the flow of the extraction process of the tableware from the tray applied to the measurement system of this invention. It is the screen which displayed the data regarding one tableware of a tableware database. It is drawing which shows the image of the tableware on a tray, and the image which binarized this. FIG. 10 (a) is a drawing showing an example of an image in which tableware (without food) can be extracted at an accurate position in the tableware extraction test, and FIG. 10 (b) is an illustration of tableware (without food) at an accurate position in the tableware extraction test. It is drawing which shows the example of an image which was not able to be extracted. It is an image which shows the aspect which cuts out tableware in the tableware extraction process from the tray applied to the measuring system of this invention, and inputs it into NN. FIG. 12A is a plan view showing 10 types of tableware to which the measurement method according to the embodiment of the present invention is applied, and FIG. 12B is a view of six types on the tray applied to the measurement method according to the embodiment of the present invention. It is a top view which shows the example of arrangement | positioning of tableware. It is a flowchart which shows in order the flow of the extraction process of the tableware applied to the measuring method which concerns on the Example of this invention. FIG. 14 (a) is a drawing showing an image obtained by performing extraction processing on five-section fish fillets applied to the measurement method according to the embodiment of the present invention, and FIG. 14 (b) is a measurement according to the embodiment of the present invention. It is drawing which shows the image which performed the extraction process with respect to the liquid food applied to the method. FIG. 15A is a plan view showing dummy tonkatsus classified into four levels of remaining amounts of 0%, 20%, 60% and 100%, and FIG. 15B is a remaining amount of 0%, 20% and 60%. It is a top view which shows the cooked rice classified into 4 levels of 100%. It is explanatory drawing illustrating the process which removes a food part from the food binarized image which shows the algorithm of the food removal applied to the measuring method which concerns on the Example of this invention. It is a flowchart which shows the algorithm of a food removal. FIG. 18A shows a correct input image of tableware to the neural network, and FIG. 18B shows an input image of tableware in which food is noise. FIG. 19A is an image showing a successful example after changing the program in which tableware (with food) can be extracted at an accurate position in the tableware extraction test, and FIG. It is an image which shows the example of failure before the program change which could not extract (with food). It is drawing which shows the front perspective image of the state which mounted the conventional general hospital food in the wagon.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Measurement system 2 Imaging device 3 Personal computer 4 Case 5 Cover plate 6 Light bulb 7 Diffusion plate (filter)
8 Digital camera

Claims (8)

1) Using an imaging device, take an image of a single meal containing food on each tableware on the tray in the pre-meal state and the post-meal state.
2) Each tableware on the tray is extracted from the pre-meal image, and the type of tableware and the position on the tray are input to a neural network in accordance with the tableware database input in advance.
3) After extracting only the food portion on each tableware image and calculating the change amount of the food portion before and after the meal by area, various nutrients such as intake calories and vitamins are calculated according to the food database prepared in advance. .
-A method for measuring in-hospital meal intake, comprising the above steps.   When discriminating the type of each tableware using a neural network, the concentration value of the food portion on the tableware is compared with the concentration value of the tableware portion so that the concentration value of the food portion temporarily matches the concentration value of the tableware portion. The ingestion of in-hospital meals according to claim 1, wherein the image is corrected, that is, the food on the tableware is temporarily erased, and the type of the tableware is determined based on image data only of the tableware previously taken with the corrected image. How to measure quantity.   Each of the tableware images can be included in the food table image and cut out at a predetermined interval in a predetermined shape such as a rectangle or a circle with the smallest area, and the food portion on each tableware matches the concentration value of the tableware. When the response value (output value) of the output unit matches the response value (output value) when the tableware database is input, the predetermined shape is completely registered at the time of registering the tableware. The method for measuring in-hospital meal intake according to claim 2, wherein it is determined that the image is a cut-out image including a tableware image, and the tableware and its position are extracted. The radius, vertical and horizontal length (number of pixels) of each tableware input to the neural network,
1) Set a threshold value for binarization. It is divided into “1” pixel and “0” pixel depending on whether it is larger or smaller than the threshold.
2) Perform 8-neighbor contraction processing to remove noise from the binarized image.
3) A label number is assigned to each portion where “1” pixels are gathered (there are “1” pixels in the vicinity of 8).
4) Find the maximum and minimum values of the x-axis and y-axis, respectively, in the label area.
5) From the maximum and minimum values obtained in 4) above, find the radius, vertical and horizontal length of the tableware.
The method for measuring in-hospital meal intake according to any one of claims 1 to 3, which is performed by-. Extraction of food from each tableware,
1) After extracting the food image from the 256 color bitmap file, extract only the blue image of RGB.
2) The blue image is binarized using a threshold value. That is, a) Pixels equal to or greater than the threshold value are set to “1” pixels (food portion). b) A pixel smaller than the threshold value is set to “0” pixel (background portion).
3) Eight neighborhood contraction processing is performed to remove noise in the blue image. The number of contractions is 1 to 100.
4) Perform label processing. That is, a) A “1” pixel is searched. b) The current label number is assigned to the searched “1” pixel. c) If there is a “1” pixel near 8 of the current label number, the same label number is set for that pixel.
5) The center position of each label area is obtained, and the maximum value and the minimum value of the area position are obtained for each obtained label area.
The method for measuring in-hospital meal intake according to any one of claims 1 to 3, which is performed by-. Calculation of the intake amount
1) Calorie Acal and area (number of pixels) S when the remaining amount of food is 100% are calculated.
2) The rest of the food after the meal is extracted for each tableware.
3) Calculate the area (pixel) s of the remaining portion of the food.
4) The calorie intake a for each tableware is calculated by the following formula.
a cal = {1-s / S} × A cal
The method for measuring in-hospital meal intake according to any one of claims 1 to 3, which is performed by-.   The method for measuring in-hospital meal intake according to any one of claims 1 to 6, wherein the calculated meal intake is digitized, recorded in a database for each patient, and used for management. A measurement system comprising a housing-like imaging device capable of taking in and out a table with a plurality of dishes arranged on a tray of a predetermined shape such as a square or a rectangle, and a personal computer connected to the imaging device,
In the imaging device, a digital camera and an illumination device are provided for capturing images from a plurality of directions, and a food image captured by the digital camera is automatically captured in the personal computer.
The personal computer includes tableware shape acquisition means by binarization of a photographed image, tableware determination means by a neural network, food extraction means from tableware, and meal intake calculation means,
An in-hospital meal intake measuring system, wherein the tableware determination means is connectable to a tableware database, and the meal intake calculation means is connectable to a food database such as a meal menu.