JP2006000428A - Meal support apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To support the smooth progress of a meal of a user regardless of the way of serving food or the variation of the environmental conditions such as illumination. <P>SOLUTION: A meal support apparatus is provided with a holding mechanism part 36 for holding the food, a container 12 is divided into a plurality of areas, an effective value indicating the possibility that holding the food becomes effective for each of the plurality of areas is calculated, and the holding mechanism part 36 is controlled so as to perform holding to a holding position decided from the plurality of areas on the basis of the predicted value of effectiveness, to hold the food served in the container 12, carry it to the mouth of the user and support the meal of the user. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a meal support apparatus for carrying food to a user's mouth.

  2. Description of the Related Art When a user with a disability in an upper limb eats, a meal support apparatus has been developed that supports food by transporting food from a container in which food is served to the user's mouth.

  Up to now, the user has operated the arm with the food gripping part consisting of a spoon and fork using a joystick or the like with the chin, etc., to grip the food placed in the container and carry it to the user's mouth Meal support devices have been developed. In such a meal support device, food is placed in a container having a predetermined shape, the user designates the position of the food that he / she wants to eat by tilting the lever of the joystick, and the food placed at that position is placed. Hold it with the food gripping part and transport it to the user's mouth. Then, the sensor detects that the user's mouth has touched the conveyed food, and releases the food from the meal gripping portion to provide the user with the food. When the user's mouth is separated from the meal gripping portion, the meal gripping portion is moved to the standby position by the arm and waits until the next food is designated.

  In addition, without receiving designation of the position of food from the user, the user performs gripping operation on each position of the container according to a predetermined procedure, and sequentially holds the food arranged at each position, A meal support device having an automatic mode for transporting to the mouth of a person has also been developed.

JP 2004-81567 A Japanese Patent Laid-Open No. 2003-62019

  However, the food is not neatly arranged in the container, and the position where the food is served and the position where the food is not served are mixed. Therefore, if gripping is performed according to a predetermined order with respect to each position of the container, it will try to grip the position where food is not served, so that the user can advance the meal smoothly. There was a problem that could not be done.

  In addition, when gripping fails when gripping is attempted and food has shifted to a position that has already been gripped, a retry cannot be performed unless the predetermined gripping order is completed. . For this reason, there has been a problem that the meal cannot be smoothly performed.

  In view of the above-described problems of the prior art, the present invention eliminates useless gripping operations by detecting a position on a container on which food is actually arranged, thereby enabling a user's meal to proceed more smoothly. An object is to provide a support device.

  The present invention is a meal support device for supporting a user's meal by gripping food placed in a container and carrying it to a user's mouth, comprising a gripping mechanism for gripping food, Dividing into a plurality of regions, calculating a prediction value of effectiveness indicating the possibility of grasping food for each of the plurality of regions, and grasping from the plurality of regions based on the prediction value of effectiveness A gripping position to be performed is determined, and the gripping mechanism unit is controlled to grip the gripping position.

  Here, the image processing apparatus includes an imaging unit that acquires an image of the container and the food arranged in the container, and obtains an occupancy rate of the food for each of the plurality of regions from the image, and the effectiveness based on at least the occupancy rate It is preferable to calculate the predicted value. It is also preferable to store a gripping trial history for each of the plurality of regions and calculate the predicted value of effectiveness based on the gripping trial history. Furthermore, it is also preferable to obtain a relative positional relationship between foods placed in the container and calculate the predicted value of effectiveness based on the positional relationship according to the structure of the gripping mechanism.

  ADVANTAGE OF THE INVENTION According to this invention, even when food is irregularly arranged in the container, the user can advance the meal more smoothly.

  As shown in FIG. 1, the meal support apparatus according to the embodiment of the present invention includes a main body unit 10, a container 12, a transport unit 14, an operation unit 16, and an imaging unit 18. In FIG. 2, the functional block diagram of the meal assistance apparatus in this Embodiment is shown.

  The main body 10 incorporates an input interface (I / F) 20, a calculator 22, a storage unit 24, an input / output interface (I / F) 26, a mode change switch 28, and a notification unit 30. The input I / F 20 is connected to the operation unit 16 and the mode change switch 28 so as to be able to transmit information. The input I / F 20 receives an instruction signal or a mode signal from the operation unit 16 and the mode change switch 28, and performs signal format conversion and the like. After processing, the data is transferred to the calculator 22. The calculator 22 controls the overall meal support apparatus by executing a control program stored in the storage unit 24. Control performed by the calculator 22 will be described later. The storage unit 24 stores and holds a control program executed by the computing unit 22 and various data used for control. The program or data stored in the storage unit 24 can be appropriately read and used by the calculator 22. The input / output I / F 26 receives signals from the imaging unit 18 and the transport unit 14 and passes them to the calculator 22. The mode changeover switch 28 is a switch for selecting any one of a manual mode, a semi-automatic mode, and an automatic mode provided in the meal support apparatus. The mode selected by the mode switch 28 is input to the computing unit 22 via the input I / F 20 as a mode signal. The notification unit 30 includes, for example, a lamp, a buzzer, and the like. When the abnormality occurs in the meal support device, the notification unit 30 receives a signal from the computing unit 22 and is abnormal for a user or a caregiver who cares for the user. Used to notify

  The container 12 is used to serve meals. The container 12 is partitioned into a plurality of divided regions. In the present embodiment, as illustrated in FIG. 1, the description is made on the assumption that the area is divided into four divided regions 12a to 12d. However, the present invention is not limited to this, and the number of divisions may be increased or decreased as necessary. You may let them. The ground color of the container 12 is the same color in the four divided regions 12a to 12d. The ground color of the container 12 is preferably a single color. Moreover, when performing a meal, the container 12 is set to the butting part 32 provided in the main-body part 10 of the meal assistance apparatus. Accordingly, the spatial relative position between the transport unit 14 and the imaging unit 18 is always maintained constant, and it is possible to perform a gripping operation and imaging with a constant positional relationship.

  The transport unit 14 includes a transport mechanism unit 34 and a gripping mechanism unit 36. The transport mechanism unit 34 includes an arm for moving the gripping mechanism unit 36. As shown in the example of FIG. 1, the transport mechanism unit 34 is preferably composed of arms 34 a to 34 d connected by a four-axis rotation drive unit 34 e. The transport mechanism unit 34 receives a control signal from the computing unit 22 via the input / output I / F 26 and drives the driving unit such as a motor with the arms 34a to 34d to make the gripping mechanism unit 36 the area of the container 12 or use it. Carry it to the person's mouth. The gripping mechanism unit 36 includes a gripping member 38 for gripping food. As shown in the example of FIG. 1, the gripping member 38 is preferably configured by a combination of a fork and a spoon. Further, as shown in FIG. 2, the gripping mechanism section 36 includes a drive section 36a, a contact detection section 36b, a grip detection section 36c, and a retracted position detection section 36d. The drive unit 36 a receives a control signal from the computing unit 22 via the input / output I / F 26 and drives the fork and spoon of the gripping member 38. When the gripping member 38 is moved to the position of the food in the container 12 based on the control signal, the food is scooped by scooping the food by providing the fork on the top and the spoon on the bottom. At this time, when it is detected that food is in contact with the gripping member 38 by the contact detection unit 36b configured by a pressure sensor or the like, a detection signal is transmitted to the calculator 22. The calculator 22 receives the detection signal and transmits a control signal for gripping food to the drive unit 36a. In response to the control signal, the driving unit 36a drives the fork and the spoon to grip the food. Furthermore, it is detected that the food is gripped by the grip detection unit 36 c configured by an angle sensor or the like, and a detection signal is transmitted to the calculator 22. The calculator 22 receives the detection signal and transmits a control signal for moving the gripping member 38 to the mouth of the user to the transport mechanism unit 34. In response to the control signal, the transport mechanism unit 34 transports the gripping member 38 to the user's mouth. When the user brings the mouth into contact with the gripping member 38 carried to the mouth, contact is detected by the contact detection unit 36b, and a detection signal is transmitted to the computing unit 22. When the grasping member 38 is at the mouth position and the arithmetic unit 22 receives a detection signal indicating contact of the user's mouth, the arithmetic unit 22 transmits a control signal for retreating the fork to the drive unit 36a. Upon receiving the control signal, the driving unit 36a moves the spoon and the fork by moving the fork backward. At this time, the retracted position detection unit 36d configured by a position sensor or the like detects the retracted position of the fork, detects that the fork has retracted to a predetermined position, and transmits a detection signal to the computing unit 22. The calculator 22 receives the detection signal and transmits a control signal for stopping the fork to move back to the drive unit 36a. The drive unit 36a receives the control signal and stops the fork from moving backward. By such an operation, food is provided to the user in an easy-to-eat state. However, the gripping member 38 is not limited to a fork and a spoon.

  The operation unit 16 is used by a user to specify the movement of the transport unit 14, and here is a joystick including a lever 40. When the user tilts the lever 40, the direction is detected by a sensor built in the operation unit 16, and is transmitted to the input I / F 20 of the main body unit 10 as an instruction signal indicating the direction. As shown in FIG. 1, the operation direction of the lever 40 is, for example, four directions, that is, the back A, the right B, the front C, and the left D. Further, when the lever 40 is pushed, the switch is turned on. When the user is requested to make a decision, when the lever 40 is pushed, a signal indicating the decision is input / output I / O of the main body 10. Sent to F26. The lever 40 is preferably shaped so that a user with a disability in the upper limb can easily operate with a jaw or the like. The operation unit 16 is not limited to a joystick, and a laser pointer or the like attached to the user's head may be used.

  The imaging unit 18 includes a camera 42 and an input / output interface (I / F) 44. The camera 42 is a color camera that can recognize the color of a subject, and can be, for example, a color CCD image sensor. The camera 42 is fixed so as to have a predetermined relative position with respect to the container 12 using a fixing tool, and is installed at a position where an entire image of the container 12 can be taken from above. When the imaging unit 18 receives a control signal instructing imaging from the computing unit 22 via the input / output I / F 26 and the input / output I / F 44, the imaging unit 18 captures an image of the container 12 and the food placed therein by the camera 42. To do. The captured image data is transmitted to the computing unit 22 via the input / output I / F 44 and the input / output I / F 26.

  Next, control of the meal support apparatus in the present embodiment will be described. The meal support apparatus is controlled according to the flowchart shown in FIG. Each step of the flowchart is performed by executing in the computing unit 22 a control program that can be executed by the computer and stored in the storage unit 24. The following processing is performed in a state where food is served in the container 12 and the container 12 is set in the abutting portion 32 of the main body 10 of the meal support apparatus.

  In step S10, an initial setting process is performed. When the main power source of the meal support apparatus is turned on, the process waits for the sub power source to be turned on. In the present embodiment, the lever 40 of the operation unit 16 also serves as a sub power source, and the sub power source is turned on by operating the lever 40 in either direction. When the main power supply and the sub power supply are turned on, the computing unit 22 performs an initial setting process for the entire apparatus. In the initial setting process, the transport mechanism 34 and the gripping mechanism 36 are trial-operated to perform initial recognition of each position in the operation range. In addition, each element of the grip counter G (m, n) that records the number of grip operations is set to zero. The grip counter G (m, n) will be described later.

  In step S12, the mouth position adjustment process is performed. The user or his / her attendant operates the transport mechanism 34 and the gripping mechanism 36 by operating the lever 40 of the operation unit 16, and places the gripping member 38 at the mouth position where the user can easily eat food. Move. A control signal is transmitted to the computing unit 22 by pushing the lever 40 when the gripping member 38 is at a good position, and the position is set as the mouth position.

  In step S14, the transport mechanism 34 and the gripping mechanism 36 are moved to the standby position. The standby position is a position set in advance as a state in which the arms 34 a to 34 b of the transport mechanism unit 34 are folded and the gripping member 38 of the gripping mechanism unit 36 faces the user side.

  In step S16, the division area of the container 12 is selected. The computing unit 22 confirms the setting state of the mode switch 28. When the operation mode is set to the manual mode or the semi-automatic mode, any one of the divided regions 12a to 12d is selected when the user tilts the lever 40 of the operation unit 16. For example, it is preferable to set so that the divided areas 12a, 12b, 12c, or 12d are selected when the tilting direction of the lever 40 is tilted to the back A, right B, front C, or left D, respectively. . When the operation mode is set to the automatic mode, the user gives an instruction to start the operation by tilting the lever 40 of the operation unit 16 to the back A. That is, in the automatic mode, the divided area is selected based on the process for determining the food to be grasped later, and therefore, only the operation start instruction needs to be given at this point.

  In step S18, it is determined whether or not a divided area has been selected. If an instruction to select a divided area or start an operation has not been given, the process returns to step S16 to wait until a selection or instruction is given. If an instruction to select a divided area or start an operation is given, the process proceeds to step S20. In this step, if the user wants to end the meal, the meal support apparatus can be stopped.

  In step S20, it is determined whether or not the operation mode is a manual mode. The calculator 22 shifts the process to step S26 if the mode changeover switch 28 is set to the manual mode, and shifts the process to step S22 if it is set to the automatic mode or the semi-automatic mode.

  In step S <b> 22, the container 12 and an image of food placed in the container 12 are acquired, and a gripping position is determined based on the image data. The gripping position is determined as any one of the layout positions obtained by further finely dividing the divided areas 12a to 12d of the container 12. In this embodiment, as shown in FIG. 4, each divided region 12a to 12d is divided into nine arrangement positions (m, n) (m is an integer of 1 to 4 indicating each divided region, and n is each divided region. 1 is an integer of 1 to 9 indicating the layout position), and any position is determined as the gripping position. FIG. 4 shows an example in which the divided region 12a is divided into nine arrangement positions. Step S22 is a subroutine and is processed according to the flowchart shown in FIG.

  Hereinafter, a gripping position determination process when the operation mode is the automatic mode or the semi-automatic mode will be described. In step S22-1, the entire image IM1 including the container 12 and the food arranged in the container 12 is acquired. The computing unit 22 transmits an imaging signal to the imaging unit 18. When the imaging unit 18 receives the imaging signal, the camera 42 installed above the container 12 captures a color image including the container 12 and the food placed in the container 12 as illustrated in FIG. 6, and the entire image IM1. To the computing unit 22. In FIG. 6, different colors are indicated by different hatching for convenience.

  In step S22-2, the color sample image IM2 is extracted. The color sample image IM2 is an image of a divided region where food is not arranged from the container 12, that is, a divided region where the ground color of the container 12 is exposed entirely. Taking FIG. 6 as an example, the position of the divided region 12c in the entire image IM1 is set in advance. Then, by not placing food in the divided region 12c of the container 12 and leaving the ground color of the container 12 exposed, a predetermined divided region 12c is cut out from the entire image IM1 to be a color sample image IM2. be able to. In this way, extracting a part of the entire image IM1 as the color sample image IM2 performs image processing in accordance with the actual environment by taking into account the actual situation such as the color unevenness, dirt, and shadow of the container 12. There is an advantage in that it can.

  In step S22-3, a color sample is extracted from the color sample image IM2. The calculator 22 calculates the hue h and saturation s of each pixel for all the pixels constituting the color sample image IM2. In the present embodiment, the processing is performed assuming that the hue h and the saturation s are both expressed in 256 gradations. Then, the appearance frequency C (h, s) is obtained for each combination of hue h and saturation s in the color sample image IM2. The initial value of the appearance frequency C (h, s) is 0. For example, if there is a pixel with hue h = 120 and saturation s = 80, the value of C (120, 80) is increased by one. Then, after examining the appearance frequency C (h, s) for all the pixels of the color sample image IM2, the color sample D (h, s) is set to 1 if the appearance frequency C (h, s) is equal to or greater than a predetermined threshold. If the appearance frequency C (h, s) is smaller than a predetermined threshold, the color sample matrix D (h, s) is set to zero. Thereby, a color sample matrix D (h, s) having a value of 0 or 1 for each combination of hue h and saturation s can be obtained. The combination of the hue h and the saturation s where the value of the color sample matrix D (h, s) is 1 is determined to be the ground color of the container 12, and the value of the color sample matrix D (h, s) is 0. It can be determined that the combination of hue h and saturation s is not the background color of the container 12.

  Note that the color sample extraction method is not limited to the above processing. For example, there is a method of capturing an image of the entire container 12 or the divided regions 12a to 12d before the meal is served, and acquiring a color having a high appearance frequency as a color sample among the pixel groups constituting the image data. In this case, the ground color of the container 12 can be made different for each divided region, and food can be more easily recognized by selecting a divided region to be arranged according to the type of food. Further, there is a method in which the boundary portion 12e that partitions the divided regions 12a to 12d shown in FIG. 4 is extracted as an edge extraction process or a preset region, and the color of the boundary portion 12e is acquired as a color sample. In this case, the color of the boundary portion 12e is the same as the ground color in the divided regions 12a to 12d of the container 12. Further, there is a method of obtaining a color sample from the color of the serving position after gripping and removing the food accumulated in the container 12.

  The ground color of the container 12 serving as a color sample is preferably a color that does not exist in food, but there is no problem if it is not completely the same as food. Moreover, even if the ground color of the container 12 is a single color or a plurality of colors, the ratio of the colors used in the region where the color sample is extracted and the region where the food is arranged is the same. That's fine.

  Further, in this embodiment, every time an instruction to select a divided area or start an operation is given in step S18, the color sample image IM2 and the color sample matrix D (h, s) are obtained, and the color sample is updated. ing. As a result, it is possible to cope with the influence of the color change on the image due to the light irradiation to the container 12 during the meal. However, the update timing is not limited to this, and the update may be performed each time a divided region selection or operation start instruction is executed a predetermined number of times in step S18. The update may be performed at the timing when the gripping mechanism unit 36 is moved to the standby position. Further, when there is almost no change in the eating environment, the color sample image IM2 and the color sample matrix D (h, s) are used only once for each meal when the meal support apparatus is turned on. You may perform the process which calculates | requires.

  In the present embodiment, hue and saturation are used among hue, saturation, and brightness, which are attributes of colors expressed in the HSB method. The color expression method is not limited to this, but the lightness is greatly affected when the eating environment changes due to lighting or the like. In the present embodiment, in order to eliminate such inconvenience, a method of expressing a color by hue and saturation is adopted.

  In step S22-4, the operation mode is determined. The computing unit 22 shifts the process to step S22-5 if the mode switch 28 is set to the semi-automatic mode, and shifts the process to step S22-8 if the mode switch 28 is set to the automatic mode. .

  First, the processing when the operation mode is set to the semi-automatic mode will be described from S22-5 to S22-7. In step S22-5, partial image IM3 is acquired. When the operation mode is set to the manual mode or the semi-automatic mode, any one of the divided regions 12a to 12d should be selected in step S18, and therefore, as shown in FIG. The image of the selected divided area is cut out as a partial image IM3. In the present embodiment, the partial image IM3 is described as being extracted as an image of 256 pixels vertically × 256 pixels horizontally. In FIG. 7, different colors are indicated by different hatching for convenience.

  In step S22-6, a binary image IM4 obtained by binarizing the partial image IM3 is generated. First, the pixel value of each pixel of the binary image IM4 of 256 pixels × 256 pixels corresponding to each pixel constituting the partial image IM3 is initialized to zero. Next, the hue h and saturation s of the target pixel selected from the pixels constituting the partial image IM3 are obtained with 256 gradations. Then, it is checked whether or not the value of the color sample matrix D (h, s) corresponding to the hue h and saturation s of the target pixel is 1. When the value of the color sample matrix D (h, s) corresponding to the hue h and the saturation s is 1, the pixel value of the binary image IM4 corresponding to the target pixel is changed to 1, and the hue h and the saturation are changed. When the value of the color sample matrix D (h, s) corresponding to the degree s is 0, the pixel value of the binary image IM4 corresponding to the target pixel is maintained at 0. Since the ground color of the divided area where food is arranged is a color represented by a color sample, the ground color of the container is represented by examining whether or not each pixel is a color included in the color sample. It can be broadly classified as a pixel or a pixel representing food.

  For example, when the hue of the pixel (vertical 50, horizontal 50) is 120 and the saturation is 80 in the partial image IM3 of the partial area 34a, 2 when the color sample matrix D (120, 80) is 1. The pixel of the value image IM4 (vertical 50, horizontal 50) is set to 1.

  By sequentially selecting all the pixels of the partial image IM3 as the target pixel and performing the above-described processing, the binary image IM4 can be generated. That is, a pixel having a pixel value of 0 in the binary image indicates that the corresponding pixel in the partial image IM4 is a color different from the ground color of the container, and a pixel having a pixel value of 1 in the binary image is a partial image. It shows that the corresponding pixel of IM4 is the same color as the background color of the container.

  In step S22-7, post-processing such as noise processing is performed. For example, it is preferable to perform reduction / enlargement processing and area processing on the binary image IM4.

  Next, the processing when the automatic mode is set will be described from S22-8 to S22-11. In step S22-8, the partial image IM3 is acquired. When the operation mode is set to the automatic mode, since the divided areas 12a to 12d are not selected in step S18, the image of each divided area is cut out as the partial image IM3 from the entire image IM1. In the present embodiment, partial images IM3a, IM3b, and IM3d are extracted as images of 256 pixels in the vertical direction and 256 pixels in the horizontal direction for the divided regions 12a, 12b, and 12d on which the food is arranged.

  In step S22-9, a binary image IM4 obtained by binarizing the partial image IM3 is generated. First, similarly to step S22-6, the pixel value of each pixel of the binary image IM4a of 256 pixels × 256 pixels corresponding to each pixel constituting the partial image IM3a is initialized to zero. Next, the hue h and saturation s of the target pixel selected from the pixels constituting the partial image IM3a are obtained with 256 gradations. Then, it is checked whether or not the value of the color sample matrix D (h, s) corresponding to the hue h and saturation s of the target pixel is 1. When the value of the color sample matrix D (h, s) corresponding to the hue h and the saturation s is 1, the pixel value of the binary image IM4 corresponding to the target pixel is changed to 1, and the hue h and the saturation are changed. When the value of the color sample matrix D (h, s) corresponding to the degree s is 0, the pixel value of the binary image IM4a corresponding to the target pixel is maintained at 0.

  The binary image IM4a can be generated by sequentially selecting all the pixels of the partial image IM3a as the target pixel and performing the above processing. Similarly, binary images IM4b and IM4d are generated for the partial images IM3b and IM3d, respectively. In this way, by determining whether or not each pixel of the partial image IM3 is a color in the color sample, a group of pixels having color information not in the color sample can be recognized as food.

  In step S22-10, post-processing such as noise processing is performed. For example, it is preferable to perform reduction / enlargement processing and area processing on the binary image IM4.

  In step S22-11, one of the binary images is selected as a representative binary image IM4. The frequency with which the pixel value is 0 is investigated for all the pixels constituting the binary images IM4a, IM4b, and IM4d corresponding to the partial images IM3a, IM3b, and IM3d, and the image with the highest frequency is selected as the representative binary image IM4. . That is, an image having the largest number of pixels having a color different from the ground color of the container is set as a representative binary image.

  After the processing of steps S22-5 to S22-7 or steps S22-8 to S22-11, the processing is shifted to step S22-12. In step S22-12, a gripping position for moving the gripping mechanism unit 36 is determined. The computing unit 22 calculates a predicted value F (m, n) of the effectiveness of the gripping operation for each arrangement position of the acquired binary image IM4. Here, m indicates the position of the divided area. For example, 1, 2, 3, and 4 are associated with the divided areas 12a, 12b, 12c, and 12d, respectively. n indicates the position of the serving position. In the present embodiment, as shown in FIG. 4, the divided regions 12 a to 12 d are each divided into nine arrangement positions, and are integers 1 to 9 in order of upper left, upper center, upper right, left center,. Represented by The effectiveness prediction value F (m, n) is determined by calculating the following parameters P1 (m, n), P2 (m, n), P3 (m, n) and P4 (m, n). Is done. And food can be reliably hold | gripped by trying holding | grip operation | movement with respect to the serving position (m, n) with a large value of this predicted value F (m, n).

  First, a layout position image IM5 corresponding to each layout position is cut out from the binary image IM4. In the present embodiment, as shown in FIG. 8, nine arrangement position images IM5-1 to IM5-9 corresponding to nine arrangement positions (m, n) are extracted.

  The parameter P1 (m, n) is a value representing the percentage of the pixel values of the pixels constituting the arrangement position images IM5-1 to IM5-9 as a percentage. That is, it is the proportion of food-like pixels in each serving position. For example, in the layout position image IM5-1 corresponding to the layout position (1, 1), if the number of pixels having a pixel value of 0 with respect to the total number of pixels 7225 is 3000, the parameter P1 (1, 1) = 0.415. The parameter P1 (m, n) indicates a predicted value of the probability that there will be food at the serving position (m, n). The greater the value of P1 (m, n), the higher the possibility that food is present at the serving position (m, n). Therefore, the predicted value F (m, n) for the effectiveness of the gripping action is larger. To be.

  The parameter P2 (m, n) is determined by whether or not P1 (m, n) is equal to or greater than a predetermined threshold value. P2 (m, n) is set to 1 when P1 (m, n) is equal to or greater than the threshold, and P2 (m, n) is set to 0 when P1 (m, n) is smaller than the threshold. Is done. The parameter P2 (m, n) is a binarized prediction value of the probability that there will be food at the serving position (m, n). The predicted value F (m, n) of the effectiveness of the gripping operation is set to a large value for the layout position (m, n) in which P2 (m, n) is set to 1.

  The parameter P3 (m, n) is set to a value equal to the grip counter G (m, n). The update process of the grip counter G (m, n) will be described later. The parameter P3 (m, n) indicates the number of times gripping has already been attempted for the layout position (m, n) (grip trial history).

  The parameter P4 (m, n) is set to 0 if the value of the parameter P2 (m, n-1) at the left adjacent layout position is 1 when n is 2, 3, 5, 6, 8, 9 If the value of the parameter P2 (m, n-1) at the left adjacent layout position is 0, it is set to 1. When n is 1, 4, or 7, there is no arrangement position on the left side, so P4 (m, n) = 1 is set. The parameter P4 (m, n) indicates whether or not there is food in the vicinity that is obstructed when food is gripped by the gripping mechanism 36. That is, in the present embodiment, the gripping member 38 is moved to a position away from the left of the food to be gripped by a certain distance, the gripping member 38 is lowered vertically there, and the left of the food to be gripped from there The gripping member 38 is moved so as to be translated to the edge, and the food is gripped by being sandwiched between the fork and spoon of the gripping member 38. In other words, the food is scooped and collected with a spoon from the left side of the gripping position, and the food is gripped by sandwiching the fork thereon. Therefore, if another food is served at the left side of the food you want to grip, there is a possibility that the food will also be gripped, and the amount of food to be gripped may be larger than the intended amount. There is a high possibility of spilling when carrying the grasped food to the mouth. Therefore, when there is a high possibility that food is present at the serving position (m, n−1) adjacent to the left of the focused serving position (m, n), it is difficult to grasp the parameter P4 (m, n) as 0. If there is a low possibility that food is present at the serving position (m, n-1) adjacent to the left of the focused serving position (m, n), the parameter P4 (m, n) is set to 1 and the grip is easy Suppose that

  Therefore, it is preferable to change the setting method of the parameter P4 (m, n) depending on which direction the gripping member 38 of the gripping mechanism 36 is configured to grip food. For example, when the gripping member 38 has a structure for gripping food from the front to the back, the parameter P2 (m, n + 3) of the serving position (m, n + 3) in front of the serving position (m, n) of interest. If the value of n + 3) is 1, the parameter P4 (m, n) is set to 0, and the parameter P2 (m, n + 3) of the layout position (m, n + 3) before the layout position (m, n) of interest is set. If the value is 0, the parameter P4 (m, n) is set to 1.

数式（１）における第１項の分子Ｐ１（ｍ，ｎ）＋Ｐ２（ｍ，ｎ）は、現在の画像を処理した結果により決定される。ここで、Ｐ１（ｍ，ｎ）は単なる百分率であるが、Ｐ２（ｍ，ｎ）を加算することによって、Ｐ１（ｍ，ｎ）がある一定の割合以上であった場合を最も優先し、他のパラメータの影響を受け難くしている。特に、Ｐ１（ｍ，ｎ）の値の差が小さく、かつ、Ｐ２（ｍ，ｎ）を決定する閾値の前後に値が分布している場合に顕著となる。 Using the parameters P1 (m, n), P2 (m, n), P3 (m, n), and P4 (m, n) calculated in this way, the effectiveness prediction value F based on the equation (1). Calculate (m, n). Here, P5 is a predetermined constant.

The first term numerator P1 (m, n) + P2 (m, n) in Equation (1) is determined by the result of processing the current image. Here, P1 (m, n) is a simple percentage, but by adding P2 (m, n), the case where P1 (m, n) is equal to or greater than a certain ratio gives the highest priority. It is hard to be influenced by the parameters of. This is particularly noticeable when the difference between the values of P1 (m, n) is small and the values are distributed before and after the threshold value for determining P2 (m, n).

  For example, if the threshold for determining P2 (m, n) is 0.40, and P1 (m, 1) = 0.39 and P1 (m, 2) = 0.41, then P1 (m, n) 1) + P2 (m, 1) = 0.39, and P1 (m, 2) + P2 (m, 2) = 1.41. As described above, even when there is no large difference in P1 (m, n), if it is equal to or greater than the threshold value, the effectiveness prediction value F (m, n) becomes high and is not easily influenced by other parameters. .

  Further, the denominator 1 + P3 (m, n) of the first term in the formula (1) is determined by the history of the gripping operation so far. That is, the effectiveness prediction value F (m, n) is lowered as the number of gripping operations performed in the past with respect to the layout position (m, n) increases. The value of P1 (m, n) may become large due to reflection of illumination, seasonings, container dirt, etc., even though there is actually no food. P3 (m, n) is for preventing repeated gripping operations with respect to such an arrangement position (m, n).

  In addition, P4 (m, n) × P5 in the second term in Expression (1) indicates that priority is given to the serving position where there is no food that obstructs the gripping operation. P5 is a coefficient that determines how much heavier P4 (m, n) is set to F (m, n), and is a value appropriately determined according to the structure of the gripping mechanism section 36.

  Effectiveness prediction values F (m, n) are respectively calculated for the layout position images IM5-1 to IM5-9. Then, the layout position having the highest effectiveness prediction value F (m, n) is determined as the gripping position. The gripping position is set on the left side of the layout position with the highest effectiveness prediction value F (m, n). When the gripping position is determined in step S22-12, the process returns to step S22 of the main routine.

  Thus, it can be said that F (m, n) is a value obtained by correcting the position of the food analyzed by the image processing based on the past history of the gripping operation and the positional relationship with other food. This can reduce the influence of reflections and shadows of light or the like in the image when a plurality of gripping operations are performed.

  When the gripping position is determined as described above, the process proceeds to step S24. In step S24, the gripping member 38 is automatically moved with respect to the determined gripping position.

  If it is determined in step S20 that the operation mode is the manual mode, the process proceeds to step S26. In step S26, it is determined whether or not the operation unit 16 has been operated. The operation at this time is a food selection operation performed by the user in the manual mode, and is held at the position of the food that the user wants to eat in the previously selected divided area by the operation of the operation unit 16. This means an operation of aligning the position of the member 38. When selection input of grasped food is performed, the process proceeds to step S28. If the operation unit 16 is not operated and a predetermined time has elapsed, the process proceeds to step S30.

  In step S <b> 28, the gripping member 38 is moved on the container 12 based on the operation of the operation unit 16. When the user tilts the lever 40 of the operation unit 16 in any of the back A, right B, front C, and left D directions, the gripping member 38 is moved in a direction corresponding to the operation direction of the lever 40. When the movement is completed, the process returns to step S26 to wait for further operations.

  In step S30, food is actually gripped. The gripping member 38 moved to the left side of the food to be gripped manually, semi-automatically or automatically is lowered, and after moving the gripping member 38 so as to move in parallel to the left edge of the food to be gripped. The food is gripped by being sandwiched between the fork and spoon of the gripping member 38. And the holding member 38 is raised so that it may leave | separate from a serving position in the state which hold | gripped the food.

  In step S32, the grip counter G (m, n) is updated. The value of m corresponding to the divided area selected as the binary image IM4 and the value of the grip counter G (m.n) corresponding to the layout position n determined as the gripping position in the divided area are increased by one.

  In step S34, the food grasped by the grasping member 38 is conveyed to the user's mouth. In step S36, food is provided to the user. In step S38, the gripping mechanism 36 is moved to the standby position. Thereafter, the process returns to step S16, and the above process is repeated.

  An example of an actual gripping operation is shown below. In the following description, it is assumed that six gripping operations have already been performed and some of the served food has already been gripped and transported and removed. FIG. 9 shows a partial image IM3 of the divided region 12a cut out from the entire image IM1 obtained by capturing the container 12 at this time. In the partial image IM3 of FIG. 9, it is assumed that an image 52 reflecting light is imprinted together with the food image 50. A binary image IM4 is generated from the partial image IM3. FIG. 10 shows layout position images IM5-1 to IM5-9 corresponding to the layout position n (n = 1 to 9) of the divided region 12a (m = 1).

  Here, based on the layout position images IM5-1 to IM5-9 in FIG. 10, P1 (1 from the ratio of the area having a color different from the ground color of the container 12 in each layout position image IM5-1 to IM5-9. , N) = [0.7, 0.5, 0.9, 0, 0, 0, 0.4, 0.3, 0]. When the threshold value for determining P2 (m, n) is 0.4, P2 (1, n) = [1,1,1,0,0,0,1,0,0]. The parameter P4 (m, n) indicating whether there is other food on the left side of each serving position n is P4 (1, n) = [1,0,0,1,1,1,1, 0, 1]. It is assumed that the parameter P3 (m, n) indicating the gripping history is P (1, n) = [0, 1, 1, 1, 1, 0, 0, 0, 1]. Further, P5 = 0.25.

  Then, the predictive value F (m, n) of effectiveness is F (1, n) = [1.95,0.75,0.95,0.25,0.25,0.25,1.65. , 0.3, 0.25], and n = 1 (upper left) is determined to be the most effective layout position for gripping.

  Here, at the n = 2,3 arrangement position corresponding to the light reflection region 52, the gripping has already been attempted once, and the effectiveness is predicted by the parameters P3 (1,2) and P3 (1,3). The values F (1,2) and F (1,3) are small. That is, it is possible to prevent an erroneous gripping operation due to light reflection or the like from being repeated.

  In the present embodiment, when the operation mode is the automatic mode or the semi-automatic mode, the gripping member 38 is moved to the layout position where the value of F (m, n) is the largest, and when the operation mode is the manual mode, The gripping member 38 is moved to the position instructed by the operation unit, and food is gripped and transported. A mode different from this is provided, and the arrangement position is close to the position of the gripping member 38 moved by the operation to the operation unit 16 by obtaining F (m, n) as in the automatic mode or the semi-automatic mode. In addition, the gripping member 38 may be positioned at a serving position where the value of F (m, n) is high, and food may be gripped and transported.

  As described above, according to the present embodiment, when a user with a disability in the upper limbs eats in a general home or facility, the image is processed using color information, and the container is arranged. Providing food to users without repeated meaningless gripping movements to accurately detect the position of food that is indefinite or indeterminate, and to consider the effectiveness of gripping movements for multiple serving positions can do. Thereby, the user can eat smoothly.

It is a figure which shows the external appearance of the meal assistance apparatus in embodiment of this invention. It is a functional block diagram of the meal assistance apparatus in embodiment of this invention. It is a figure which shows the flowchart of the meal assistance method in embodiment of this invention. It is a figure which shows the division | segmentation area | region and serving position of the container of the meal assistance apparatus in embodiment of this invention. It is a figure which shows the flowchart of the subroutine of step S20 in embodiment of this invention. It is a figure which shows the example of the whole image of the container in embodiment of this invention. It is a figure which shows the example of the partial image in embodiment of this invention. It is a figure which shows the example of the layout position image in embodiment of this invention. It is a figure which shows the example of the partial image in embodiment of this invention. It is a figure which shows the example of the layout position image in embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 Main body part, 12 containers, 12a-12d division | segmentation area | region, 12e boundary part, 14 conveyance part, 16 operation part, 18 imaging part, 20 input interface, 22 calculator, 24 memory | storage part, 26 input / output interface, 28 mode change switch , 30 Notification means, 32 Abutting section, 34 Transport mechanism section, 34a to 34d arm, 34e Rotation drive section, 36 Grip mechanism section, 36a Drive section, 36b Contact detection section, 36c Grip detection section, 36d Retraction position detection section, 38 gripping member, 40 lever, 42 camera, 44 input / output interface, 50 food image, 52 light reflected image.

Claims (4)

A meal support device that supports a user's meal by grasping the food placed in the container and carrying it to the user's mouth,
A gripping mechanism for gripping food,
Dividing the container into a plurality of regions;
Calculating a predictive value of effectiveness indicating the possibility of gripping food for each of the plurality of regions;
Determining a gripping position for gripping from the plurality of regions based on the effectiveness prediction value;
A meal support apparatus that controls the gripping mechanism so as to grip the gripping position. The meal support apparatus according to claim 1,
An imaging unit for acquiring an image of the food that is arranged in the container and the container;
A meal support apparatus, wherein an occupancy ratio of food is obtained for each of the plurality of regions from the image, and a predicted value of the effectiveness is calculated based on at least the occupancy ratio. In the meal support apparatus according to claim 1 or 2,
Storing a gripping trial history for each of the plurality of regions;
A meal support apparatus, wherein the effectiveness prediction value is calculated based on the grasping trial history. In the meal assistance device according to any one of claims 1 to 3,
Find the relative positional relationship between the foods in the container,
A meal support apparatus that calculates the predicted value of effectiveness based on the positional relationship.

Images