JP5950256B2 - Feeding movement measuring system and measuring method - Google Patents

Description

  The present invention relates to an eating exercise measuring system and a measuring method for measuring an eating exercise when a person swallows food and drinks, medicines and the like.

  The “feeling of thirst” that is a sensation when the food or drink passes through the throat is an important factor as an index representing the “taste” of the food or drink. Therefore, in the process of developing foods and drinks, sensory tests are widely conducted to obtain evaluations by conducting questionnaires etc. on subjects in order to evaluate feelings (feeling of drinking) such as “ease of swallowing” and “feeling of drinking” . However, in this sensory test, it is difficult to objectively evaluate a “sense of swallowing a thing” because a subjective sensitivity such as a subject's sense and preference greatly affects. Also, in the development stage of orally ingested pharmaceuticals, it is used as a reference when evaluating the ease of swallowing of tablets, etc., and determining the shape and size with less burden. However, as described above, in the sensory test using a questionnaire or the like, an objective evaluation cannot be obtained because the subjectivity of the subject greatly affects. Therefore, in order to objectively evaluate the sense of swallowing things such as “ease of swallowing” of foods and drinks and orally ingested drugs, various methods for measuring swallowing movements expressed during eating and drinking have been proposed. Traditionally, swallowing movements have been measured by special X-ray imaging called videofluoroscopic examination of swallowing (VF). However, in this inspection method, since the exposure dose of the subject increases in proportion to the number of tests, there is a risk that the physical burden on the subject increases. In addition, it is necessary to mix a contrast medium into the test target product, which affects the taste and properties of the test target product, so that accurate evaluation is difficult.

  Therefore, as a non-invasive measurement method of swallowing movement, for example, in Patent Document 1, a plurality of pressure sensors are attached to the front neck of a subject, and swallowing is performed based on the pressure change of the front neck measured by the pressure sensor. A method for measuring movement is disclosed. Patent Document 2 proposes a method for measuring swallowing movement by attaching an electrode to the throat surface of a subject to acquire waveform data of the surface myoelectric potential of the throat muscle and performing frequency analysis on the waveform data. ing.

JP 2009-160459 A JP 2009-39516 A

  However, when a measurement tool such as a sensor or an electrode is attached to the subject during measurement of swallowing movement, there is a problem that the burden on the subject increases. In addition, since the test subject swallows food and drinks and medicines in a very unnatural state with the measuring device worn around the throat, the “sense of swallowing things” cannot be properly evaluated. Furthermore, there is a drawback that it is difficult to distinguish and judge whether the measured value is based on swallowing motion or due to other motion such as breathing of the subject.

  In addition to the sensation of swallowing such a thing as an index representing the “taste” of a food or drink, the sensation of biting is another important factor. For this reason, in the process of developing foods and beverages, sensory tests such as questionnaires for subjects are also conducted on the senses obtained from chewing such as “ease of biting” and “feeling of biting”. However, since it is difficult to obtain an objective evaluation in a sensory test using a questionnaire or the like, the masticatory movement expressed during eating and drinking is measured using a pressure sensor and electrodes as in the case of the swallowing movement described above. Although a method has been proposed, the use of these measuring tools has inherent problems similar to those described above.

  That is, the present invention has been proposed in view of the above-described problems inherent in the prior art, and has been proposed to solve these problems, and the mastication movement and swallowing movement during feeding are in a natural state without imposing a burden on the subject. It is an object of the present invention to provide a feeding movement measuring system and a measuring method that can be measured by the above method.

In order to solve the above-mentioned problem and achieve an intended purpose, a feeding movement measuring system according to claim 1 of the present invention includes:
Three or more light emitting means for irradiating the head and neck surface with light having different wavelengths from each other;
Imaging means for spectrally acquiring spectral image data in a time series by dispersing the light reflected from the head and neck surface;
Surface normal vector calculation means for calculating each surface normal vector data by illuminance difference stereo method based on each spectral image data acquired by the imaging means;
The position of the tracking shape in each surface normal vector data by template matching using the surface normal vector data of the tracking shape including the uneven shape on the head and neck surface and each surface normal vector data. Template matching means for obtaining
The gist of the invention is to measure the mastication movement or the swallowing movement by obtaining the position of the tracking shape in time series by the template matching means.
According to the first aspect of the present invention, the surface normal vector data of the surface of the subject's head and neck is calculated by the illuminance difference stereo method, and the surface of the head and neck associated with the mastication movement or the swallowing movement is template-matched from the surface normal vector data. Since the tracking shape is tracked as a template, mastication or swallowing movement can be accurately measured. At this time, since it is not necessary to attach a sensor, an electrode, or the like to the subject at the time of measuring the mastication movement or the swallowing movement, it is possible to measure a natural mastication movement or a swallowing movement. In addition, since it is configured to track the tracking shape including the uneven shape of the head and neck surface, it is less affected by the skin properties of the subject's head and neck surface, and even if there is some looseness or wrinkles on the head and neck surface The chewing or swallowing movement can be accurately measured. In addition, the configuration is such that surface normal vector data is calculated by irradiating light having different wavelengths from three or more light emitting means, acquiring a spectral image by the imaging means, and calculating the surface normal vector data. That is, it is possible to obtain surface normal vector data of the surface of the head and neck that changes with time, which cannot be calculated when using light of the same wavelength, by simultaneously irradiating light of three or more different wavelengths. In addition, by splitting light of three or more different wavelengths, spectral normal images can be obtained instantaneously and surface normal vector data can be calculated. It is not necessary to acquire a spectral image while emitting light one by one, and measurement can be performed efficiently. Further, since it is not necessary to attach a pressure sensor, an electrode, or the like to the subject at the time of measurement, the measurement of mastication or swallowing movement can be performed easily and at low cost.

The gist movement measuring system according to claim 2 is characterized in that the light emitting means is configured to irradiate light having wavelengths that do not interfere with each other in the spectral sensitivity characteristics of the imaging means.
According to the invention of claim 2, since it is configured to irradiate light having a wavelength that does not interfere with each other in the spectral sensitivity characteristics of the imaging means, it is possible to obtain an accurate spectral image by splitting the light reflected from the head and neck surface. it can. Accordingly, since accurate surface normal vector data can be calculated based on a highly accurate spectral image, it is possible to measure the mastication movement or the swallowing movement with high accuracy.

The feeding movement measuring system according to claim 3 includes color image generation means for generating color image data by color-coding according to the direction of each surface normal vector of the surface normal vector data. Is the gist.
According to the invention of claim 3, since the color image generating means color-codes the surface normal vector data and generates the color image data, the change of the head and neck surface can be visually recognized, and the chewing motion or The swallowing movement can be observed and analyzed in detail.

The eating movement measuring system according to claim 4 comprises:
The template matching means includes
A template setting unit for setting the template from reference surface normal vector data;
A storage unit for storing the template set by the template setting unit;
A search area setting unit for setting a search area for performing template matching on each surface normal vector data;
A position specifying unit for specifying the position of the tracking shape by performing template matching between the template stored in the storage unit and each surface normal vector data within the search region set by the search region setting unit; Is the gist.
According to the fourth aspect of the present invention, since the template and the surface normal vector data are subjected to template matching within the search region, the matching range is limited, and the calculation amount in the position specifying unit can be suppressed.

In the feeding movement measuring system according to claim 5, the tracking shape includes an uneven shape caused by thyroid cartilage or cricoid cartilage on the head and neck surface,
The range of the template is set in a linear region that passes through the reference point set in the tracking shape in the surface normal vector data and extends in the moving direction of the thyroid cartilage or cricoid cartilage during swallowing movement,
The gist of the present invention is that the search area is set to a linear area that passes through the reference point in the surface normal vector data and includes the range of the template.
According to the invention of claim 5, the range of the template is set to a linear area extending in the moving direction of the thyroid cartilage or cricoid cartilage during swallowing movement, and the linear area that includes the template area as the search area Since the area is set, it is possible to reduce the amount of calculation for template matching as compared with the case where the template range or the search area is set as a surface area. That is, since the template range and the search region are set to linear regions using the characteristics of the thyroid cartilage or cricoid cartilage moving linearly, unnecessary calculations at the time of template matching can be reduced.

The eating movement measuring system according to claim 6, wherein the tracking shape includes an uneven shape caused by a mandible on a head surface,
The range of the template is set in a linear region that passes through the reference point set in the tracking shape in the surface normal vector data and extends in the movement direction of the mandible during mastication movement,
The gist of the present invention is that the search area is set to a linear area that passes through the reference point in the surface normal vector data and includes the range of the template.
According to invention of Claim 6, while setting the range of a template to the linear area | region extended in the moving direction of the mandible at the time of a chewing movement, the linear area | region which includes the range of a template is set as a search area | region. Therefore, the amount of calculation at the time of template matching can be suppressed as compared with the case where the range of the template or the search area is set as a surface area. That is, since the template range and the search region are set to linear regions using the characteristic that the mandible moves linearly during mastication, unnecessary calculations during template matching can be reduced.

In order to solve the above-mentioned problems and achieve the intended purpose, the eating movement measuring method according to claim 7 of the present invention comprises:
Three or more light emitting means irradiate the head and neck surface with light having different wavelengths from each other, the light reflected by the head and neck surface is dispersed, and spectral image data is acquired in time series,
Based on each spectral image data acquired by the imaging means, each surface normal vector data is calculated by illuminance difference stereo method,
The position of the tracking shape in each surface normal vector data by template matching using the surface normal vector data of the tracking shape including the uneven shape on the head and neck surface and each surface normal vector data. Seeking
The gist is to measure the mastication movement or the swallowing movement by obtaining the position of the tracking shape in time series.
According to the invention of claim 7, the surface normal vector data of the surface of the subject's head and neck is calculated by the illuminance difference stereo method, and the surface of the head and neck associated with the mastication movement or the swallowing movement by template matching from the surface normal vector data. Since the tracking shape is tracked as a template, mastication or swallowing movement can be accurately measured. At this time, since it is not necessary to attach a sensor, an electrode, or the like to the subject at the time of measuring the mastication movement or the swallowing movement, it is possible to measure a natural mastication movement or a swallowing movement. In addition, since the tracking shape including the uneven shape on the head and neck surface is tracked, it is less affected by the skin properties of the subject's head and neck surface, and even if there is some slack or wrinkles on the head and neck surface In addition, chewing and swallowing movements can be accurately measured. In addition, the surface normal vector data is calculated by irradiating light having different wavelengths from three or more light emitting means, acquiring a spectral image by the imaging means, and calculating surface normal vector data. That is, it is possible to obtain surface normal vector data of the surface of the head and neck that changes with time, which cannot be calculated when using light of the same wavelength, by simultaneously irradiating light of three or more different wavelengths. In addition, by splitting light of three or more different wavelengths, spectral normal images can be obtained instantaneously and surface normal vector data can be calculated. It is not necessary to acquire a spectral image while emitting light one by one, and measurement can be performed efficiently. Further, since it is not necessary to attach a pressure sensor, an electrode, or the like to the subject at the time of measurement, the measurement of mastication or swallowing movement can be performed easily and at low cost.

The gating movement measuring method according to claim 8 is characterized in that the light emitting means irradiates light having wavelengths that do not interfere with each other in the spectral sensitivity characteristics of the imaging means.
According to the eighth aspect of the present invention, since the light having a wavelength that does not interfere with each other in the spectral sensitivity characteristics of the imaging means is used, the light reflected from the head and neck surface can be dispersed to obtain an accurate spectral image. it can. Accordingly, since accurate surface normal vector data can be calculated based on a highly accurate spectral image, it is possible to measure the mastication movement or the swallowing movement with high accuracy.

The gating movement measuring method according to claim 9 is summarized in that color-separated image data is generated by color-coding according to the direction of each surface normal vector of the surface normal vector data.
According to the invention of claim 9, since the surface normal vector data is color-coded and color image data is created, a change in the head and neck surface can be visually observed, and the chewing or swallowing movement can be observed in detail and Can be analyzed.

The method for measuring eating movement according to claim 10 comprises:
The tracking shape includes an uneven shape caused by thyroid cartilage or cricoid cartilage on the head and neck surface,
The range of the template is set in a linear region that passes through the reference point set in the tracking shape in the surface normal vector data and extends in the moving direction of the thyroid cartilage or cricoid cartilage during swallowing movement,
A linear area that passes through the reference point and includes the range of the template is set in each surface normal vector data as a search area for performing template matching,
The gist is to specify the position of the tracking shape by template matching the template and each surface normal vector data in the search region.
According to the invention of claim 10, the range of the template is set to a linear region extending in the moving direction of the thyroid cartilage or cricoid cartilage during swallowing movement, and the linear region that includes the template range as the search region Since the area is set, it is possible to reduce the amount of calculation for template matching as compared with the case where the template range or the search area is set as a surface area. That is, since the template range and the search region are set as linear regions using the characteristic that the thyroid cartilage or cricoid cartilage moves linearly during swallowing, unnecessary calculations during template matching can be reduced.

The method for measuring eating movement according to claim 11 comprises:
The tracking shape includes an uneven shape caused by the mandible on the head surface,
The range of the template is set to a linear region that passes through the reference point set in the tracking shape in the surface normal vector data and extends in the movement direction of the mandible during mastication movement,
A linear area that passes through the reference point and includes the range of the template is set in each surface normal vector data as a search area for performing template matching,
The gist is to specify the position of the tracking shape by template matching the template and each surface normal vector data in the search region.
According to the invention of claim 11, the template range is set to a linear region extending in the movement direction of the mandible during mastication, and a linear region including the template range is set as a search region Therefore, the amount of calculation at the time of template matching can be suppressed as compared with the case where the range of the template or the search area is set as a surface area. That is, since the template range and the search region are set to linear regions using the characteristic that the mandible moves linearly during mastication, unnecessary calculations during template matching can be reduced.

  Since the eating movement measuring system and measuring method according to the present invention are non-contact and non-invasive methods, natural mastication movements and swallowing movements can be accurately measured without placing a burden on the subject.

1 is a block diagram illustrating an overall configuration of a feeding movement measurement system according to Embodiment 1. FIG. (a) is the perspective view which shows the positional relationship of 3CCD camera, LED, and a test subject, (b) is the front view which looked at 3CCD camera and LED from the test subject side. It is explanatory drawing which shows the specification of each LED employ | adopted in the Example. It is explanatory drawing which shows a mode that the light of LED is disperse | distributed within 3CCD camera. It is explanatory drawing which shows the specification of the lens part of 3CCD camera employ | adopted in the Example. It is a graph which shows the relationship between the spectral sensitivity characteristic of 3CCD camera employ | adopted in the Example, and the wavelength of the light of each LED. (a) is explanatory drawing which shows the relationship between the light source and the brightness of an observation surface, (b) is explanatory drawing of an illuminance difference stereo method. It is a figure which shows the setting screen displayed on the output means. It is explanatory drawing showing the relationship between NCC, SAD, and SSD. (a) is an explanatory view showing the correspondence between the direction of the surface normal vector color-coded by the color image generation means and the color, (b) is a color image of the throat surface displayed on the output means, and (c) is a color image An actual image of the throat surface, which is the base of the minute image, is shown. It is a flowchart figure which shows the whole flow of the eating movement measuring method which concerns on an Example. It is a flowchart figure which shows a surface normal vector data calculation process. It is a flowchart figure which shows a setting process. It is a flowchart figure which shows a template matching process. It is a graph which shows the time change of the position of the top part of a thyroid cartilage. It is a flowchart figure which shows the flow which displays the color image of a throat surface. 6 is a graph showing the measurement results of Experimental Example 1. FIG. It is a graph which shows the measurement result of Experimental example 2. It is a graph which shows the measurement result of Experimental example 3. It is a conceptual diagram which shows a human head and neck part from a temporal region. It is a block diagram which shows the whole structure of the eating movement measuring system which concerns on Example 2. FIG. 6 is a graph showing spectral characteristics of a bandpass filter according to Example 2. FIG. It is a figure which shows the setting screen of Example 2 displayed on the output means. (a) is a color image of the surface of the head and neck displayed on the output means, (b) is an explanatory diagram showing the direction and color of the surface normal vector color-coded by the color image generation means, and (c) is an output means. It is explanatory drawing which displayed the color image of the displayed head and neck part surface with the pattern shown to Fig.10 (a). It is a graph which shows the measurement result of the test subject 1 in Example 4, Comprising: (a) shows the case where squid is ingested, (b) shows the case where tofu is ingested, (c) ingests warabimochi Show the case. It is a graph which shows the measurement result of the test subject 2 in Example 4, Comprising: (a) shows the case where squid is ingested, (b) shows the case where tofu is ingested, (c) ingests warabimochi Show the case. It is a graph which shows the measurement result of the subject 3 in Example 4, Comprising: (a) shows the case where squid is ingested, (b) shows the case where tofu is ingested, (c) ingests warabimochi Show the case. It is a graph which shows the measurement result of the test subject 4 in Example 4, Comprising: (a) shows the case where squid is ingested, (b) shows the case where tofu is ingested, (c) ingests warabimochi Show the case.

  Next, the eating movement measuring system and the measuring method according to the present invention will be described in detail below with reference to the accompanying drawings by giving a preferred embodiment. The “feeding exercise” as used in the present invention includes a mastication exercise and a swallowing exercise performed in a series of operations of taking food and drink into the oral cavity and then feeding it into the stomach. Here, the “chewing exercise” is an exercise that forms a bolus of a size that is easy to swallow by chewing food and drink taken into the oral cavity using the lower jaw, teeth, tongue, etc. and mixing with saliva. The “swallowing exercise” refers to an exercise when swallowing (swallowing) a bolus of food in the mouth such as a beverage formed in the oral cavity or a chewed food. According to the eating movement measuring system and measuring method of the present invention, “ease of swallowing”, “ease of eating”, “feeling of getting caught in the throat”, “feeling of drinking”, etc. By measuring the position of the tracking target on the head and neck surface that moves with swallowing movements, various sensations felt when swallowing can be evaluated easily, quickly, accurately, with good sensitivity, and objectively It becomes possible. Similarly, according to the eating movement measuring system and measuring method of the present invention, various sensations felt when chewing food or drink in the oral cavity, such as “ease of chewing”, “feeling of chewing”, etc. By measuring the position of the tracking target on the surface of the head and neck that moves with movement, it is possible to evaluate the object simply, quickly, accurately, with good sensitivity, and objectively.

  In the eating movement measuring system and measuring method of Example 1, as shown in FIG. 20, the throat surface (head and neck surface) caused by the top (throat Buddha) of the thyroid cartilage 12 that moves up and down during swallowing movement accompanying eating By setting the shape including the concavo-convex shape to the tracking shape (template) and identifying the position of the tracking shape (specifically, the reference point P) appearing on the throat surface during swallowing movement in time series, It is configured to measure movement. The reference point P is a tracking point that is arbitrarily set within the tracking shape. In the first embodiment, the thyroid cartilage moves up and down following the swallowing movement, with a concavo-convex shape remarkably appearing on the throat surface. A location corresponding to the top of 12 is designated as the reference point P. However, in the present invention, as the tracking shape for tracking the swallowing movement accompanying feeding, any part of the subject's throat surface (head and neck surface) may be set as the tracking shape as long as it includes a concavo-convex shape. . For example, as the tracking shape, a shape including an uneven shape of the throat surface due to the cricoid cartilage 14 below the thyroid cartilage 12 and moving up and down with the thyroid cartilage 12 during swallowing may be set. Also, the reference point P is not limited to the location corresponding to the top of the thyroid cartilage 12, and the reference point P may be set at any location within the tracking shape. In addition, by designating the uneven shape caused by the thyroid cartilage 12 as a reference point, the vertical movement accompanying the swallowing movement is remarkably expressed, and the swallowing movement can be accurately tracked.

  FIG. 1 is a block diagram illustrating the overall configuration of the eating movement measurement system 10 according to the first embodiment. The feeding movement measuring system 10 of the first embodiment images three LEDs (light emitting means) 16, 18, and 20 that irradiate light of different wavelengths toward the throat surface including the thyroid cartilage 12 of the subject, and the throat surface. A 3CCD camera (imaging means) 22 for acquiring spectral image data and a controller 24 for specifying the position of the tracking shape (reference point P) in time series from the spectral image data acquired by the 3CCD camera 22. Yes. The eating movement measuring system 10 basically includes an input means 25 for inputting the position of the reference point P and the like to the control device 24 and an output means 26 for outputting a calculation result in the control device 24. Is done.

[About LED and 3CCD camera]
As shown in FIG. 2A, the three LEDs 16, 18, 20 and the 3CCD camera 22 are three-dimensional in an XYZ orthogonal coordinate system with the lens portion 28 (described later) of the 3CCD camera 22 as the origin. It is provided in the space. As shown in FIG. 2A, the XYZ orthogonal coordinate system was set such that the X axis and the Z axis extend in the horizontal direction and the Y axis extends in the vertical direction. At the time of measurement, the subject is seated on the front of the 3CCD camera 22 so that the throat surface is opposed to the lens unit 28 in the Z-axis direction, and the 3CCD camera 22 takes an image. It is assumed that the thyroid cartilage 12 during the swallowing movement moves in the Y-axis direction (vertical direction). The lens unit 28 is separated from the subject's throat surface by a predetermined distance (for example, about 30 cm).

  As shown in FIG. 2B, the three LEDs 16, 18, and 20 are arranged concentrically around the 3CCD camera 22 on the XY plane. Further, the LEDs 16, 18, and 20 are spaced apart from each other by a predetermined distance, and the three LEDs 16, 18, and 20 are arranged in an inverted equilateral triangle shape. The distance between the LEDs 16, 18, and 20 is set to about 12 cm in consideration of an average value (11.6 cm) in the width dimension of the adult throat surface (see dimension 1 in FIG. 8). Each LED 16, 18, and 20 is installed so as to be directed toward the subject's throat surface, and is configured so that light is emitted from each LED 16, 18, and 20 toward the throat surface in different irradiation directions. Moreover, each LED16,18,20 is comprised so that the light from which a wavelength differs mutually may be irradiated. In the first embodiment, as the LEDs 16, 18, and 20, a red LED 16 that emits red light, a blue LED 18 that emits blue light, and a green LED 20 that emits green light are employed. The three LEDs 16, 18, and 20 are designed to irradiate light to the throat surface all at once, and the throat surface is a combination of red, blue, and green light depending on the uneven shape of the throat surface. Configured to illuminate. The red LED 16, blue LED 18 and green LED 20 are adapted to irradiate light having wavelengths of 465 nm, 520 nm and 625 nm, respectively, in consideration of the spectral sensitivity characteristics (described later) of the 3CCD camera 22. The specifications of the LEDs 16, 18, and 20 employed in Example 1 are shown in FIG.

  The 3CCD camera 22 separates the light reflected from the throat surface into red, blue, and green colors and acquires spectral image data (luminance data) of each color in time series. As shown in FIG. 4, the 3CCD camera 22 is configured such that a lens unit 28, a spectroscopic unit 32, and three CCDs (charge coupled devices) 34, 36, and 38 are accommodated inside the camera body 30. FIG. 5 shows the specifications of the lens unit 28 employed in the first embodiment. The light splitting unit 32 splits the light synthesized by the three colors into light for each wavelength of red, blue, and green. In the first embodiment, a dichroic prism is used as the light splitting unit 32. As shown in FIG. 4, the dichroic prism internally reflects blue light and red light in different directions and disperses light by transmitting green light.

  The three CCDs are composed of a red CCD 34 corresponding to red light, a blue CCD 36 corresponding to blue light, and a green CCD 38 corresponding to green light. Each CCD 34, 36, 38 receives light of a corresponding color that is split by the beam splitting unit 32. Here, the curve graph of FIG. 6 shows the spectral sensitivity characteristics of the red CCD 34, the blue CCD 36, and the green CCD 38 employed in the first embodiment. Moreover, the bar graph of FIG. 6 has shown the wavelength of the light which the said red LED16, blue LED18, and green LED20 irradiate. As described above, the wavelengths of the light emitted from the LEDs 16, 18, and 20 are set to values at which the spectral sensitivities of the CCDs 34, 36, and 38 do not interfere with each other and the spectral sensitivities are as high as possible. Then, the red CCD 34 acquires luminance data (hereinafter referred to as red spectral image data) from the input red light. The blue CCD 36 acquires luminance data (hereinafter referred to as blue spectral image data) from the input blue light. Further, the green CCD 38 acquires luminance data (hereinafter referred to as green spectral image data) from the input green light. Spectral image data obtained by the CCDs 34, 36, and 38 is output to a surface normal vector calculation means 40 (described later) of the control device 24, respectively. Note that the number of frames per unit time of the 3CCD camera 22 of the first embodiment is 30 fps. However, the number of frames of the 3CCD camera 22 is not limited to 30 fps. If the number of frames is increased, the tracking shape can be tracked in detail, while the processing speed is lowered. Therefore, the number of frames of the 3CCD camera 22 is appropriately selected according to the required measurement accuracy.

[About the control unit]
As shown in FIG. 1, the control device 24 includes surface normal vector calculation means 40 for calculating surface normal vector data from spectral image data of each color acquired by the 3CCD camera 22, and surface normal vector data of a tracking shape. Template matching means 46 for obtaining the position of the tracking shape (reference point P) in each surface normal vector data by performing template matching using the template and each surface normal vector data. The control device 24 also includes a data storage means 42 for temporarily storing the surface normal vector data calculated by the surface normal vector calculation means 40, and a color for creating color image data from the surface normal vector data. And a minute image creating means 44.

[About surface normal vector calculation means]
The surface normal vector calculation means 40 is configured to calculate surface normal vector data by the illuminance difference stereo method based on the spectral image data input from the CCDs 34, 36, and 38. Here, the illuminance difference stereo method is an image processing method for calculating the surface normal vector of the observation surface based on the luminance (brightness) of the reflected light of the light irradiated on the observation surface. The principle of this illuminance difference stereo method will be briefly described below. Note that the illuminance difference stereo method is based on the premise that the observation surface is a completely diffuse reflection surface, and applies the principle of Lambert reflection in which the brightness looks the same regardless of the direction of the observation surface. As for the throat surface of the subject in Example 1, it is assumed that the light emitted from the LEDs 16, 18, and 20 is a completely diffuse reflecting surface on which Lambertian reflection is performed.

  As shown in FIG. 7A, when the surface is illuminated by one light source from the θ direction with respect to the surface normal direction of the observation surface, the intensity of the light source is L and the reflectance of the surface is ρ. The luminance x of the surface photographed by the camera is expressed by the following equation (1). Further, if the normal direction vector n ′ multiplied by the reflectance is the surface normal vector n, and the light source direction vector s ′ multiplied by the intensity of the light source is the light source vector s, it can be expressed by the following equation (2). . In each equation, a vector is represented by an arrow symbol above the symbol.

Next, as shown in FIG. 7B, the observation surface is illuminated sequentially from _three different light sources (light source direction vectors s ₁ , s ₂ , s ₃ ). A vector representing this direction is a vector S = [s ₁ s ₂ s ₃ ]. Assuming that the observed brightness is x ₁ , x ₂ , and x ₃ , a vector X = [x ₁ x ₂ x ₃ ] having these as elements is used.
^T is expressed by the equation (3), and the surface normal vector n of the observation surface can be obtained from the luminance values of the respective pixels of the three pieces of image data observed separately under three light sources having different directions.

  Here, since the illuminance difference stereo method described above uses images obtained by sequentially emitting light sources one by one and photographing the observation surface separately, the illuminance difference stereo method is used for a throat surface accompanied by an action such as swallowing movement. It is difficult to apply the stereo method as it is. Therefore, in Example 1, as described above, light of three colors is simultaneously irradiated onto the throat surface from the three LEDs 16, 18, and 20, and the reflected light is dispersed to obtain spectral image data that is luminance data for each color. By obtaining in real time, a surface normal vector n of the throat surface whose surface shape continuously changes with swallowing motion can be calculated (three-color light illuminance difference stereo method). That is, the surface normal vector calculation means 40 uses the red spectral image data obtained by the red CCD 34, the blue spectral image data obtained by the blue CCD 36, and the green spectral image data obtained by the green CCD 38. The surface normal vector n of the throat surface is calculated from the equation (3).

[Data storage means]
The data storage means 42 is a temporary storage device such as a RAM that stores the surface normal vector n data (surface normal vector data) calculated by the surface normal vector calculation means 40 in time series. The surface normal vector data stored in the data storage unit 42 is read when template matching is performed by the template matching unit 46 or when color image data is generated by the color image generation unit 44.

[About template matching means]
As shown in FIG. 1, the template matching unit 46 includes a template setting unit 48 for setting a template from reference surface normal vector data, and a template storage unit (memory for storing the template set by the template setting unit 48). Part) 56, a search area setting unit 50 for setting a search area for template matching with respect to the surface normal vector data, and template matching between the template and each surface normal vector data, the tracking shape (reference point P And a position specifying unit 52 that specifies the position of ().

  The template setting unit 48 sets a template from reference surface normal vector data among the surface normal vector data stored in the data storage means 42. As described above, the template includes surface normal vector data corresponding to a tracking shape including an uneven shape caused by the top of the thyroid cartilage 12. Specifically, as shown in FIG. 8, the measurer inputs the position of the reference point P and the template range via the input unit 25 on the setting screen displayed on the output unit 26 described later, and the input value Is set as a template from the surface normal vector data. In the first embodiment, the surface normal vector data serving as a reference for creating a template is set to the surface normal vector data on the throat surface before the subject starts swallowing motion. The setting screen displays image data (hereinafter referred to as reference image data) corresponding to the reference surface normal vector data, and the reference point P is a position corresponding to the top of the thyroid cartilage 12 in the reference image data. Is designated by the input means 25. The template range is determined by slidably moving the cursor 54a to the left and right via the input means 25 on the range setting bar 54 provided on the setting screen, and determining the number of pixels (number of pixels) in the range as the template. Is called. When the number of pixels is input, the template setting unit 48 passes through the reference point P and extends linearly with the length of the number of pixels input in the movement direction (Y-axis direction) of the thyroid cartilage 12 during swallowing movement. Is set as the template range. That is, the template setting unit 48 sets a line segment extending in the vertical direction with the reference point P as the center as a template. In the first embodiment, the template range in the width direction (X-axis direction) is 1 pixel.

  The template storage unit 56 is a temporary storage device such as a RAM that temporarily stores the template set by the template setting unit 48. The template stored in the template storage unit 56 is read when the position specifying unit 52 performs template matching. The search area setting unit 50 sets a search area that is a range in which template matching is performed on the surface normal vector data. In the first embodiment, the search area setting unit 50 sets a linear area that passes through the reference point P and includes the range of the template as the search area. Specifically, in the search area setting bar 58 provided on the setting screen shown in FIG. 8, the measurer slides the cursor 58 a to the left and right via the input unit 25, and the search range in the surface normal vector data is expressed in pixels. Enter a number (number of pixels). Based on the number of input pixels, the search area setting unit 50 searches for a linear area extending in the Y-axis direction that is longer than the template and centered on the reference point P, as shown in FIG. Set as area. Note that the number of pixels in the Y-axis direction input as the search area is set to a value larger than the number of pixels in the template range. Further, the dimension of the search area in the width direction (X-axis direction) is 1 pixel as in the template.

  The position specifying unit 52 is configured to read each surface normal vector data stored in the data storage unit 42 and a template stored in the template storage unit 56 and perform template matching in the search region. . Here, the principle of template matching will be briefly described below. Template matching is a technique for calculating a similarity by superimposing a template and a specific image and specifying the position of the template when the similarity is maximum. In Example 1, NCC (Normalized Cross-Correlation) was used for calculating the similarity.

  This NCC is expressed by the equation (4). However, the template size (number of elements of the pixel value) is M × N, the pixel value at the template position (i, j) is T (i, j), and the pixel value of the image superimposed with the template is I (i , j).

  As shown in FIG. 9, when considering a vector T and a vector I of M × N elements of the template T (i, j) and the target image I (i, j), NCC is a cosine formed by the vector. Therefore, when NCC is used, the closer to 1, the higher the similarity.

For calculating the similarity, a known similarity calculation method can be employed in addition to NCC. For example, as shown in FIG. 9, in similarity calculation, SSD (Sum of Squared Difference), which is the sum of squares of absolute differences in luminance, or SAD (Sum, which is the sum of absolute differences in luminance).
of Absolute Difference) can also be used. Formulas of SSD and SAD are shown in Formulas 5 and 6. When SSD and SAD are used, the closer to 0, the higher the similarity.

  The position specifying unit 52 calculates the similarity with the template in each surface normal vector data read from the data storage means 42 in the search area. Then, the position of the reference point P in the surface normal vector data is specified from the position of the template having the maximum similarity value in the search area. That is, the position specifying unit 52 tracks the tracking shape in the swallowing movement by specifying the reference point P in each surface normal vector data acquired in time series.

[About color image creation means]
The color image generation unit 44 is configured to generate color image data based on the surface normal vector data stored in the data storage unit 42. This color image data is an image obtained by color-dividing each surface normal vector of the surface normal vector data according to the direction and color-dividing according to the uneven shape of the throat surface by the output means 26 (hereinafter referred to as a color image). ) Is data for displaying. In the first embodiment, as shown in FIG. 10A, the color image generating unit 44 classifies the surface normal vectors into eight directions on the XY plane, and uses eight types of colors corresponding to each direction. Color coding is performed. For example, when the surface normal vector at a predetermined position on the throat surface is rightward on the XY plane, the color image generation unit 44 generates image data that displays the color at the position on the throat surface in yellow ( (See FIG. 10 (b)). Similarly, when the surface normal vector at a predetermined position on the throat surface is upward on the XY plane, the color image generation unit 44 generates image data for displaying the color at the position on the throat surface in blue. (See FIG. 10 (b)). In FIGS. 10A and 10B, a pattern is added instead of the color. FIG. 10C is an actual image of the throat surface that is the base of the color image of FIG. The part corresponding to the top of the thyroid cartilage 12 on the throat surface appears as a part where the color changes radially in the color image.

  The input means 25 is an input device such as a mouse or a keyboard operated by a measurer. As described above, the position of the reference point P and the template range are set with respect to the template setting unit 48 when setting a template. It comes to input. Further, the search area range is input to the search area setting unit 50 when the search area is set. The output means 26 is a display device such as a liquid crystal display device, and is configured to output the calculation result of the control device 24. That is, the output means 26 displays the time change of the position of the reference point P specified by the position specifying unit 52 (see FIG. 15). Here, in the first embodiment, the output unit 26 uses the position of the reference point P specified by the above-described reference image data as the reference position, and the pixel amount (hereinafter referred to as the movement) by which the reference point P has moved in the Y-axis direction from the reference position. It is designed to display the time change (called pixel amount). Further, as shown in FIG. 10B, the output means 26 displays a color image on the throat surface based on the image data created by the color image creation means 44. As described above, the output unit 26 is configured to display a setting screen used when setting a template in the template setting unit 48 or setting a search region in the search region setting unit 50. .

[Operation of Example 1]
Next, a method for measuring eating movement by the above-described eating movement measuring system 10 will be described below. As shown in FIG. 11, the method for measuring the eating movement includes step S <b> 1 for irradiating light toward the throat surface (head and neck surface) of the subject, step S <b> 2 for obtaining spectral image data, and surface normal vector data. Step S3 for calculating the value, Step S4 for setting the template and search region, Step S5 for performing template matching, and Step S6 for outputting the position of the apex P of the identified thyroid cartilage 12.

  Red light, blue light, and green light emitted from the red LED 16, blue LED 18, and green LED 20 reach the throat surface at different irradiation angles to illuminate the throat surface. As shown in FIG. 4, the light reflected by the throat surface passes through the lens unit 28 of the 3CCD camera 22 and enters the camera body 30 to reach the spectroscopic unit 32. The light splitting unit 32 splits the light into red, blue, and green light while the light passes through. The red light, the blue light and the green light separated by the spectroscopic unit 32 are respectively input to the red CCD 34, the blue CCD 36 and the green CCD 38, and spectral image data (red spectral image data, blue color light) are respectively input to the CCDs 34, 36 and 38. (Spectral image data, green spectral image data) is acquired in time series for each photographing frame (frame) of the 3CCD camera 22 (step S2). At this time, as shown in FIG. 6, the wavelengths of the red, blue, and green light are set to values at which the spectral sensitivity characteristics of the CCDs 34, 36, and 38 do not interfere with each other and the spectral sensitivity increases. In each CCD 34, 36, 38, accurate spectral image data for each color is acquired. The spectral image data acquired by each CCD 34, 36, 38 is output to the surface normal vector calculation means 40.

  When spectral image data of each color is input to the surface normal vector calculation means 40, as shown in FIG. 12, the surface normal vector calculation means 40 calculates the surface normal vector data by the illuminance difference stereo method (step S7). . That is, the surface normal vector calculation means 40 calculates the surface normal vector data in time series based on the equation 3 based on the red spectral image data, the blue spectral image data, and the green spectral image data. As described above, the surface normal vector calculation means 40 uses the three-color light illuminance difference stereo method that calculates the surface normal vector data based on the three types of spectral image data acquired at the same time. As described above, it is not necessary to acquire image data while causing the LEDs to emit light one by one. Therefore, even on the throat surface whose shape continuously changes with swallowing movement, it is possible to calculate the surface normal vector data in real time without interruption. In addition, since the accurate spectral image data obtained by the CCDs 34, 36, and 38 is used in calculating the surface normal vector data, surface normal vector data with high accuracy in accordance with the uneven shape of the actual throat surface is obtained. be able to. The surface normal vector data calculated by the surface normal vector calculation means 40 is stored in time series in the data storage means 42 (step S8).

  In the setting process, as shown in FIG. 13, a template serving as a template matching reference is set by the template setting unit 48 from the calculated surface normal vector data (step S9). That is, as shown in FIG. 8, in the reference image displayed on the setting screen of the output means 26, the position corresponding to the top of the thyroid cartilage 12 is designated as the reference point P via the input means 25. In the range setting bar 54 on the setting screen, the cursor 54a is slid to the left and right to determine the template range (number of pixels). Based on these inputs, the surface normal vector data included in the linear range having the length of the number of input pixels and extending in the Y-axis direction around the reference point P is used as a template setting unit. 48. The template set by the template setting unit 48 is stored in the template storage unit 56 (step S10).

  The search area setting unit 50 sets a search area for performing template matching on the surface normal vector data (step S11). That is, as shown in FIG. 8, in the search area setting bar 58 on the setting screen of the output means 26, the cursor 58a is slid left and right via the input means 25 to determine the range (number of pixels) of the search area. . At this time, the number of pixels in the search area is set to a value larger than the template range. When the number of pixels of the search area is input, a linear area including the template range centering on the reference point P is set by the search area setting unit 50 as the search area. That is, the search area is set as a range extending in the vertical direction (Y-axis direction), like the template. When the search area setting unit 50 sets the search area, the setting process ends.

  Next, template matching processing by the template matching means 46 will be described (step S5). As shown in FIG. 14, in the template matching process, first, the position specifying unit 52 reads the surface normal vector data stored in the data storage means 42 and the template stored in the template storage unit 56 (step S12). Then, the position specifying unit 52 performs template matching within the search region set by the search region setting unit 50 based on the read surface normal vector data and the template (step S13). Specifically, the position specifying unit 52 calculates the degree of similarity between the surface normal vector data and the template in the search region using the formula (5). Then, the position of the reference point P (the position corresponding to the top of the thyroid cartilage 12) in the surface normal vector data is specified from the position of the template having the highest similarity (NCC is close to 1) in the search area. (Step S14). That is, the swallowing movement is measured by specifying the reference point P in each surface normal vector data acquired in time series. In Example 1, since NCC defined as the cosine of the vector was used when calculating the similarity (see FIG. 9), the thyroid cartilage 12 is not affected by the change in the brightness of the throat surface. It is possible to accurately calculate the similarity of the throat surface, whose surface shape changes with movement of the throat. However, as the calculation of the similarity, a known method capable of calculating the similarity, such as the above-described SSD (Equation 5) or SAD (Equation 6), can be appropriately employed.

  Here, as described above, the template range and the search region are set as linear regions in the template setting unit 48 and the search region setting unit 50. Therefore, since the amount of calculation at the time of template matching can be reduced compared with the case where the range of the template or the search area is set as a surface area, template matching can be performed efficiently, and the position of the reference point P can be quickly determined. Can be identified. When template matching for all the surface normal vector data ends (Yes in step S15), the template matching process ends.

  The position of the reference point P specified by the position specifying unit 52 by the template matching process is input to the output unit 26. As shown in FIG. 15, the output means 26 calculates a moving pixel amount from the input position of the reference point P, and displays a temporal change of the moving pixel amount in a graph (step S6). In FIG. 15, the horizontal axis indicates time, and the vertical axis indicates the moving pixel amount. In this way, the output means 26 graphs and displays the temporal change in the moving pixel amount of the reference point P, so that the movement of the tracking shape (that is, the thyroid cartilage 12) during the swallowing exercise can be observed in detail. It is possible to analyze the swallowing movement. Moreover, since it is not necessary to attach a sensor or the like to the subject or fix the neck of the subject at the time of measurement, natural swallowing movement can be accurately measured. Therefore, by analyzing the measured swallowing motion, it is possible to objectively evaluate “food texture” of food and drink, “ease of swallowing” of pharmaceuticals, and the like. In addition, since it is not necessary to attach a measuring instrument such as a sensor or an electrode to the subject, swallowing movement can be measured easily and at low cost.

  Next, a case where a color image of the throat surface is output will be described. Note that when displaying the color image of the throat surface, the surface normal vector data calculated by the above-described surface normal vector data calculation processing is used. As shown in FIG. 16, the color image generating unit 44 reads the surface normal vector data stored in the data storage unit 42 (step S16). Next, the color image generating unit 44 generates color image data from the read surface normal vector data (step S17). At this time, as shown in FIG. 10A, the color image generation unit 44 classifies the surface normal vectors into eight directions and generates color image data that is color-coded corresponding to each direction. Next, the color image creating unit 44 outputs the created color image data to the output unit 26 (step S18). And the output means 26 displays the image which color-coded the throat surface based on the input color image data as shown in FIG.10 (b). Thus, according to the eating movement measuring system 10 and the measuring method according to the first embodiment, it is possible to display the color-coded image according to the inclination of the throat surface (the direction of the surface normal vector). It is possible to visually grasp changes in the throat surface over time. Therefore, it is possible to observe and analyze the swallowing movement in detail based on the color image, and to evaluate “ease of swallowing” of food and drinks.

[Experimental Example 1]
Next, a measurement experiment using the eating movement measuring system 10 and the measuring method according to the present invention was performed. A man in his early twenties as a test subject swallowed water (200 ml) contained in a cup in several times, and the swallowing motion at that time was measured. FIG. 17 shows the measurement results. The ideal value in FIG. 17 is the moving pixel amount from the reference position when the position corresponding to the top of the thyroid cartilage 12 is visually identified from the image of the throat surface displayed on the output means 26. In Experimental Example 1, the similarity was calculated using the equation (4). As can be seen from the results of Experimental Example 1, the actual measurement value measured by the eating movement measurement system 10 according to the present invention substantially matches the ideal value. That is, according to the measurement system and the measurement method according to the present invention, the position of the reference point P (the position corresponding to the top of the thyroid cartilage 12) can be accurately tracked, and the swallowing movement can be accurately measured. I understand.

[Experimental example 2]
Next, as Experimental Example 2, a difference in swallowing motion due to a difference in beverage was measured using the swallowing motion measurement system 10 of the present invention. The subjects were allowed to drink the same amount (200 ml) of water and saline, and the measurement results when drinking water were compared with the measurement results when drinking saline. The saline solution had a concentration of 3% by weight, which makes people feel resistance to swallowing. The result of Experimental Example 2 is shown in FIG. In Experimental Example 2, the similarity was calculated using the equation (4).

  As can be seen from the graph of FIG. 18, the time until drinking is different between the case of drinking water and the case of drinking saline (the time of saline is longer). This is considered that it took time for the subject to feel that it was difficult to drink saline and to completely dry out the saline. Further, in the graph of saline solution, the slope of the rise and fall of the graph is gentler than that of water. This is determined to be a hesitation for the subject to swallow because the saline solution is hot. Thus, by using the measurement system and the measurement method of the present invention, it is possible to evaluate “ease of swallowing” and the like by comparing swallowing exercises when different types of foods and drinks and medicines are drunk.

[Experimental Example 3]
Next, an experiment was conducted to compare the difference in the measurement results when the method of calculating the similarity was changed. In the experimental example, NCC (see Equation 4) was used to calculate the degree of similarity as in Example 1 described above. In the comparative example, the SAD represented by Equation 6 was used for calculating the similarity. The test subjects were allowed to drink 200 ml of water, and the measurement results were compared. The measurement results are shown in FIG. Note that the ideal value is obtained by visually identifying the position of the apex of the thyroid cartilage 12 as described in Experimental Example 1.

  From the results of FIG. 19, it can be seen that the experimental example using NCC (solid line) matches the ideal value better than the comparative example using SAD (dashed line). This is because, as shown in FIG. 9, SAD is a vector difference, and SAD is used because it is greatly affected by the brightness value (vector length) of the throat surface that changes as the thyroid cartilage 12 moves up and down. In the similarity calculation, it is considered that an error has occurred when the position of the reference point P is specified. On the other hand, NCC depends on the gradient of the vector (cosine) and is not affected by the change in the brightness of the throat surface, so it is determined that the reference point P can be accurately identified. That is, it can be seen that the swallowing movement can be measured more accurately by using NCC for calculating the similarity in the present invention.

  In the eating movement measuring system and measuring method according to the second embodiment, the shape of the head and neck surface (specifically, the surface of the lower jaw) including the uneven shape and the shape that moves while maintaining the surface shape generally during the mastication movement are tracked. It is configured to measure the masticatory movement of the subject by specifying the position of the tracking shape in time series when chewing with eating. In the second embodiment, a shape including the reference point P1 is defined with a position corresponding to the mandible bulge 62a of the mandible moving up and down following the mastication motion as a reference point P1 with a concavity and convexity remarkably appearing on the lower jaw surface. Is set as a tracking shape (first template). Moreover, in the eating movement measuring system and the measuring method according to the second embodiment, as in the first embodiment, the shape includes the irregular shape of the head and neck surface (specifically, the throat surface), and the surface shape is generally set during the swallowing operation. The tracking shape is set as a tracking shape, and the swallowing movement of the subject is measured by specifying the position of the tracking shape in time series when swallowing with eating. In Example 2, an uneven shape appears remarkably on the throat surface, and a location corresponding to the top of the thyroid cartilage 12 that moves up and down following the swallowing motion is defined as a reference point P2, and a shape including the reference point P2 is obtained. It is set as a tracking shape (second template). In the present invention, as the tracking shape for tracking the mastication movement associated with eating, any part of the mandibular surface may be set as the tracking shape as long as it includes a concavo-convex shape. For example, as the tracking shape, a shape including an uneven shape of the mandible surface due to the mandibular base 64 and the mandibular angle 66 (see FIG. 20) of the mandible 62 that moves up and down during the mastication motion may be set as the tracking shape. In addition, by designating the concavo-convex shape resulting from the chin ridge 62a of the mandible 62 as the reference point P1, the vertical movement accompanying the mastication movement is remarkably expressed, and the mastication movement can be accurately tracked.

  Since the basic configuration of the eating movement measuring system according to the second embodiment is the same as that of the eating movement measuring system 10 shown in the first embodiment, components having the same functions are denoted by the same reference numerals. Detailed description will be omitted, and different configurations will be described below. FIG. 21 is a block diagram illustrating the overall configuration of the eating movement measurement system 10 according to the second embodiment. In the eating movement measuring system 10 according to the second embodiment, a plurality of (three in the embodiment) infrared LEDs 72, 74, and 76 that irradiate light in an infrared region that is an invisible region as light emitting means for irradiating light of different wavelengths. Is used to irradiate infrared rays toward the head and neck surface (throat surface and mandibular surface) including the thyroid cartilage 12 and the mandible 62 of the subject. In the eating movement measurement system 10 according to the second embodiment, a 3CCD camera 78 capable of photographing infrared rays is used as an imaging unit that captures the mandibular surface and the throat surface to acquire spectral image data. In the eating movement measuring system 10 according to the second embodiment, the infrared LEDs 72, 74, 76 and the 3CCD camera 78 are arranged in the same positional relationship as in the first embodiment, and the subject's throat surface and lower jaw surface are irradiated. The 3CCD camera 78 can be photographed by the infrared light. In addition, the three infrared LEDs 72, 74, and 76 according to the second embodiment irradiate light of a wavelength that does not interfere with each other in the near-infrared region that is invisible and that the normal 3CCD camera 78 has sufficient sensitivity. It is configured to

  Each of the infrared LEDs 72, 74, 76 and the 3CCD camera 78 is provided with a bandpass filter having a different transmission wavelength. When infrared rays having different wavelengths are irradiated on the head and neck surface (mandibular surface and throat surface), 3 One spectral image data can be acquired by the 3CCD camera 78. The spectral image data of each wavelength acquired by the 3CCD camera 78 is output to the surface normal vector calculation means 40 of the control device 24. Here, FIG. 22 shows the spectral characteristics of the bandpass filter used in the second embodiment. In other words, the second embodiment is configured to acquire spectral image data using near infrared rays of 780 ± 5 [nm], 850 ± 5 [nm], and 880 ± 5 [nm]. The number of frames per unit time of the 3CCD camera 78 of the second embodiment is 30 fps as in the first embodiment, but is not limited to this.

  Then, the surface normal vector calculation means 40 of the control device 24 according to the second embodiment obtains the surface normal vector data by the above-described three-color light illuminance difference stereo method based on the spectral image data of each wavelength acquired by the 3CCD camera 78. It is configured to calculate. In other words, infrared light of three different wavelengths from the three infrared LEDs 72, 74, 76 are simultaneously irradiated onto the head and neck surface (mandibular surface and throat surface), and the reflected light is separated to obtain spectral image data that is luminance data for each wavelength. Is obtained in real time, and the surface normal vector n of the mandibular surface whose surface shape continuously changes with the mastication motion and the surface normal vector of the throat surface whose surface shape continuously changes with the swallowing motion Each of n can be calculated. The surface normal vector data calculated by the surface normal vector calculating means 40 is stored in the data storage means 42 in time series, and when the template matching is performed by the template matching means 46, the color image generating means 44. The color image data is read when the color image data is created.

  Further, the template setting unit 48 of the template matching unit 46 provided in the eating movement measuring system 10 according to the second embodiment uses the surface normal vector as a reference among the surface normal vector data stored in the data storage unit 42. Two types of first template and second template can be set from the vector data. Here, the first template set by the template setting unit 48 is composed of surface normal vector data corresponding to a tracking shape including an uneven shape caused by the chin ridge 62a of the mandible 62, and the second template As in the first embodiment, the template is composed of surface normal vector data corresponding to a tracking shape including an uneven shape caused by the top of the thyroid cartilage 12. Specifically, as shown in FIG. 23, the position of the reference point P1 and the range as the first template are input to the setting screen displayed on the output unit 26 via the input unit 25, and based on the input value. Then, the template setting unit 48 sets the first template from the surface normal vector data. In Example 2, the surface normal vector data serving as a reference for creating the first template is set to the surface normal vector data on the mandibular surface before the subject starts mastication. The setting screen displays image data corresponding to the reference surface normal vector data (hereinafter referred to as reference image data), and the reference point P1 corresponds to the bulge 62a of the mandible 62 in the reference image data. It is set by designating the position to be performed by the input means 25. The range of the first template is the number of pixels (number of pixels) in the range set as the first template by sliding the cursor 55a to the left and right via the input means 25 in the range setting bar 54 provided on the setting screen. It is done by determining. When the number of pixels is input, the template setting unit 48 passes through the reference point P1 and extends in the length of the number of pixels input in the movement direction (Y-axis direction) of the mandible 62 during the mastication movement. Is set as the range of the first template. That is, the template setting unit 48 sets a line segment extending in the vertical direction around the reference point P1 as the first template. Here, the range in the width direction (X-axis direction) of the first template is set to 1 pixel. Since the setting of the second template corresponding to the top of the thyroid cartilage 12 is the same as the setting of the template in the first embodiment described above, detailed description thereof is omitted. In addition, the first template and the second template are temporarily stored in the template storage unit 56 and read when the position specifying unit 52 performs template matching.

  The search area setting unit 50 according to the second embodiment sets a linear area that passes through the reference point P1 and includes the range of the first template as a search area for measuring mastication movement, and measures swallowing movement. As a search area to be performed, a linear area that passes through the reference point P2 and includes the range of the second template is set. Specifically, as shown in FIG. 23, a search area setting bar 59 for setting a search area for mastication movement and a search area setting bar 58 for setting a search area for swallowing movement are displayed on the setting screen. The measurer searches the mastication motion in the surface normal vector data by sliding the cursor 59a of the search area setting bar 59 and the cursor 58a of the search area setting bar 58 to the left and right via the input means 25. The range and the range to search for swallowing motion are individually input in the number of pixels (number of pixels). Then, based on the number of input pixels, the search area setting unit 50 has a line extending in the Y-axis direction that is longer than the first template and centered on the reference point P1, as shown in FIG. A linear region as a masticatory motion search region, and a linear region extending in the Y-axis direction that is longer than the second template and centering on the reference point P2 is set as a swallowing motion search region It is configured. Note that the number of pixels in the Y-axis direction input as the search area for each motion is set to a value larger than the number of pixels in the template range corresponding to each motion.

  The position specifying unit 52 reads each surface normal vector data stored in the data storage unit 42, and also reads the first template and the second template stored in the template storage unit 56, so that the mastication is performed. Template matching is performed in each of the motion search region and the swallowing motion search region. That is, the position specifying unit 52 calculates the similarity with the first template in each surface normal vector data read from the data storage means 42 within the search area of the mastication movement. And the position of the reference point P1 in the said surface normal vector data is specified from the position of the 1st template in which the value of the similarity became the maximum within the search area | region of a mastication movement. That is, the position specifying unit 52 tracks the mastication movement by specifying the position of the first template (reference point P1) in each surface normal vector data acquired in time series. Similarly, the position specifying unit 52 calculates the similarity with the second template in each surface normal vector data read from the data storage means 42 within the search region for swallowing movement. Then, the position of the reference point P2 in the surface normal vector data is specified from the position of the second template having the maximum similarity value within the swallowing movement search region. That is, the position specifying unit 52 tracks the tracking shape in the swallowing movement by specifying the position of the second template (reference point P2) in each surface normal vector data acquired in time series. It has become.

  In the second embodiment, as shown in FIG. 24A, the color image generation unit 44 generates color image data obtained by color-coding the direction of the surface normal vector in the XY plane in all directions on the plane. The color image of the head and neck surface is generated and displayed on the output means 26 based on the color image data. FIG. 24B shows a state in which the orientations of the surface normal vectors color-coded by the color image creating unit 44 are displayed in gradation. FIG. 24C shows a state in which the surface normal vectors are classified into eight directions on the XY plane in the same manner as in the first embodiment, with the pattern changed to the color. That is, in the second embodiment, the bodily characteristic portions that protrude like the reference point P1 and the reference point P2 have a characteristic display mode in which the color changes radially in the color image.

[Operation of Example 2]
Next, a method for measuring eating movement by the eating movement measuring system 10 according to the second embodiment will be described. In the eating movement measuring method by the eating movement measuring system 10 according to the second embodiment, the wavelength of light irradiated on the subject is different from that in the first embodiment, and a plurality of reference points P1 and P2 are set. Differing in tracking the mandibular and throat surface movements. That is, in Example 2, by irradiating infrared rays as a light source for measurement, the subject does not feel dazzling at the time of measurement, and the burden on the subject can be reduced. In addition, by using near infrared light outside the wavelength region of a fluorescent lamp, measurement in an indoor environment under the fluorescent lamp is also possible. In this way, by using infrared rays as a light source for measurement, it is possible to measure in an environment close to the state in which the subject normally eats and drinks, and more natural feeding movements (chewing movements and swallowing movements) can be measured it can.

  Further, in the second embodiment, in the reference image displayed on the setting screen of the output unit 26, the position corresponding to the chin ridge 62a of the mandible 62 is designated as the reference point P1 through the input unit 25, and the thyroid cartilage 12 is specified. By designating the position corresponding to the top of the reference point P2, the surface normal vector data included in the linear range with the reference points P1 and P2 as the center is set as the first template and the second template. Is done. Then, by separately setting the search area including the first template and the search area including the second template, the surface normal vector data stored in the data storage unit 42, the first and second Template matching is performed in each search region based on the template. Specifically, the similarity between the surface normal vector data in each search region and each corresponding template is calculated by the position specifying unit 52 using the equation of Formula 5, and the similarity is the highest in each search region. Is increased (NCC is close to 1) based on the template position, the position of the reference point P1 in the surface normal vector data (the position corresponding to the chin ridge 62a of the mandible 62) and the position of the reference point P2 ( The position corresponding to the top of the thyroid cartilage 12) is specified. That is, by specifying the reference points P1 and P2 in each surface normal vector data acquired in time series, the masticatory movement and the swallowing movement are measured simultaneously and continuously. Then, the output means 26 displays the time variation of the moving pixel amount of the reference points P1 and P2 in a graph, so that the movement of the tracking shape (that is, the mandible 62 and the thyroid cartilage 12) at the time of feeding exercise can be simultaneously performed. It is possible to observe in detail and to analyze the feeding movement. Thus, since it is not necessary to attach a sensor or the like to the subject or fix the subject's neck at the time of measurement, it is possible to accurately measure the mastication movement and the swallowing movement simultaneously in a natural state. Therefore, by analyzing the measured eating movement, it is possible to objectively evaluate “feel” and “ease of swallowing” such as chewing response of the eaten food and drink. In addition, since it is not necessary to wear a measuring instrument such as a sensor or an electrode on the subject, feeding exercise can be measured easily and at low cost.

[Experimental Example 4]
Next, a measurement experiment performed using the eating movement measurement system 10 and the measurement method according to Example 2 will be described. In this experimental example, four men in their early twenties as subjects took three types of foods with different textures, and the eating movement measurement system 10 described above for the chewing and swallowing movements at that time was described above. And measured by the measuring method. In this experimental example, three types of cuttlefish that have been cut open and dried and hard to bite, soft and easy-to-break tofu, and hard but soft and sticky warabimochi are used and taken orally. A series of movements from swallowing to swallowing are measured. The measurement result of each subject is shown in FIGS. Table 1 shows the number of mastications performed from the start of each mastication to the first swallowing exercise, and Table 2 shows the number of frames (time) until the first swallowing exercise. Note that the number of frames taken by the 3CCD camera 78 used in this experimental example per unit time is 30 fps (frames / second).

  From Fig. 25 to Fig. 28 and Tables 1 and 2, when squid squid that is hard to chew is ingested, the number of chewing cycles before swallowing is increased compared to the intake of relatively soft warabimochi and tofu. It can also be seen that it takes time to chew until swallowing is possible. Contrary to this, it can be seen that when soft and brittle tofu is ingested, it moves to swallowing movements with a smaller number of chewing cycles than when ingesting hard squid and elastic warabimochi. It can also be seen that when ingesting tofu that is soft and less chewable, the chewing movement of the subject is smaller than that of the squid and warabimochi that are relatively chewy. That is, according to the eating movement measuring system 10 and the measuring method thereof according to the present invention, when different foods are ingested, it is possible to grasp the number of mastication times and the mastication time until shifting from mastication to swallowing. Based on the swallowing motion, sensations obtained from mastication such as “ease of biting” and “feeling of biting” can be evaluated in a non-contact and non-invasive state with respect to the subject. Further, since the swallowing movement is measured, it is possible to evaluate “ease of swallowing” and the like as in the first embodiment.

  Moreover, the relationship between the chewing movement and the swallowing movement for each ingested food shows a certain tendency regardless of the subject. From this experimental example, by using the eating movement measuring system and the measuring method according to the present invention, it is possible to relatively evaluate the relationship between the masticatory movement and the swallowing movement without depending on the individual sense of the subject. I understand that I can do it. In addition, for each ingested food, it is possible to grasp the distribution of the number of chewing cycles until swallowing and the size of the chewing movement, so it is possible to estimate what kind of texture the ingested food is I understand.

[Example of change]
The eating movement measuring system and measuring method according to the present invention are not limited to those shown in each embodiment, and for example, the following modifications are possible.
(1) In the embodiment, the LED is used as the light emitting means, but is not necessarily limited to the LED. As the light emitting means, other light emitting devices such as a halogen lamp, a dye laser, a light emitting diode (LD), and electroluminescence (EL) can be adopted as long as they can irradiate different wavelengths. However, LEDs are particularly preferably used from the viewpoint of cost and heat generation.
(2) In the embodiment, light is irradiated by three light emitting means, but the number of light emitting means may be four or more. In the embodiment, the three light emitting means are arranged in a regular triangle shape with the imaging means as the center. However, the light emitting means may be located at different positions as long as the light can be emitted at different irradiation angles with respect to the throat surface. May be placed in relationship
(3) In the embodiment, the 3CCD camera is employed as the imaging means, but other cameras can be employed as long as the spectral image data of each color can be acquired.
(4) In the embodiment, the template range and the search area are linear areas, but both may be planar areas. However, the search area needs to be set to an area including the template. Alternatively, only the template may be a linear range, and the search area may be a planar area including the template. Further, it is not always necessary to set a search area, and template matching may be performed on the entire range of the surface normal vector data.
(5) In the embodiment, the template setting unit sets the template by inputting the reference point and the range of the template via the input means. However, the template setting unit sets the reference point and template set in advance. The range may be set automatically. In addition, the search area setting unit does not necessarily set the search area based on the range of the search area input via the input unit, and the search area setting unit automatically sets a preset search area. You may do it.
(6) In the embodiment, the reference point or the like is input to the setting screen displayed on the output means via the input means. For example, the measurement point is set by directly touching the output means with the position of the reference point. It is also possible to adopt a touch panel method.
(7) In the embodiment, the template is created from the surface normal vector data before the subject starts the eating exercise, but the template may be created from the surface normal vector data during the eating exercise.
(8) In the embodiment, the position corresponding to the top of the thyroid cartilage is used as a reference point, and the position of the reference point is specified. However, any part of the throat surface may be set as the tracking shape. For example, when the cricoid cartilage is set in a tracking shape, swallowing movement can be measured even on the throat surface of a woman with a small amount of protrusion at the top of the thyroid cartilage (throat bud).
(9) In the embodiment, the position corresponding to the bulge of the mandible is used as a reference point, and the position of the reference point is specified. However, any part of the lower jaw surface may be set as the tracking shape. For example, when the uneven shape caused by the mandibular base or mandibular angle is set as the tracking shape, the masticatory movement can be measured even when the tip of the jaw is round and the amount of protrusion is small.
(10) In the example, the imaging means was set directly in front of the subject's head and neck surface, and the mandibular surface and throat surface were photographed. The imaging means may be set obliquely with respect to the part surface.
(11) In Example 1, the swallowing motion when swallowing a drink such as water was measured, but the swallowing motion when swallowing a solid or jelly-like food or swallowing a medicine such as a liquid or tablet was measured. Is also possible.
(12) In the examples, the case where only the swallowing motion is measured and the case where the chewing motion and the swallowing motion are measured are shown, but it is natural that only the chewing motion can be measured.

12 Thyroid cartilage, 14 Cricoid cartilage, 16 Red LED (light emitting means)
18 Blue LED (light emitting means), 20 Green LED (light emitting means)
22 3CCD camera (imaging means), 40 plane normal vector calculation means 44 color image creation means, 46 template matching means 48 template setting section, 50 search area setting section, 52 position specifying section 56 template storage section (storage section)
62 Mandible, 72 Infrared LED (light emitting means), 74 Infrared LED (light emitting means)
76 Infrared LED (light emitting means), 78 3 CCD camera (imaging means)

Claims (11)

Three or more light emitting means (16, 18, 20, 72, 74, 76) for irradiating the head and neck surface with light having different wavelengths from each other;
Spectral light reflected from the head and neck surface, imaging means for acquiring spectral image data in time series (22, 78),
Surface normal vector calculation means (40) for calculating each surface normal vector data by illuminance difference stereo method based on each spectral image data acquired by the imaging means (22, 78),
The position of the tracking shape in each surface normal vector data by template matching using the surface normal vector data of the tracking shape including the uneven shape on the head and neck surface and each surface normal vector data. Template matching means (46) for obtaining
A feeding movement measuring system configured to measure a mastication movement or a swallowing movement by obtaining the position of the tracking shape in time series by the template matching means (46).   The light emitting means (16, 18, 20, 72, 74, 76) is configured to irradiate light having wavelengths that do not interfere with each other in the spectral sensitivity characteristics of the imaging means (22, 78). Feeding movement measurement system.   3. Feeding exercise according to claim 1 or 2, further comprising color image generation means (44) for generating color image data by color-coding according to the direction of each surface normal vector of the surface normal vector data. Measuring system. The template matching means (46)
A template setting unit (48) for setting the template from the reference surface normal vector data;
A storage unit (56) for storing the template set in the template setting unit (48);
A search area setting unit (50) for setting a search area for performing template matching for each surface normal vector data;
A position for identifying the position of the tracking shape by template matching the template stored in the storage unit (56) and each surface normal vector data within the search region set by the search region setting unit (50) The eating movement measuring system according to any one of claims 1 to 3, further comprising a specific unit (52). The tracking shape includes an uneven shape caused by thyroid cartilage (12) or cricoid cartilage (14) on the throat surface,
The range of the template is a linear region that passes through a reference point set in the tracking shape in the surface normal vector data and extends in the moving direction of the thyroid cartilage (12) or cricoid cartilage (14) during swallowing movement Set to
5. The eating movement measuring system according to claim 4, wherein the search area is set to a linear area that passes through the reference point in the surface normal vector data and includes the range of the template. The tracking shape includes an uneven shape caused by the mandible (62) on the head surface,
The range of the template is set in a linear region that passes through the reference point set in the tracking shape in the surface normal vector data and extends in the movement direction of the mandible (62) during mastication movement,
6. The eating movement measuring system according to claim 4, wherein the search area is set to a linear area that passes through the reference point in the surface normal vector data and includes the range of the template. Light with different wavelengths is irradiated to the head and neck surface from three or more light emitting means (16,18,20,72,74,76), and the light reflected from the head and neck surface is dispersed to obtain a spectral image Data is acquired by the imaging means (22,78) in time series,
Based on each spectral image data acquired by the imaging means (22, 78), each surface normal vector data is calculated by illuminance difference stereo method,
The position of the tracking shape in each surface normal vector data by template matching using the surface normal vector data of the tracking shape including the uneven shape on the head and neck surface and each surface normal vector data. Seeking
A method for measuring an eating movement, characterized by measuring a mastication movement or a swallowing movement by determining the position of the tracking shape in time series.   The eating motion measurement according to claim 7, wherein the light emitting means (16, 18, 20, 72, 74, 76) irradiates light having wavelengths that do not interfere with each other in the spectral sensitivity characteristics of the imaging means (22, 78). Method.   The feeding movement measuring method according to claim 7 or 8, wherein color image data is generated by color-coding according to the direction of each surface normal vector of the surface normal vector data. The tracking shape includes an uneven shape caused by thyroid cartilage (12) or cricoid cartilage (14) on the throat surface,
The range of the template is a linear shape that passes through a reference point set in the tracking shape in the surface normal vector data and extends in the moving direction of the thyroid cartilage (12) or the cricoid cartilage (14) during feeding movement. Is set to
A linear area that passes through the reference point and includes the range of the template is set in each surface normal vector data as a search area for performing template matching,
The feeding movement measuring method according to any one of claims 7 to 9, wherein the template and each surface normal vector data are subjected to template matching in the search region to specify the position of the tracking shape. The tracking shape includes an uneven shape due to the mandible (62) on the head surface,
The range of the template is set in a linear region that passes through the reference point set in the tracking shape in the surface normal vector data and extends in the movement direction of the mandible (62) during mastication movement,
A linear area that passes through the reference point and includes the range of the template is set in each surface normal vector data as a search area for performing template matching,
The feeding movement measurement method according to any one of claims 7 to 10, wherein the template and each surface normal vector data are subjected to template matching in the search region to specify the position of the tracking shape.