JPWO2006064635A1 - Diagnostic system - Google Patents

Abstract

An object of the present invention is to improve the accuracy of diagnosis by discriminating between a normal state and an abnormal state in a diagnostic system that analyzes an image of each part of a human body. An imaging apparatus 2 that includes a diagnostic system 1 that includes a subject illumination unit 5 that irradiates a subject with a light source of a specific color light during imaging, and an image capturing unit 6 that converts reflected light of the subject from the light source into a pixel value and captures an image; The data management unit 22 that is connected to the imaging device 2 through a network and manages the image data transmitted from the imaging device 2, and the image data transmitted from the imaging device 2 is analyzed to detect a normal state and an abnormality. A data management device 13 including an image analysis unit 15 that determines a state is provided.

Description

  The present invention relates to a diagnostic system, and more particularly to a diagnostic system that performs medical diagnosis and the like from captured images of each part of a human body.

  2. Description of the Related Art Conventionally, a diagnostic system has been developed in which changes in the physiological state of a human body are detected by an imaging device such as a sensor or a CCD camera, and image data obtained thereby is analyzed to make a diagnosis such as a medical diagnosis or a beauty diagnosis.

  For example, Patent Document 1 discloses that skin color is analyzed regardless of image color reproduction of a display by quantifying objective biological information obtained from skin color such as face color in an image photographed with a digital camera. A diagnostic system is disclosed.

  Further, Patent Document 2 discloses that the physical condition (pulse wave, body temperature, skin in daily life) without installing the sensor in a place where a finger or the like is brought into contact in daily life without being conscious of being observed. A diagnostic system is disclosed that acquires data relating to color, respiratory rate, etc.) and diagnoses a health condition.

Patent Document 3 discloses a diagnostic system capable of managing and analyzing body temperature data measured by a basal body thermometer by referring to information via a network.
JP 10-165375 A JP 2000-139856 JP 2001-353157 A

  However, in the diagnostic system described in Patent Document 1, it is described that the data of the photographed image is quantified in order to analyze the skin color without being influenced by the image color reproduction of the display. There was no description of the configuration of the imaging device itself for obtaining data.

  In addition, in the diagnostic system described in Patent Document 2, it is described that sensors are provided at “positions where fingers are easy to touch” or “positions where voice is likely to be emitted” in order to acquire data related to physical condition in daily life. However, there has been no description about the configuration of the sensor itself for accurately acquiring data from the posture, orientation, movement, etc. taken on a daily basis.

  Moreover, the diagnostic system described in Patent Document 3 performs medical diagnosis based on body temperature data. However, in order to perform more accurate diagnosis by comparing data in a normal state and an abnormal state, a larger number of viewpoints are used. It was desired to acquire and analyze data from

  In view of the above-described points, an object of the present invention is to improve the accuracy of discrimination between a normal state and an abnormal state in a diagnostic system that analyzes an image obtained by imaging each part of a human body.

In order to solve the above-mentioned problem, the invention described in claim 1 is a diagnostic system comprising: a subject illumination unit that irradiates a subject with a light source of specific color light at the time of photographing; and an image photographing unit that captures the subject. An imaging device comprising the imaging device, a data management unit that is connected to the imaging device via a network, and that manages image data transmitted from the imaging device, and the image data transmitted from the imaging device is analyzed and healthy And a data management device including an image analysis unit that discriminates between a state and an abnormal state.
The invention according to claim 2 is a diagnostic system, in which a subject illumination unit that irradiates a subject with a plurality of light sources that emit light of different colors, and image capturing that captures the subject illuminated by the subject illumination unit And an image analysis unit that separates reflected light on the surface of the subject photographed by the image photographing unit and analyzes information on the reflected light suitable for the part of the subject that is a diagnosis target to discriminate between a normal state and an abnormal state And is provided.
The invention according to claim 3 is a diagnostic system comprising a plurality of light sources that emit different color lights, a subject illumination unit that irradiates the subject, a subject, and a light source of color light suitable for the subject. A system information holding unit for holding the correlation information, a control unit for controlling the subject illumination unit to operate a light source of color light suitable for the subject among the plurality of light sources based on the correlation information, and a subject An image capturing unit that captures a subject irradiated with a light source of suitable color light, and an image analysis unit that analyzes image data captured by the image capturing unit to distinguish between a normal state and an abnormal state are provided. It is characterized by that.

  The invention described in claim 4 is the diagnostic system according to any one of claims 1 to 4, wherein the image analysis unit converts the pixel value of the image data into color data. It is converted and represented on the coordinates, and when the coordinates are separated from the coordinates in the normal state by a predetermined distance or more, it is determined as abnormal data.

  According to the present invention, it becomes easier to detect lesions and abnormalities from image data by irradiation with irradiation of specific color light, and it becomes easy to separate normal state and abnormal state data. The accuracy of diagnosis can be improved.

1 is a schematic diagram showing an overall configuration of a diagnostic system according to an embodiment of the present invention. It is a block diagram showing the whole diagnostic system composition concerning an embodiment of the present invention. It is a front view which shows the image display part and subject illumination part which concern on embodiment of this invention. It is the elements on larger scale which show the example of the light source with which the to-be-illuminated part which concerns on embodiment of this invention is provided. It is the elements on larger scale which show the example of the other light source with which the to-be-illuminated part which concerns on embodiment of this invention is provided. It is a front view which shows arrangement | positioning of the color light unit which the light source of the to-be-photographed object illumination part which concerns on embodiment of this invention comprises. It is a front view which shows arrangement | positioning of the color light unit which the other light source of the to-be-illuminated part which concerns on embodiment of this invention comprises. It is a graph showing the example of the radiation spectrum of LED. It is a graph showing the spectrum example of the transmitted light by a white light source and an interference filter. It is a graph showing the other spectrum example of the transmitted light by a white light source and an interference filter. It is a graph showing the other spectrum example of the transmitted light by a white light source and an interference filter. It is a graph showing the other spectrum example of the transmitted light by a white light source and an interference filter. It is a graph showing the characteristic of the light source with which the to-be-illuminated part which concerns on embodiment of this invention is provided. It is a graph showing the spectral reflectance of the specific part (a) of the subject. It is a graph showing the spectral reflectance of the other specific part (b) of the subject. It is a graph which shows the reflected light of the specific part (a) of a to-be-photographed object. It is a graph which shows the reflected light of the specific part (b) of a to-be-photographed object. It is a figure which shows a preferable example about the composition of a to-be-photographed object. It is a figure which shows arrangement | positioning of the image imaging part which concerns on embodiment of this invention. It is an example of the synthesized image by the image imaging | photography part which concerns on embodiment of this invention. It is a figure which shows other arrangement | positioning of the image imaging part which concerns on embodiment of this invention. It is a figure which shows other arrangement | positioning of the image imaging part which concerns on embodiment of this invention. It is a figure which shows the example of the cut-out image by the image imaging part which concerns on embodiment of this invention. It is a graph of the pixel value which converted the reflected light of the specific part (a) of a to-be-photographed object. It is a graph of the pixel value which converted the reflected light of the specific part (b) of a subject. It is the figure which represented on a coordinate the pixel value of the specific part (a) of a to-be-photographed object as color data. It is the figure which represented on a coordinate the pixel value of the specific part (b) of a to-be-photographed object as color data. It is a figure which shows the example of the table with which the system information holding part which concerns on embodiment of this invention is provided. It is a flowchart which shows the effect | action of the diagnostic system which concerns on embodiment of this invention.

  Embodiments of the present invention will be described below with reference to the drawings.

  As shown in FIG. 1, the diagnostic system 1 of the present embodiment includes an imaging device 2, and a data management device 13 is connected to the imaging device 2 via a network that can communicate with each other. Furthermore, one or more external devices 24 are connected to the data management device 13 via a network that can communicate with each other. Note that the imaging device 2 can also be configured to be able to communicate with the external device 24.

  The network in the present embodiment is not particularly limited as long as it means a communication network capable of data communication. For example, the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), a telephone line network, ISDN (Integrated Services). Digital network), CATV (Cable Television), optical communication, etc. can be included. Moreover, it is good also as a structure which can communicate not only by wire but by radio | wireless.

  The external device 24 is a personal computer or the like, and is preferably installed in a place where some kind of consulting or diagnosis can be received. For example, when health-related data is obtained by the diagnostic system 1 according to the present embodiment, it may be installed in a public facility such as a hospital or a health care facility. In addition, in the case of beauty data, it may be possible to install it in cosmetic-related stores, facilities, or hospitals. The external device 24 may be configured as an Internet site from which consulting information can be obtained, or as a mobile terminal such as a consultant, a doctor, or a store clerk.

  As shown in FIG. 2, the imaging device 2 provided in the diagnostic system 1 of the present embodiment is provided with the following components.

  The external communication unit 3 is configured to perform information communication with the data management device 13 and the external device 24 by wired or wireless communication means. In addition, since the imaging device 2 according to the present embodiment handles image information, it is preferable that the imaging device 2 be in a communication mode that can transmit as fast as possible. Then, the result of image analysis, image data, vital data, and the like are transmitted to the data management device 13 via the external communication unit 3.

  The image display unit 4 includes a display such as a CRT, a liquid crystal, an organic EL, plasma, or a projection method, and in addition to a captured image, information provided from the external device 24, information obtained by image processing, or information of the imaging device 2 Information about the state of each component is displayed. In addition, it is also possible to display a warning when abnormal data is detected from the photographed image, or to notify the detection of abnormal data by voice using voice means.

  As shown in FIG. 3, the subject illumination unit 5 is provided at the peripheral edge of the image display unit 4. With such a configuration, illumination light can be given to the subject from various directions, and an image with relatively little illumination unevenness can be obtained. In addition, the subject illumination unit 5 may be arranged only at the upper part of the peripheral edge of the image display unit 4 or only at the left and right parts. Note that the subject illumination unit 5 may be arranged at a location that is not close to the image display unit 4.

  FIG. 4 is an enlarged view of the dotted line portion of the subject illumination unit 5 shown in FIG. As shown in FIG. 4, the subject illumination unit 5 of the present embodiment is arranged with circular light sources that emit different colored lights. Further, it is possible to arrange rectangular light sources as shown in FIG. Then, a combination of light sources in a predetermined area as shown by the dotted line portion in FIG. 5 is used as one color light set, and this color light set is arranged at the peripheral edge of the image display unit 2 as shown in FIG. . With such a configuration, it is possible to uniformly irradiate the subject with each color light from the light source. It is also possible to use a rod-shaped light source instead of a circular light source. In this case, a colored light set of the rod-shaped light source can be arranged at the peripheral edge of the image display unit 2 as shown in FIG.

  Artificial light sources such as fluorescent lamps, incandescent light bulbs, LEDs, lasers, and organic ELs can be used as the light source provided in the subject illumination unit 5, but in order to suppress the influence of ambient ambient light on the photographed image. It is desirable to use a light source such as a strong strobe light or a high brightness LED. Further, it is desirable that the light source has a light emission characteristic not only in the visible light region but also in the vicinity thereof (ultraviolet region, near infrared region, infrared region). In order to suppress the influence of ambient light as much as possible, it is desirable to use a light source that can emit high-brightness light such as a strobe. However, when shooting moving images of subjects, characteristics such as intensity and spectrum are stable for a long time. It is desirable that the light source be capable of irradiating light. Further, when the line / surface light source is used instead of the point light source, light with a spatially reduced intensity distribution can be obtained for the user. That is, uneven illumination to the user is suppressed and a desirable image is obtained.

  As the light source of this embodiment, an LED is used, or a white light source is used together with an interference filter. When a white light source is used, one or more interference filters are arranged in front of the white light source to narrow the characteristics of the illumination light. Here, FIG. 8 shows the light emission characteristics of the LED, and FIGS. 9 to 12 show examples of transmitted light spectra of the white light source and the interference filter. FIG. 13 shows the characteristics of the light source used in this embodiment, and two types of light sources (colored light H and colored light I) are used among the light sources having the characteristics shown in FIGS. For example, Toshiba's TLSH series (peak wavelength 623 nm, half width 13 nm) is used as color light H, TLPGE1100B (peak wavelength 562 nm, half width 11 nm), and other commercially available LEDs are used as color light I. Is possible. As the interference filter, a commercially available filter such as a bandpass filter manufactured by Omega Optical can be used.

  FIG. 14 and FIG. 15 show the spectral reflectances at the time of health (normal) and unhealthy (abnormal) in the specific part (a) and another specific part (b) of the subject. In general, when the subject is unhealthy, the structure and structure of the body changes, so the spectral reflectance on the subject surface is different from the normal state. However, which wavelength region is different depends on the subject. In the example of the specific part (a) shown in FIG. 14, the difference between the low wavelength side (overlapping the band of the colored light A) is large between the normal state and the abnormal state, and in the example of the specific part (b) shown in FIG. The difference between the areas (which overlap the band of the colored light B) is large. When the color light A and the color light B are irradiated to the specific part (a) of the subject, the spectrum of the reflected light of the subject is a product of the color light A or the color light B and the spectral reflectance of the subject as shown in FIG. Yes. Similarly, when the color light A and the color light B are irradiated to another specific part (b) of the subject, the spectrum of the reflected light of the subject is the product of the color light A or the color light B and the spectral reflectance of the subject as shown in FIG. It has become.

Returning to FIG. 2, the image capturing unit 6 includes one or a plurality of cameras that can acquire a still image or a moving image by an image sensor such as a CCD or a CMOS. For example, you may comprise with a camera module attached to a digital camera, a video camera, other mobile phones, etc.
Here, as shown in FIG. 18, it is desirable that an image of a subject be taken with a stable composition regardless of the standing position and height of the subject. Therefore, in the present embodiment, as shown in FIG. 19, a plurality of image photographing units 6 are arranged at equal intervals to photograph a subject, and an image having a composition that may be synthesized as shown in FIG. 20 is output. It is configured. Further, as shown in FIG. 21, a mechanism capable of three-dimensionally adjusting the position of one image capturing unit 6 may be provided so that the user's head and face can be captured. Further, as shown in FIG. 22, the image photographing unit 6 is configured by a combination of a large number of camera modules, and the user can be collectively photographed to cut out an image having a predetermined composition as shown in FIG. May be.

  Furthermore, when it is desirable to obtain an image whose composition changes in conjunction with the movement of the subject, rather than an image whose composition is fixed as described above, for example, when the subject comes up when the subject approaches the display screen, Images captured by one or more cameras can be combined and displayed as they are with the origin of display coordinates fixed.

  Further, the image capturing unit 6 of the present embodiment has three image channel numbers, but may be a multi-channel (multi-band) having three or more image channels. As a format of the photographed image, a general format such as jpeg, tiff and a raw file corresponding to the raw image are provided. Image processing (such as demosaic processing, gamma processing, color conversion processing, and enhancement processing) for a raw file may or may not be performed at the time of file output. However, each parameter of the image processing can be referred to. The image capturing unit 6 may be disposed on the back surface or the same surface of the image display unit 4, but is disposed on the back surface of the image display unit 4 in the present embodiment. When the image display unit 4 is arranged on the back surface, it is desirable that the image display unit 4 be configured with a half mirror structure / material that does not interfere with image capturing and the user does not know the presence of the camera.

  The image photographing unit 6 captures reflected light on the surface of the subject and converts it into pixel values of each channel corresponding to R, G, B signals of the RGB image. As described above, the spectral reflectances in the normal state and the abnormal state in the specific part (a) of the subject are as shown in FIG. 16, and this reflected light is imaged by the image capturing unit 6 and channel 1 and channel 2 are captured. When the reflected light caused by the color light A or the color light B is separated, the pixel value of each channel can be obtained as shown in FIG. Here, the separation of the reflected light caused by the colored light A or the colored light B is performed by the difference in the band of each channel. For example, the color light A is included in the channel 1 band of RGB, and the color light B is included in the channel 2 band. Further, it is also possible to separate the color light A and the color light B by photographing with different timings of light emission. Similarly, the spectral reflectance at the other specific part (b) of the subject is as shown in FIG. 17, but when the reflected light caused by the colored light A or the colored light B is separated, a pixel value is obtained as shown in FIG. be able to.

Thus, by separating the reflected light caused by the color light A or the color light B, the captured image of the specific part (a) of the subject is in the channel 1 and the captured image of the specific part (b) is in the channel 2 in the normal state. In addition, separation of pixel values in an abnormal state is facilitated.
The subject illumination unit 5 performs the function of illuminating the subject when photographing an image of the face area as described above, and also changes the intensity and wavelength of illumination rapidly when photographing the movement of the pupil of the subject. It is also possible to perform a function that makes it possible to determine whether the eye is healthy or abnormal by measuring the movement of the pupil before and after the irradiation.

  Returning to FIG. 2, the surrounding environment information acquisition unit 7 includes a measuring instrument as an environment information acquisition unit, acquires information on the surrounding environment where the imaging device 2 is installed, and assists in determining abnormal data when analyzing the captured image. Can be done. Here, the surrounding environment refers to the intensity and characteristics of ambient illumination light, temperature such as air temperature and water temperature, humidity, and atmospheric pressure. In addition, there are (a) illuminance meter type, (b) spectrum measurement type, (c) color temperature measurement type, and the like as the method for acquiring the environmental illumination light, but the type (b) that can include all is desirable. As for other temperatures, humidity, and atmospheric pressure, measuring instruments and sensors for measuring each temperature may be used, and information may be obtained from the external device 24 through a network.

  The I / O unit 8 includes a vital sensor (such as a thermometer, a weight scale, a body fat percentage meter, a blood pressure meter, an electrocardiograph, a skin age meter, a bone densitometer, a spirometer) as a vital information acquisition means, a CF card In addition, devices that can handle portable devices such as SD cards and USB memory cards can be connected, and measurement data and image data can be input and output from these devices. Although the vital sensor can be configured as a part of the imaging device 2, the present embodiment can be configured so that introduction of arbitrary biological information data can be taken into account and state-of-the-art measurement equipment can be easily introduced. Thus, it is desirable that a necessary vital sensor is connected to the I / O unit 8 for use.

  In addition, a vital sensor connected to the I / O unit 8 can be used as a stimulus adding means to capture temporal changes before and after the stimulus is applied to the subject, and distinguish the normal state and the abnormal state of the subject from the image. Here, the stimulus refers to giving light, pushing, applying pressure, changing temperature, adding scent, applying sound, stabting with a needle, painting an object, injecting an object, and the like. For example, by compressing a part of the skin with a pressure sensor or the like and measuring the state before and after the compression, the normal state and the abnormal state can be distinguished using the amount of change of each measured value as feature data. In addition, by warming the skin with water, hot water, or far-infrared rays, and observing the state before and after that, the change in blood flow is measured to determine whether the change is different from normal. You can also.

  The memory unit 9 includes a RAM, a ROM, a DIMM, and the like, and temporarily stores data necessary for processing in each component of the imaging device 2, thereby causing the imaging device 2 to operate stably at high speed. Yes.

  The data processing unit 10 is an image of each part of the subject captured by the image capturing unit 6, peripheral environment information acquired by the peripheral environment information acquiring unit 7, data such as measurement data and image data input from the I / O unit 8. Processing is to be performed.

  That is, first, the data processing unit 10 extracts a region of interest to be diagnosed from a captured image. For example, when the region of interest is the user's face, for example, a standard user face image registered in advance is used as the basic image of the template, and the template is subjected to a raster operation from the upper left of the screen, and the correlation between the template and the image is uniform. The highest part is set as the face area. In addition, when further specifying a region from the region of interest, the user can specify a means by surrounding the face frame with a rectangular pointer or the like, or when the user specifies a point, a rectangle of a predetermined size centered on the specified position is displayed. It is also possible to provide a designation means. When the region of interest is extracted, the data processing unit 10 transmits an image of the region of interest to the data management device 13 via the external communication unit 3.

  If the image capturing unit 6 does not perform image processing in the case of a raw file at the time of file output, the data processing unit 10 may be configured to perform image processing.

  Further, in order to display an image with stable color reproduction for the purpose of viewing even if the environmental light characteristics change, the data processing unit 10 acquires the environmental light characteristics by the surrounding environment information acquisition unit 7 and converts the color data into the color data. The display white color point and brightness are adjusted accordingly. In other words, these characteristics are corrected (calibrated) in order to display an image with stable color reproduction regardless of the characteristics of ambient ambient light and light sources. Usually, in calibration, a test pattern is photographed and corrected so that the photographed data has a separately designated set value. However, it is desirable that the same processing can be performed without using a test pattern that is relatively expensive and difficult to store. Therefore, it is possible to use the user's teeth and background data at the time of image capturing, which seem to hardly change over time. At that time, if a plurality of users are registered in the same device, it is necessary to register tooth data for each user.

  In addition, when the subject is photographed with the same composition in the image photographing unit 6, the data processing unit 10 determines the brightness and size of the subject on the screen depending on how the illumination is applied when the subject moves in the optical axis direction of the camera. A function is provided to correct this so as not to change. In this case, the distance between the imaging device 2 and the user is estimated from the image captured by the image capturing unit 6. As this method, general methods such as a three-dimensional survey method and a stereo measurement method can be used. Alternatively, laser photometry equipment often used in a camera may be attached to the imaging device 2 for measurement. Moreover, it is possible to estimate the three-dimensional shape of the user by performing distance estimation based on the captured image at each pixel. Further, if the distance between the subject and the camera in each pixel is known, the distance between the light source and the subject can be obtained for each pixel, so that the brightness of the display image can be corrected more accurately and in detail.

  Further, the data processing unit 10 is adapted to specify an individual from a captured image for user authentication. That is, the personal face area image captured at the time of initial registration by the image capturing unit 6, the measurement value of the three-dimensional shape of the face, and other characteristic information (information such as the use of glasses, moles, and spots) are automatically generated. Alternatively, it is manually acquired and registered in the memory unit 9, and in subsequent use, an image photographed by the image photographing unit 6 is analyzed and automatically obtained from personal data registered in the memory unit 9. It is configured so that individuals can be identified by collation. This specification is performed, for example, by comparing the correlation between the extracted template image and the basic template image tailored to the individual after extracting the facial part.

  The user interface unit 11 includes a keyboard, a mouse, a trackball, and the like. The user interface unit 11 allows the user to input instructions and can transmit the status and requests of the imaging device 2 to the user. Although it is possible to use a conventional interface such as a keyboard, a mouse, a trackball, etc., it is desirable to have a device configuration with less burden on the user, so that the interface is configured as a touch panel integrally with the image display unit 4. Can do. In addition, it is desirable to have a configuration that can communicate with a user's voice, gesture, and gesture (including advanced communication means such as sign language) by providing acoustic equipment such as a speaker and a microphone.

  In this user interface unit 11, for example, when photographing a subject, if the region of interest for diagnosis based on the photographed image is the tongue, the user is instructed by voice or characters to put out the tongue, and the tongue comes out from the movement of the mouth. The image photographing unit 6 can also be photographed by guessing that. Further, the display screen in the image display unit can be switched by voice, gesture, or gesture through the user interface unit 11.

  In addition, when the data processing unit 10 extracts a region of interest to be diagnosed from the photographed image and specifies a region from the region of interest, the user surrounds the face frame with a rectangular pointer or the like in the interface unit 10. It is possible to provide means for designating or designating means for displaying a rectangle of a predetermined size centering on the designated position when the user designates a point.

  Further, the user interface unit 11 is provided with data acquisition means for the purpose of assisting in determining abnormal data when analyzing a captured image. For example, it is possible to construct a system in which an inquiry can be made for each shooting in the user interface unit 11. Specifically, it is possible to simply press a button for distinguishing between health and malfunction. The data contents to be acquired include self-judgment of the current physical condition (heavy, fever, cold, etc.) and personal data related to the condition (sleeping time, drinking, smoking, medication information, etc.) Is included. It may be acquired interactively by voice, or may be acquired through a user interface such as a touch panel by displaying an inquiry form on the image output device. In addition, when acquiring by voice, the tone of the voice is often related to the physical condition at that time, so the analysis result may be added to the inquiry data by analyzing the frequency and tone of the voice.

  The user interface unit 11 is provided with personal authentication means. That is, when a plurality of users use the diagnostic system 1, it is necessary to specify an individual who is currently using it. Therefore, it is possible to confirm to the person whether or not there is an error in the identification result based on the individual identification result obtained by collation with the personal face area image or the like photographed at the time of initial registration in the data processing unit 10.

  As a confirmation method, there are methods of (a) asking “Is you Mr. XX?” By voice, and (b) displaying “You are Mr. XX?” On the image display unit 4. The answering methods are (c) answering with voice, (d) gesturing, answering with gestures (nodding, shaking head vertically (positive), shaking head sideways (denial), etc.), etc. is there. This makes it a device that is friendly to people with poor eyes, people with poor vision, people with poor ears, and the like.

  If the specific result is correct, the normal process is continued. If the specific result is incorrect, (e) the next highest collation result in the data processing unit 10 is specified and asked in the same manner. (F) Prompt the user to input a name (in the case of voice, ask "What is your name?", And the user answers the name registered in the diagnostic system 1. If displayed, "What is your name?" At the same time, the registrant list is displayed, and the user can select one of them). As another means for identifying an individual, there is a method of displaying the name of the registrant on a menu and selecting and instructing the user himself / herself.

  In addition, the user interface unit 11 is provided with means for prompting the user to use the diagnostic system 1 by outputting sound at regular intervals. The means for prompting the user to use may be a display on the image display unit 4. That is, in the diagnostic system 1 of the present embodiment, since it is necessary to check and update the user's healthy state data at regular intervals, by prompting the user to use a vital sensor at regular intervals, for example, every month It is possible to update data in a normal state. Further, depending on the purpose of diagnosis, it may be necessary to use the diagnostic system 1 on a daily or regular basis. Therefore, when the situation where the diagnostic system 1 is not used due to the absence of the user or the like, or when the imaging by the image capturing unit 2 is not performed regularly, the user interface unit 11 displays an alert or emits sound. It is configured to report to the outside. However, since it may be due to a schedule such as a long-term business trip or a trip, an alert may be issued in consideration of an individual schedule input from the data management device 13 via the external communication unit 3. it can.

  The control unit 12 includes a CPU and a RAM, and drives and controls each component of the imaging device 2. Since the imaging apparatus 2 of the present embodiment also handles moving images, it is desirable that the control unit 12 be configured with a chip that can operate and control as fast as possible. The control unit 12 also turns On / Off of each light source in the subject illumination unit 5 based on an instruction signal transmitted from the data management apparatus via the external communication unit 3 or an instruction signal input from the user interface unit 11. Are controlled independently. In addition to performing control based on the instruction signal, the subject can be estimated by other known methods.

  Next, as shown in FIG. 2, the following components are provided in the data management device 13 included in the diagnostic system 1 of the present embodiment.

  The external communication unit 14 is configured to perform information communication with the imaging device 2 and the external device 24 by wired or wireless communication means. Similarly to the external communication unit 3, the result of image analysis, image data, vital data, and the like are transmitted to the data management device 13 or the external device 24 via the external communication unit 3. Here, the external device 24 is preferably a place where some kind of consulting or diagnosis can be received, such as a hospital, health care facility, cosmetic-related store, facility, or hospital.

  The image analysis unit 15 includes image analysis means, performs image analysis based on image data transmitted from the imaging device 2 to the data management device via the external communication unit 3 and the external communication unit 14, and based on the analysis result Diagnosis such as medical diagnosis.

  Here, in the present embodiment, as described above, focusing on the fact that the lesion or abnormality of the object such as human skin appears in the narrow band characteristics of the subject's spectral spectrum, the subject illumination unit 5 affects the influence of such a band. It is easy to detect lesions and abnormalities from images by shooting with illumination that can be enhanced. That is, when the reflected light on the surface of the subject is imaged by the image capturing unit 2 of the imaging device 2 and converted into pixel values for each channel, the specific part (a) of the subject is healthy in channel 1 as shown in FIG. Separation of the pixel values of the state and the abnormal state is facilitated, and as shown in FIG. 25, regarding the other specific part (b) of the subject, the pixel values of the normal state and the abnormal state can be easily separated in the channel 2. It has become.

And the image analysis means with which the image analysis part 15 is provided isolate | separates the pixel value of a healthy state and an abnormal state for every specific part of a to-be-photographed object based on the image data of a region of interest. Thereafter, tristimulus value data (RGB) is calculated from the respective pixel values in the normal state and the abnormal state, reaction color data is calculated from the tristimulus value data by primary conversion, and uniform color space data is calculated from the reaction color data. It is supposed to convert to. As an example of the uniform color space, there are CIE L ^* a ^* b ^* , L ^* u ^* v ^* , u′v ′ in which the lightness direction is standardized, and their derived spaces are also included. Then, with this uniform color space data, as shown in FIG. 26 or FIG. 27, the color data in the normal state / abnormal state is expressed as coordinates of two or more dimensions (two dimensions in this embodiment), and obtained as a result of analyzing the image. If the coordinates are separated by a certain scale compared with the coordinates in the normal state, it is determined as abnormal data. Here, FIG. 26 shows the coordinates of the color data in the specific part (a), and FIG. 27 shows the coordinates of the color data in the other specific part (b). In FIG. 26 or 27, “feature amount 1” is a value such as CIE a ^* , u ^* , or u ′, and “feature amount 2” is, for example, CIE b ^* , v ^* , or v ′ ^. Value. Note that other color data or measurement values other than the color data (three-dimensional shape, texture, etc.) may be used. Instead of separately irradiating the color light A and the color light B and then separating them, the image analysis may be performed after the color light A and the color light B are separately irradiated and an image is taken.

  As a judgment of a healthy state and an abnormal state, for example, it is known that irradiation with a narrow band of color light centering around 620 nm emphasizes the involvement of hemoglobin in the blood, and blood flow is interrupted It seems to be effective for detecting such abnormalities. In this case, by using the color light H of FIG. 8 or FIG. 11, the accuracy of discrimination between a normal state and an abnormal state by image analysis is improved.

  There is also a report that capillary blood vessels can be enhanced by irradiating a narrow band of color light centered around 550 nm. It seems to be effective in detecting abnormalities in the part where capillaries such as fingertips gather. In this case, it can be said that the use of the color light I of FIG. 8 or FIG. 10 improves the accuracy of discrimination between a normal state and an abnormal state by image analysis.

  In addition, the color image, three-dimensional shape, and texture of the subject are measured (estimated) by analyzing the photographed image of the subject photographed in time series by the image photographing unit 6, that is, continuously photographed with a still image or photographed with a moving image. The amount of hemoglobin, the amount of melanin, the degree of oxygen saturation, etc. related to the tissue structure under the skin can also be estimated from the estimated data. Further, data such as the number of blinks, the color of the eyes (particularly white eyes), the state of the eyes, and the like may be acquired from the eye image.

  The image analysis unit 15 ends the diagnosis process when it is diagnosed as healthy by image analysis. On the other hand, when unhealthy is diagnosed by image analysis, the result of the image analysis is transmitted to the external device 24 via the external communication unit 14. The destination is preferably a place where some kind of consulting or diagnosis can be received as described above. For example, public facilities such as hospitals and health care facilities, cosmetic-related shops and facilities / hospitals, Internet sites, and portable terminals are considered.

  Further, the image analysis means provided in the image analysis unit 15 changes the portion of data that can be transmitted according to the user's permission when sending image data to a consultant or the like through the external communication unit 14. For example, when the user desires to remove or change the image around the eyes or mouth to transmit the individual in order to avoid the identification of the individual, the corresponding part in the image is automatically extracted and the part is changed to an arbitrary color. Change or add mosaic or blur.

  In addition, in order to safely conceal personal information, transmitted data and data inside the data management device 13 are encrypted by changing the encryption key for each individual. As a method for producing the key, known methods can be used, but biometrics that enable key setting for each individual are desirable. In this case, the positional relationship of the constituent elements in the face such as the eyes, nose and mouth of the individual, the height of the nose, the three-dimensional shape such as the shape of the entire face, iris information, etc. are used as the encryption key.

Note that the image analysis unit 15 of the data management device 13 analyzes the captured image for personal authentication of the user in the data processing unit 10 of the imaging device 2 described above, and the identification result is transmitted from the external communication unit 14 to the imaging device 2. You may make it transmit.
The I / O unit 16 is configured to connect a portable device such as a CF card, an SD card, or a USB memory card. A measurement device such as a vital sensor may be connected to the I / O unit 16 for use.

  The user interface unit 17 is configured to allow the user to input instructions and to transmit the status and request of the apparatus to the user. As with the user interface unit 11 of the imaging device 2, a configuration that does not place a burden on the user is desirable, but since complicated processing by the device administrator such as maintenance and backup may be necessary, use only a keyboard and a mouse. You can also. However, the user interface unit 17 is not an essential component in the data management device 13, and the user interface unit 11 of the imaging device 2 can also function as the user interface unit 17 of the data management device 13. In this case, data input / output through the user interface unit 11 of the imaging device 2 is transferred to the data management device 13 via the external communication unit 3 and the external communication unit 14.

The display unit 18 includes a display such as a liquid crystal display, and displays the operation status of the data management device 13 and instruction input by the user interface unit 18. Also, a warning when abnormal data is detected from the captured image is displayed. The warning when abnormal data is detected can be configured to be notified by voice means. However, the display unit 18 is not an essential component in the data management device 13, and the image display unit 4 of the imaging device 2 can also be used. In this case, the displayed content is transferred to the image input / output device via the external communication unit 14 and the external communication unit 3.
The storage unit 19 includes a memory such as a buffer, and includes a data holding unit 20, a work area unit 21, and a system information holding unit.

  The data holding unit 20 includes various types of data such as an image of an area of interest to be diagnosed, measurement, biological information, device status, and environmental status transmitted from the imaging device 2, consulting / diagnosis data input from the external device 24, etc. Is supposed to hold. Here, the data held in the data holding unit 20 is image data and various measurement data. In particular, since the image data has a large data size, it is necessary to devise data storage. Therefore, the data holding unit 20 uses the memory effectively by compressing and storing the data determined as “healthy” by the daily health check.

  The work area unit 21 temporarily holds various data used for data processing in order to perform data processing in the data management device 13 at high speed.

The system information holding unit 22 holds system information of the imaging device 2 and the data management device 13.
First, the system information holding unit 22 tabulates a management number of an LED or a filter as a light source included in the subject illumination unit 5 of the imaging apparatus 2, an effective subject, and a feature quantity that can be extracted as shown in FIG. Register and manage. The control unit 12 of the imaging device 2 can control the On / Off of each light source provided in the subject illumination unit 5 independently by this table to give color light suitable for the subject.
The system information holding unit 22 manages the light source and filter spectral characteristics, color data, temporal characteristics, and the like included in the subject illumination unit 5 of the imaging device 2, and the image analysis unit 15 is necessary for image analysis. In such cases, these data can be referred to.

  In addition, the system information holding unit 22 creates “health state data” that serves as a reference for diagnosis by image analysis in the image analysis unit 15 using measurement values measured from image data determined to be “healthy”. It comes to hold. If normal state data is created, the abnormal state data can be expressed as a complementary set of normal data. The normal state data is updated after confirming the normal state of the user at regular intervals in order to increase the reliability of the data. That is, in the image display unit 4 or the user interface unit 17, for example, the health state data is updated by prompting the user to use a vital sensor or the like every month. It can also be updated daily based on recent month data. Thereby, it is thought that it can respond also to the change of the physical condition by a season or aging. Moreover, you may produce with reference to predetermined days, such as 30 days, except the day containing abnormal data.

  In addition, when the diagnostic system 1 is introduced or just after a new user is registered, there is little information about the user's individual skin color and other characteristics, and data on a normal state or an abnormal state cannot be configured. . Therefore, average data of ordinary persons is registered in the system information holding unit 22 in advance, and a temporary measurement value is obtained using the average data for a predetermined period from the first time. Moreover, a temporary measured value can also be obtained using the average data of the user who uses the diagnostic system 1. Then, after the lapse of a predetermined period, the measurement value is updated with respect to the image data up to that time by switching to the actual data of the user. The “predetermined period” is registered in advance in the system information holding unit 22.

  The system information holding unit 22 holds a personal schedule transferred from a device connected to the I / O unit 8 of the imaging device 2 or the I / O unit 16 of the data management device 13. Based on the personal schedule held by the system information holding unit 22, the user interface unit 11 generates an alert that prompts the user to use the diagnostic system 1.

  Furthermore, the system information holding unit 22 holds the characteristics of each component of the imaging device 2 or the data management device 13. That is, the characteristics of the subject illumination unit 5, the image display unit 4, the image photographing unit 6 and the like of the imaging device 2 vary with time, and usually deteriorate with time. Therefore, the timing for correcting (adjusting) the change with time is set in the system information holding unit 22, and the time information and the like are read from the tag information, so that the characteristics of the imaging device 2 or the data management device are measured / according to the designated timing. It is to be corrected. This makes it possible to obtain image measurement values that are stable over time.

  In addition to calibration that is performed periodically, when the surrounding situation changes, the user instructs the calibration timing or automatically detects the change in the surrounding situation from the image and performs calibration. It has become. Here, “the surrounding situation has changed” means, for example, that the surrounding environment lighting has changed significantly (e.g., the fluorescent length has been changed, a new window has been installed nearby), or the surrounding has changed (the background of the image Change the room, change the position of the device, etc.). The difference in the peripheral situation can be determined by comparing the past data stored in the system information holding unit 22 with the current data and determining whether or not to perform calibration according to the degree of the difference.

  Furthermore, the light source included in the subject illumination unit 5 and the camera or sensor included in the image capturing unit 6 are also considered consumables, and may need to be replaced at regular intervals in order to maintain a stable device. Therefore, information such as the light source and the camera (replacement date, etc.) is added to the tag information of the image data or the system information holding unit 22, and when these devices have been used for a certain period of time or have been replaced due to damage In such a case, it may be considered that the user interface unit 11 issues an alert.

  The data management unit 22 manages each individual image data held in the data holding unit 20 of the storage unit 19 by attaching a tag (accompanying information). The tagging method may be attached to the header or footer of each data, or may be collected in a separate file in association with each data name.

Here, the content of the tag includes the image shooting date and time, the user name and its attributes, information on the imaging device 2 and the data management device 13, information on the surrounding environment, the captured image, the I / O unit 8 or the I / O unit 16 Measured data values acquired from a vital sensor connected to, diagnostic information when a captured image is subjectively diagnosed, contents of inquiry data acquired separately, and the like are included. In particular, a tag that can grasp time information such as photographing date and time for tracking image data in time series, and other data contexts is essential.
Further, information required for the imaging device 2 and the data management device 13 is information on the subject illumination unit 5 related to estimation of measurement data from images, characteristics of the image capturing unit 6, characteristics of the imaging device 2, and data processing unit. 10 is data processing method information. Such hardware and software information is generally managed by version, and this embodiment follows that idea. Therefore, each version number is described in the tag.

  In addition, software for image analysis may be upgraded to improve functions (including improved accuracy of measurement values). When the version is upgraded, the measured value is recalculated and updated to the new measured value by using the tag contents in the image data and the new software. Therefore, the tag content is newly updated.

  Further, when the held image data is held in a portable memory and carried, there may be a case where a memory capacity limit is taken into consideration. Therefore, if the image data and the tag data are easily separated, the image data cannot be stored in the memory when carried. A method of managing tag data in a separate file is desirable. Also, by doing so, it becomes easy to add special processing only to the tag data.

  In addition, each image data is divided into directories in units (one role) for each individual user. This makes it easier to manage data for each individual.

  In addition, as a place where the imaging device 2 or the data management device 13 included in the diagnosis system 1 of the present embodiment is installed, a place near a water area such as a washroom or a bathroom can be considered. Therefore, it is desirable to take waterproof measures such as anti-fogging processing and coating so as not to be fogged with moisture. It is also desirable to take measures against dirt and dust such as an antifouling seal, considering installation in a place where dust tends to accumulate.

  Next, the operation of the above-described diagnostic system 1 will be described with reference to the flowchart of FIG.

  When the user starts using the diagnostic system 1, first, the image photographing unit 6 photographs a subject.

  The data processing unit 10 includes an image of a subject captured by the image capturing unit 6, peripheral environment information acquired by the peripheral environment information acquiring unit 7, measurement data and image data input from the I / O unit 8, or the user interface unit 11. The input information such as the user instruction input data that has been input is analyzed, the image is displayed on the image display unit 4, and the light source of the subject illumination unit 5 is adjusted. Further, the face area image of the individual photographed at the time of initial registration by the image photographing unit 6 and the image acquired by the image photographing unit 6 are analyzed, and automatically collated from the registered personal data to identify the individual ( Step S1).

  Next, the user interface unit 11 confirms the individual identification result by the data processing unit 10 by the personal authentication means (step S2). As a result, if the identification of the individual is incorrect, the data processing unit 10 identifies the individual again (steps S3 and S1).

  On the other hand, if the identification of the individual is correct, the user interface unit 11 makes an inquiry about the health condition (steps S3 and S4). Through this inquiry, self-judgment of the current physical condition (heavy body, fever, coldness, etc.) and personal data related to the condition (sleeping time, drinking, smoking, medication information, etc.) can be obtained.

  Next, the image capturing unit 6 captures an image of a part to be diagnosed in the subject. At this time, the subject irradiating unit 5 irradiates a plurality of light sources and performs photographing (step S5). That is, an LED is used as a light source or a white light source is used together with an interference filter. However, the subject illumination unit 5 of the present embodiment uses two types of light sources (color light) among the light sources having the characteristics shown in FIGS. H and color light I) are used. Thereby, in the example of the specific part (a) shown in FIG. 14, the difference in the low wavelength side (overlapping with the band of the colored light A) is large between the normal state and the abnormal state, and in the example of the specific part (b) shown in FIG. The difference between the middle range and the high range (which overlaps the band of the colored light B) becomes large. Then, when the color light A and the color light B are irradiated to the specific part (a) of the subject, the spectrum of the reflected light of the subject becomes the product of the color light A or the color light B and the spectral reflectance of the subject as shown in FIG. Similarly, when the color light A and the color light B are irradiated to another specific part (b) of the subject, the spectrum of the reflected light of the subject is the product of the color light A or the color light B and the spectral reflectance of the subject as shown in FIG. It becomes.

  Then, the image photographing unit 6 captures the reflected light on the subject surface and converts it into a pixel value. At that time, the reflected light caused by each color light is separated by the channel, thereby making it easy to separate the pixel values in the normal state and the abnormal state for each specific part of the subject. Further, the data processing unit 10 extracts a region of interest from the captured image (step S6).

  The captured image of the region of interest is transmitted from the imaging device 2 to the data management device 13 via the external communication unit 3 and the external communication unit 14 (step S7).

  The captured image of the region of interest transmitted to the data management device 13 is held by the data holding unit 20 of the storage unit 19. Then, the data management unit 22 manages each individual image data held in the data holding unit 20 by attaching a tag (accompanying information).

  Next, the image analysis unit 15 separates the pixel values of the normal state and the abnormal state for each specific part of the subject based on the image data of the region of interest held by the data holding unit 20 of the storage unit 19 (step S8). Then, tristimulus value data (RGB) is calculated from the respective pixel values in the normal state and the abnormal state, and reaction color data is calculated from the tristimulus value data by primary conversion, and the reaction color data is converted into uniform color space data. Conversion is performed (step S9). Then, with this uniform color space data, as shown in FIG. 26, the color data of the normal state / abnormal state is represented as coordinates of two or more dimensions (in this embodiment, two-dimensional), and the coordinates of the normal state and the unhealthy state are compared. (Step S10). As a result, when the obtained coordinates are separated by a certain scale compared with the coordinates in the normal state, it is determined as abnormal data (step S11). The healthy state data is held in the system information holding unit 22 of the storage unit 19.

  As a judgment of this normal state and abnormal state, for example, there is a report that irradiation with a narrow band of color light centering around 620 nm emphasizes the involvement of hemoglobin in the blood image, and blood flow is disrupted It seems to be effective for detecting abnormalities. In this case, the color light H in FIG. 8 or FIG. 11 may be used.

  There is also a report that capillary blood vessels can be enhanced by irradiating a narrow band of color light centered around 550 nm. It seems to be effective in detecting abnormalities in the part where capillaries such as fingertips gather. In this case, the color light B shown in FIG. 8 or 10 is preferably used.

  In addition, the color data, three-dimensional shape, and texture of the subject are measured (estimated) by analyzing the photographed image of the subject photographed in time series by the image photographing unit 6 (continuous still images and moving images). The amount of hemoglobin, the amount of melanin, the degree of oxygen saturation, etc. related to the tissue structure under the skin can also be estimated from the estimated data. Further, data such as the number of blinks, the color of the eyes (particularly white eyes), the state of the eyes, and the like may be acquired from the eye image.

  When the image analysis unit 15 diagnoses health through image analysis, the diagnosis process is terminated (step S12). On the other hand, when the image analysis unit 15 diagnoses unhealthy by image analysis (step S13), the diagnosis result is displayed on the display unit 18 or the image display unit 4 (step S14), and the result of the image analysis is displayed on the external communication unit. 14 to the external device 24, that is, to an external consultant or medical institution (step S15). The destination is preferably a place where some kind of consulting or diagnosis can be received as described above. For example, public facilities such as hospitals and health care facilities, cosmetic-related shops and facilities / hospitals, Internet sites, and portable terminals are considered.

  The image analysis means provided in the image analysis unit 15 changes the portion of data that can be transmitted by mosaic or blurring when sending image data to a consultant or the like through the external communication unit 3 or the external communication unit 14 To do. In addition, in order to safely conceal personal information, transmitted data and data in the data management device 13 are encrypted by changing the encryption key for each individual.

  When a diagnostic system is introduced or when a new user has just been registered, a measurement value is obtained using average data of general persons registered in advance. Moreover, a temporary measured value can also be obtained using the average value of the data of the user who uses the diagnostic system 1. In this case, after the lapse of a predetermined period, the actual data of the user is switched and the measurement value is updated with respect to the temporary image data up to that time.

  In addition, an alert that prompts the user to use the diagnostic system 1 is displayed on the image display unit 4, the user interface unit 11, or the display unit 18 every predetermined period. Thereby, data is updated for every predetermined period, and the reliability of a diagnostic result is ensured.

  In addition, the system information holding unit 22 sets timings for correcting (adjusting) changes in characteristics of the devices constituting the diagnostic system 1 in order to obtain image measurement values that are stable over time, and sets each timing according to the designated timing. Measure and correct device characteristics. Further, when the surrounding situation changes, the user instructs the calibration timing or automatically detects the change in the surrounding situation from the image and performs calibration. Furthermore, when the consumables have been used for a certain period of time or when replacement is necessary due to damage, the user interface unit 11 issues an alert to prompt replacement.

  As described above, according to the diagnosis system 1 of the present embodiment, the lesion or abnormality of the subject appears in the narrow band characteristics of the spectral spectrum of the subject surface, but the subject is irradiated with the specific color light that can emphasize the influence of such a band. Since imaging is performed, it becomes easy to detect lesions and abnormalities from image data.

  In addition, it is subject-dependent whether the spectral reflectance of the subject surface in the normal state and the abnormal state differs significantly in which wavelength region, but the reflected light caused by each color light corresponding to this wavelength region is separated by channel, so it is specific It is easy to separate normal and abnormal pixel values in a specific channel for a subject.

  Furthermore, it is possible to easily discriminate visually the data of the normal state and the abnormal state on the coordinates.

  As described above in detail, according to the diagnostic system of the present invention, it becomes easy to detect lesions and abnormalities from image data by irradiation with a specific color light, and it is easy to separate the data of the normal state and the abnormal state Therefore, it is possible to improve the accuracy of diagnosis by distinguishing between a normal state and an abnormal state.

  In addition, it is easy to separate the pixel values of the normal state and the abnormal state in a specific channel for a specific subject.

  Furthermore, it is possible to easily discriminate visually the data of the normal state and the abnormal state on the coordinates.

  The “diagnostic system” in the present application may be in any form as long as it has a system configuration. Moreover, it can be said that even if what comprised each component contained in the diagnostic system as one apparatus, or comprised by several apparatus, it is in the category of this invention.

Claims (4)

An imaging device comprising: a subject illumination unit that irradiates a subject with a light source of a specific color light at the time of shooting; and an image shooting unit that images the subject;
A data management unit that is communicably connected to the imaging device via a network and that manages image data transmitted from the imaging device, and that a normal state and an abnormal state are analyzed by analyzing the image data transmitted from the imaging device. And a data management apparatus including an image analysis unit for determining the diagnosis system. A subject illumination unit that illuminates the subject with a plurality of light sources that emit different colored light; and
An image capturing unit that captures an image of the subject irradiated by the subject illumination unit;
An image analysis unit that separates reflected light on the surface of the subject imaged by the image capturing unit and analyzes information on the reflected light suitable for the part of the subject to be diagnosed to determine a normal state and an abnormal state;
A diagnostic system characterized in that is provided. A subject illumination unit that has a plurality of light sources that emit light of different colors and illuminates the subject;
A system information holding unit for holding correlation information between a subject and a light source of color light suitable for the subject;
Based on the correlation information, a control unit that controls the subject illumination unit to operate a light source of color light suitable for a subject among the plurality of light sources;
An image capturing unit that captures an object illuminated by a light source of color light suitable for the object;
An image analysis unit that analyzes image data captured by the image capturing unit to determine a normal state and an abnormal state;
A diagnostic system characterized in that is provided.   The image analysis unit converts pixel values of image data into color data and expresses them on coordinates, and determines that the data is abnormal data when the coordinates are more than a predetermined distance apart from the coordinates in a normal state. The diagnostic system according to any one of claims 1 to 3, wherein: