US9345430B2 - Imaging apparatus and imaging method thereof, image processing apparatus and image processing method thereof, and program - Google Patents
Imaging apparatus and imaging method thereof, image processing apparatus and image processing method thereof, and program Download PDFInfo
- Publication number
- US9345430B2 US9345430B2 US14/446,858 US201414446858A US9345430B2 US 9345430 B2 US9345430 B2 US 9345430B2 US 201414446858 A US201414446858 A US 201414446858A US 9345430 B2 US9345430 B2 US 9345430B2
- Authority
- US
- United States
- Prior art keywords
- image
- light
- polarized light
- unpolarized
- emitting portion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 178
- 238000012545 processing Methods 0.000 title claims description 185
- 238000003672 processing method Methods 0.000 title claims description 9
- 230000010287 polarization Effects 0.000 claims abstract description 155
- 238000011156 evaluation Methods 0.000 claims description 143
- 238000004458 analytical method Methods 0.000 claims description 39
- 230000036548 skin texture Effects 0.000 claims description 35
- 238000000034 method Methods 0.000 claims description 29
- 210000002374 sebum Anatomy 0.000 claims description 11
- 208000019300 CLIPPERS Diseases 0.000 claims description 10
- 208000021930 chronic lymphocytic inflammation with pontine perivascular enhancement responsive to steroids Diseases 0.000 claims description 10
- 210000002615 epidermis Anatomy 0.000 description 199
- 210000003491 skin Anatomy 0.000 description 84
- 235000019557 luminance Nutrition 0.000 description 43
- 230000014509 gene expression Effects 0.000 description 41
- 238000010586 diagram Methods 0.000 description 24
- 238000004364 calculation method Methods 0.000 description 20
- 238000007781 pre-processing Methods 0.000 description 19
- 238000002372 labelling Methods 0.000 description 18
- 230000008569 process Effects 0.000 description 18
- 230000006870 function Effects 0.000 description 17
- 238000001514 detection method Methods 0.000 description 16
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 16
- NUHSROFQTUXZQQ-UHFFFAOYSA-N isopentenyl diphosphate Chemical compound CC(=C)CCO[P@](O)(=O)OP(O)(O)=O NUHSROFQTUXZQQ-UHFFFAOYSA-N 0.000 description 12
- 238000012937 correction Methods 0.000 description 8
- 238000010606 normalization Methods 0.000 description 6
- 230000008859 change Effects 0.000 description 5
- 230000007423 decrease Effects 0.000 description 5
- 206010013786 Dry skin Diseases 0.000 description 4
- 206010039792 Seborrhoea Diseases 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 229910003460 diamond Inorganic materials 0.000 description 4
- 239000010432 diamond Substances 0.000 description 4
- 230000037336 dry skin Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000037312 oily skin Effects 0.000 description 4
- 230000001186 cumulative effect Effects 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 238000009499 grossing Methods 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 230000032683 aging Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000005401 electroluminescence Methods 0.000 description 2
- 230000036541 health Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000037311 normal skin Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000003705 background correction Methods 0.000 description 1
- 230000002146 bilateral effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000012447 hatching Effects 0.000 description 1
- 238000003702 image correction Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 210000000106 sweat gland Anatomy 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/44—Detecting, measuring or recording for evaluating the integumentary system, e.g. skin, hair or nails
- A61B5/441—Skin evaluation, e.g. for skin disorder diagnosis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
- A61B5/0077—Devices for viewing the surface of the body, e.g. camera, magnifying lens
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B2576/00—Medical imaging apparatus involving image processing or analysis
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/21—Polarisation-affecting properties
-
- G06K9/00885—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10141—Special mode during image acquisition
- G06T2207/10152—Varying illumination
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30088—Skin; Dermal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
Definitions
- the present disclosure relates to an imaging apparatus and an imaging method thereof, an image processing apparatus and an image processing method thereof, and a program, and in particular to an imaging apparatus and an imaging method thereof, an image processing apparatus and an image processing method thereof, and a program, that enable a skin surface condition to be evaluated with a lower-cost configuration.
- Patent Document 1 Japanese Patent Application Laid-open No. 2010-273737
- an imaging apparatus including: an unpolarized light-emitting portion configured to emit light having an unpolarized component; a polarized light-emitting portion configured to emit light having a predetermined polarized component via a first polarization filter; and an imaging element configured to image a subject, which is irradiated with light by one of the unpolarized light-emitting portion and the polarized light-emitting portion, through a second polarization filter, the first polarization filter and the second polarization filter having polarization directions in an orthogonal relationship to each other, the imaging element being further configured to image the subject irradiated with light by the unpolarized light-emitting portion and the subject irradiated with light by the polarized light-emitting portion in a time division manner and output an unpolarized light image and an orthogonal polarized light image that are obtained as a result of imaging.
- an imaging method of an imaging apparatus including an unpolarized light-emitting portion configured to emit light having an unpolarized component, a polarized light-emitting portion configured to emit light having a predetermined polarized component through the first polarization filter, and an imaging element configured to image a subject, which is irradiated with light by one of the unpolarized light-emitting portion and the polarized light-emitting portion, through a second polarization filter, the first polarization filter and the second polarization filter having polarization directions in an orthogonal relationship to each other, the method including by the imaging apparatus: imaging the subject irradiated with light by the unpolarized light-emitting portion and the subject irradiated with light by the polarized light-emitting portion in a time division manner; and outputting an unpolarized light image and an orthogonal polarized light image.
- the subject irradiated with light by the unpolarized light-emitting portion and the subject irradiated with light by the polarized light-emitting portion are imaged in the time division manner and the unpolarized light image and the orthogonal polarized light image are output.
- an image processing apparatus including a specular reflection image generator configured to acquire an unpolarized light image and an orthogonal polarized light image that are captured by an imaging apparatus including an unpolarized light-emitting portion configured to emit light having an unpolarized component, a polarized light-emitting portion configured to emit light having a predetermined polarized component through the first polarization filter, and an imaging element configured to image a subject, which is irradiated with light by one of the unpolarized light-emitting portion and the polarized light-emitting portion, through a second polarization filter, the first polarization filter and the second polarization filter having polarization directions in an orthogonal relationship to each other, and generate, from the unpolarized light image and the orthogonal polarized light image, a specular reflection image being an image having a specular reflection component.
- an imaging processing method of an image processing apparatus including: acquiring an unpolarized light image and an orthogonal polarized light image that are captured by an imaging apparatus including an unpolarized light-emitting portion configured to emit light having an unpolarized component, a polarized light-emitting portion configured to emit light having a predetermined polarized component through the first polarization filter, and an imaging element configured to image a subject, which is irradiated with light by one of the unpolarized light-emitting portion and the polarized light-emitting portion, through a second polarization filter, the first polarization filter and the second polarization filter having polarization directions in an orthogonal relationship to each other; and generating a specular reflection image being an image having a specular reflection component from the unpolarized light image and the orthogonal polarized light image.
- a program that causes a computer to execute a process, the computer being configured to process an unpolarized light image and an orthogonal polarized light image that are captured by an imaging apparatus including an unpolarized light-emitting portion configured to emit light having an unpolarized component, a polarized light-emitting portion configured to emit light having a predetermined polarized component through the first polarization filter, and an imaging element configured to image a subject, which is irradiated with light by one of the unpolarized light-emitting portion and the polarized light-emitting portion, through a second polarization filter, the first polarization filter and the second polarization filter having polarization directions in an orthogonal relationship to each other, the process including generating a specular reflection image being an image having a specular reflection component from the unpolarized light image and the orthogonal polarized light image.
- the specular reflection image being the image having the specular reflection component is generated from the unpolarized light image and the orthogonal polarized light image.
- an image processing apparatus including a parallel polarized light image generator configured to acquire an unpolarized light image and an orthogonal polarized light image that are captured by an imaging apparatus including an unpolarized light-emitting portion configured to emit light having an unpolarized component, a polarized light-emitting portion configured to emit light having a predetermined polarized component through the first polarization filter, and an imaging element configured to image a subject, which is irradiated with light by one of the unpolarized light-emitting portion and the polarized light-emitting portion, through a second polarization filter, the first polarization filter and the second polarization filter having polarization directions in an orthogonal relationship to each other, and generate a parallel polarized light image being an image having a parallel polarized component from the unpolarized light image and the orthogonal polarized light image.
- an imaging processing method of an image processing apparatus including: acquiring an unpolarized light image and an orthogonal polarized light image that are captured by an imaging apparatus including an unpolarized light-emitting portion configured to emit light having an unpolarized component, a polarized light-emitting portion configured to emit light having a predetermined polarized component through the first polarization filter, and an imaging element configured to image a subject, which is irradiated with light by one of the unpolarized light-emitting portion and the polarized light-emitting portion, through a second polarization filter, the first polarization filter and the second polarization filter having polarization directions in an orthogonal relationship to each other; and generating a parallel polarized light image being an image having a parallel polarized component from the unpolarized light image and the orthogonal polarized light image.
- a program that causes a computer to execute a process, the computer being configured to process an unpolarized light image and an orthogonal polarized light image that are captured by an imaging apparatus including an unpolarized light-emitting portion configured to emit light having an unpolarized component, a polarized light-emitting portion configured to emit light having a predetermined polarized component through the first polarization filter, and an imaging element configured to image a subject, which is irradiated with light by one of the unpolarized light-emitting portion and the polarized light-emitting portion, through a second polarization filter, the first polarization filter and the second polarization filter having polarization directions in an orthogonal relationship to each other, the process including generating a parallel polarized light image being an image having a parallel polarized component from the unpolarized light image and the orthogonal polarized light image.
- the parallel polarized light image being the image having the parallel polarized component is generated from the unpolarized light image and the orthogonal polarized light image.
- program may be provided by being transmitted through a transmission medium or recorded in a recording medium.
- the imaging apparatus and the image processing apparatus may be independent apparatuses or may be internal blocks constituting a single apparatus.
- FIG. 1 is a block diagram showing an imaging system according to a first embodiment of the present disclosure
- FIG. 2 is a view showing a lens-barrel portion of an imaging apparatus
- FIG. 3 is a view showing an outline of preprocessing by a preprocessor
- FIG. 4 is a block diagram showing a detailed configuration example of a specular reflection image generator
- FIG. 5 is a view explaining the processing contents of clipping processing by a clipper
- FIG. 6 is a view explaining the contents of processing by a shine image calculator
- FIG. 7 is a view showing an example of a shine evaluation value calculation graph stored in a shine evaluator
- FIG. 8 is a view showing an example of an evaluation result presentation screen
- FIG. 9 is a view showing another example of the evaluation result presentation screen.
- FIG. 10 is a flowchart explaining skin shine evaluation processing in the imaging system according to the first embodiment
- FIG. 11 is a block diagram showing an imaging system according to a second embodiment of the present disclosure.
- FIG. 12 is a block diagram indicating a detailed configuration example of a parallel polarized light image generator
- FIG. 13 is a view explaining a difference in processing between the first embodiment and the second embodiment
- FIG. 14 is a view explaining the difference in processing between the first embodiment and the second embodiment
- FIG. 15 is a block diagram showing a detailed configuration example of a texture analyzer
- FIG. 16 is a block diagram showing functional configuration examples of an epidermis image processor and an epidermis pattern detector
- FIG. 17 is a block diagram showing a functional configuration example of an acquired-element analyzer
- FIG. 18 is a flowchart explaining skin texture evaluation processing in the imaging system according to the second embodiment.
- FIG. 19 is a flowchart explaining details of texture analysis processing
- FIG. 20 is a flowchart explaining details of epidermis image processing
- FIG. 21 is a flowchart explaining details of epidermis pattern detection processing
- FIG. 22 is a flowchart explaining details of acquired-element analysis processing
- FIG. 23 is a view showing an example of a histogram of a size of a ridge region
- FIG. 24 is a view showing an example of a normalization curve of an epidermis size distribution analyzer
- FIG. 25 is a flowchart explaining details of epidermis shape distribution analysis processing 1 ;
- FIG. 26 is a view showing an example of a normalization curve of an epidermis shape distribution analyzer
- FIG. 27 is a flowchart explaining details of epidermis shape distribution analysis processing 2 ;
- FIG. 28 is a view explaining examples of a reference shape
- FIG. 29 is a view showing a presentation example of a texture evaluation result
- FIG. 30 is a view showing another presentation example of the texture evaluation result
- FIG. 31 is a block diagram showing an imaging system according to a third embodiment of the present disclosure.
- FIG. 32 is a block diagram showing an imaging system according to a fourth embodiment of the present disclosure.
- FIG. 33 is a block diagram showing a configuration example of a computer according to an embodiment of the present disclosure.
- FIG. 1 is a block diagram showing an imaging system according to a first embodiment of the present disclosure.
- An imaging system 1 shown in FIG. 1 includes an imaging apparatus 11 , an image processing apparatus 12 , and a display apparatus 13 .
- the imaging system 1 is a system that captures a skin image and evaluates skin shine as a skin surface condition.
- the imaging apparatus 11 captures a skin image of an examinee and supplies the captured skin image to the image processing apparatus 12 . More specifically, the imaging apparatus 11 captures, as the skin image, two kinds of (two) images of an unpolarized light image I T and an orthogonal polarized light image I PV , and supplies them to the image processing apparatus 12 .
- FIG. 2 is a view showing a lens-barrel portion of the imaging apparatus 11 .
- a lens-barrel 41 of the imaging apparatus 11 includes a plurality of light-emitting portions 42 in an annular form.
- the lens-barrel 41 includes an image sensor 43 at a center thereof.
- the image sensor 43 is configured by an imaging element, for example, a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD).
- CMOS complementary metal oxide semiconductor
- CCD charge coupled device
- the image sensor 43 is, at a front surface thereof, provided with a polarization filter.
- the light-emitting portions 42 include unpolarized light-emitting portions 42 A and polarized light-emitting portions 42 B.
- the unpolarized light-emitting portions 42 A emit light having an unpolarized component to the skin of the examinee.
- the polarized light-emitting portions 42 B emit light having a predetermined polarized component to the skin of the examinee.
- Each of the unpolarized light-emitting portions 42 A includes a light-emitting diode (LED) light source that emits white light.
- LED light-emitting diode
- each of the polarized light-emitting portions 42 B includes an LED light source that emits white light and a polarization filter provided at a front surface thereof.
- the polarization filter of the polarized light-emitting portions 42 B and the polarization filter of the image sensor 43 are attached such that polarization directions thereof are orthogonal to each other.
- the unpolarized light-emitting portions 42 A and the polarized light-emitting portions 42 B are arranged in a point symmetrical manner with respect to the image sensor 43 as shown in FIG. 2 .
- the number of unpolarized light-emitting portions 42 A and polarized light-emitting portions 42 B is not limited thereto. At least two unpolarized light-emitting portions 42 A and two polarized light-emitting portions 42 B only need to be provided in a point symmetrical manner. Alternatively, six unpolarized light-emitting portions 42 A and six polarized light-emitting portions 42 B may be provided in a point symmetrical manner.
- the unpolarized light-emitting portions 42 A and the polarized light-emitting portions 42 B in a point symmetrical manner and set the number of unpolarized light-emitting portions 42 A and the number of polarized light-emitting portions 42 B to be equal.
- other arrangements are possible.
- the imaging apparatus 11 First, by the image sensor 43 imaging the skin of the examinee via the polarization filter of a first polarization direction (horizontal direction) in a state in which the skin of the examinee serving as a subject is irradiated with light by the unpolarized light-emitting portions 42 A, the imaging apparatus 11 generates an unpolarized light image I T of the skin of the examinee. Then, the imaging apparatus 11 supplies the unpolarized light image I T to the image processing apparatus 12 .
- the imaging apparatus 11 generates an orthogonal polarized light image I PV of the skin of the examinee. Then, the imaging apparatus 11 supplies the orthogonal polarized light image I PV to the image processing apparatus 12 .
- two of the three polarization filters are placed on the light source side such that two kinds of polarizations of the parallel polarization and the orthogonal polarization can be provided, and one of the three polarization filters is placed on the image sensor side, only two kinds of polarization filters are necessary.
- the polarization filters are placed on the light source side and the image sensor side, respectively, and imaging is performed with either the light source or the image sensor being rotated at 90 degrees, the rotation function can be omitted.
- the order of imaging by the unpolarized light-emitting portions 42 A and imaging by the polarized light-emitting portions 42 B may be opposite.
- the imaging apparatus 11 only needs to be able to perform imaging by the unpolarized light-emitting portions 42 A and imaging by the polarized light-emitting portions 42 B in a time division manner, to thereby generate an unpolarized light image I T and an orthogonal polarized light image I PV .
- the polarization directions of the image sensor 43 and the polarized light-emitting portions 42 B are not limited.
- the image processing apparatus 12 includes a preprocessor 21 , a specular reflection image generator 22 , a shine analyzer 23 , and an evaluation result presentation unit 24 .
- the shine analyzer 23 includes a shine image calculator 31 , a shine value calculator 32 , and a shine evaluator 33 .
- the preprocessor 21 is supplied with two (two kinds of) skin images captured by the imaging apparatus 11 , that is, the unpolarized light image I T and the orthogonal polarized light image I PV from the imaging apparatus 11 .
- the preprocessor 21 performs preprocessing for making it easy to perform processing at the subsequent stages on the unpolarized light image I T and the orthogonal polarized light image I PV supplied from the imaging apparatus 11 . Specifically, the preprocessor 21 adjusts luminance levels of the two images to an optimal luminance level (average luminance).
- FIG. 3 is a view showing an outline of preprocessing by the preprocessor 21 .
- the preprocessor 21 adjusts the luminance value src_val of a predetermined pixel of the image before the adjustment to a luminance value dst_val of the image after the adjustment according to the following expression.
- dst _val src _val+(std_val ⁇ src _avg)
- the preprocessor 21 calculates a luminance value I T P (x, y) after the adjustment based on a luminance value I T P (x, y) before the adjustment of a pixel (x, y) of the unpolarized light image I T according to the following Expression (1).
- I T P ⁇ ( x , y ) I T ⁇ ( x , y ) + ( std_val - ⁇ x , y ⁇ I T ⁇ ( x , y ) N ) ( 1 )
- the preprocessor 21 calculates a luminance value I PV P (x, y) after the adjustment based on a luminance value I PV (x, y) before the adjustment of the pixel (x, y) of the orthogonal polarized light image I PV according to the following Expression (2).
- I PV P ⁇ ( x , y ) I PV ⁇ ( x , y ) + ( std_val - ⁇ x , y ⁇ I PV ⁇ ( x , y ) N ) ( 2 )
- N being a denominator of the fraction in each of Expressions (1) and (2) indicates the number of pixels of the unpolarized light image I T or the orthogonal polarized light image I PV .
- the standard value std_val is set and input as a fixed value in advance.
- the preprocessor 21 supplies the unpolarized light image I T P and the orthogonal polarized light image I PV P after preprocessing to the specular reflection image generator 22 . Further, the preprocessor 21 supplies the unpolarized light image I T P afterpreprocessing also to the evaluation result presentation unit 24 .
- the specular reflection image generator 22 uses the unpolarized light image I T P and the orthogonal polarized light image I PV P after preprocessing that are supplied from the preprocessor 21 . Using the unpolarized light image I T P and the orthogonal polarized light image I PV P after preprocessing that are supplied from the preprocessor 21 , the specular reflection image generator 22 generates a specular reflection image I S being an image having a specular reflection component. Then, the specular reflection image generator 22 supplies the specular reflection image I S to the shine image calculator 31 and the shine value calculator 32 .
- FIG. 4 is a block diagram showing a detailed configuration example of the specular reflection image generator 22 .
- the specular reflection image generator 22 includes a gain multiplier 51 , a subtractor 52 , and a clipper 53 .
- the gain multiplier 51 multiplies a gain Gain PP with the orthogonal polarized light image I PV P according to the following Expression (3) and obtains an orthogonal polarized light image I PV P,G after the gain multiplication.
- I PV P,G ( x,y ) Gain PP ⁇ I PV P ( x,y ) (3)
- a value for making the average luminance of the unpolarized light image I T equal to the average luminance of the orthogonal polarized light image I PV is set as the gain Gain PP .
- the preprocessor 21 may be omitted.
- the gain Gain PP for making the average luminance of the unpolarized light image I T equal to the average luminance of the orthogonal polarized light image I PV is calculated by another block or input from an external device and multiplied with respect to the orthogonal polarized light image I PV P according to Expression (3).
- a value calculated according to Expression (6) in a second embodiment to be described later may be employed as the gain Gainpp.
- the subtractor 52 subtracts from the unpolarized light image I T P the orthogonal polarized light image I PV P, G after the gain multiplication and generates a difference image I Diff1 . Specifically, the subtractor 52 performs a calculation according to the following Expression (4) on each pixel of the unpolarized light image I T P .
- I Diff1 ( x,y ) I T P ( x,y ) ⁇ I PV P,G ( x,y ) (4)
- a fluctuating component of the images for example, a minute structure of a surface or a specular reflection component can be extracted.
- the clipper 53 performs clipping processing of clipping the difference image I Diff1 calculated by the subtractor 52 such that the image is in an appropriate range. Then, the clipper 53 outputs the image after clipping processing as the specular reflection image I S .
- FIG. 5 shows a conversion graph indicating processing contents of clipping processing by the clipper 53 .
- a horizontal axis indicates the luminance value I Diff1 (x, y) of the pixel (x, y) of the difference image I Diff1 and a vertical axis indicates the luminance value I S (x, y) of the specular reflection image I S after clipping processing.
- the clipper 53 performs processing of setting a negative luminance value generated by the difference calculation of the subtractor 52 to 0.
- the specular reflection image generator 22 calculates the specular reflection image I S , using the unpolarized light image I T P and the orthogonal polarized light image I PV P after preprocessing that are supplied from the preprocessor 21 . Then, the specular reflection image generator 22 supplies the specular reflection image I S to the shine image calculator 31 and the shine value calculator 32 .
- the shine analyzer 23 analyzes the skin shine, using the specular reflection image I S supplied from the specular reflection image generator 22 .
- the shine image calculator 31 calculates a shine image I G based on the specular reflection image I S supplied from the specular reflection image generator 22 .
- the shine image I G is an image indicating an amount of sebum of the examinee.
- FIG. 6 is a view showing the contents of processing by the shine image calculator 31 with respect to the luminance value I S (x, y) of the pixel (x, y) of the specular reflection image I S .
- a horizontal axis indicates the luminance value I S (x, y) of the pixel (x, y) of the specular reflection image I S and a vertical axis indicates the luminance value I G (x, y) of the pixel (x, y) of the shine image I G .
- the shine image calculator 31 calculates the shine image I G by mapping a range of from the luminance value Is_th_min to the luminance value Is_th_max of the specular reflection image I S to luminance values of from 0 to 255.
- the shine value calculator 32 calculates a shine value (amount of sebum) Gloss_val based on the specular reflection image I S supplied from the specular reflection image generator 22 .
- the shine value Gloss_val is calculated according to the following Expression (5).
- N of a denominator indicates the number of pixels of the specular reflection image I S .
- the shine value Gloss_val can be calculated based on the average luminance of the specular reflection image I S .
- the shine image I G calculated by the shine image calculator 31 and the shine value Gloss_val calculated by the shine value calculator 32 are supplied to the shine evaluator 33 .
- the shine evaluator 33 calculates a shine evaluation value Gloss_eval from the shine value Gloss_val calculated by the shine value calculator 32 , using a shine evaluation value calculation graph stored therein.
- FIG. 7 shows an example of the shine evaluation value calculation graph stored in the shine evaluator 33 .
- the shine evaluation value calculation graph is, for example, as shown in FIG. 7 , a graph that assigns 0 as the shine evaluation value Gloss_eval to a shine value Gloss_val smaller than a first value gloss_val_th_min, assigns a shine evaluation value Gloss_eval of from 0 to 100 to a shine value Gloss_val equal to or larger than the first value gloss_val_th_min and equal to or smaller than a second value gloss_val_th_max, and assigns a shine evaluation value Gloss_eval of 100 to a shine value Gloss_val larger than the second value gloss_val_th_max.
- the shine value Gloss_val is converted into the shine evaluation value Gloss_eval taking any value of from 0 to 100.
- the shine evaluation value calculation graph is not limited to the example of FIG. 7 .
- the shine evaluator 33 supplies the shine evaluation value Gloss_eval calculated based on the shine evaluation value calculation graph to the evaluation result presentation unit 24 together with the shine image I G supplied from the shine image calculator 31 .
- the evaluation result presentation unit 24 causes the display apparatus 13 to display information indicating an evaluation result of the skin shine of the examinee, using the unpolarized light image I T P after preprocessing that is supplied from the preprocessor 21 and the shine evaluation value Gloss_eval and the shine image I G that are supplied from the shine evaluator 33 .
- the display apparatus 13 includes a liquid crystal display (LCD), an organic electro luminescence (EL) display, and the like.
- the display apparatus 13 displays a predetermined image based on an image signal supplied from the evaluation result presentation unit 24 .
- the display apparatus 13 may be included as a part of the image processing apparatus 12 or may be an apparatus having a function other than the display function, for example, a portable information terminal such as a cellular phone or a television receiver. In other words, any types of apparatuses having at least the display function may be used as the display apparatus 13 .
- FIG. 8 shows an example of the evaluation result presentation screen that presents the evaluation result of the skin shine of the examinee using the shine evaluation value Gloss_eval.
- An evaluation result presentation screen 60 shown in FIG. 8 includes a message presentation portion 61 , a water content presentation portion 62 , a oil content presentation portion 63 , and a skin condition map presentation portion 64 .
- a predetermined message selected from among a plurality of messages, which are prepared in advance, based on the oil content and the water content of the examinee is shown.
- a message saying “Skin in most sensitive condition. Sufficiently moisturize skin for preventing skin troubles!” is shown.
- a measurement result of the water content of the examinee is shown.
- the water content of the examinee is measured by, for example, a water content measuring instrument (measurement unit) that measures the water content of the skin in an electrostatic capacitance manner.
- the measured water content is supplied to the evaluation result presentation unit 24 .
- “0” is shown as the water content.
- the oil content presentation portion 63 a measurement result of the oil content of the examinee is shown.
- the shine evaluation value Gloss_eval supplied from the shine evaluator 33 is shown in the oil content presentation portion 63 as the oil content of the examinee.
- “51” is shown as the oil content.
- the oil content and the water content of the examinee are shown in a two-dimensional map indicating the oil content on a horizontal axis and the water content on a vertical axis.
- the skin condition is classified into “normal skin,” “oily skin,” “oily and dry skin,” and “dry skin” respectively corresponding to quadrants of the two-dimensional map.
- the skin condition is considered as “dry skin” when the water content is smaller than 50 and the oil content is smaller than 50, as “oily and dry skin” when the water content is smaller than 50 and the oil content is equal to or larger than 50, as “normal skin” when the water content is equal to or larger than 50 and the oil content is smaller than 50, and as “oily skin” when the water content is equal to or larger than 50 and the oil content is equal to or larger than 50.
- a word “You” indicating the skin condition of the examinee is shown.
- FIG. 9 shows another example of the evaluation result presentation screen that presents the evaluation result of the skin shine of the examinee.
- An evaluation result presentation screen 70 shown in FIG. 9 includes a shine evaluation value presentation portion 71 , a message presentation portion 72 , and a skin image presentation portion 73 .
- the shine evaluation value Gloss_eval supplied from the shine evaluator 33 is shown in the shine evaluation value presentation portion 71 .
- FIG. 9 is an example in the case where the shine evaluation value Gloss_eval supplied from the shine evaluator 33 is 75, “Shine evaluation value: 75/100” is shown in the shine evaluation value presentation portion 71 .
- a predetermined message selected from among a plurality of messages, which are prepared in advance, based on the shine evaluation value Gloss_eval of the examinee is shown.
- a message saying “Very shiny. Take care of reducing sebum.” is shown.
- the skin image presentation portion 73 an image overlapping the shine image I G supplied from the shine evaluator 33 on the unpolarized light image I T P after preprocessing that is supplied from the preprocessor 21 is shown.
- the shine image I G has a higher luminance in a shinier region. Thus, the examinee can easily check the skin surface condition.
- Step S 1 the imaging apparatus 11 captures skin images. Specifically, the imaging apparatus 11 generates two kinds of (two) images of the unpolarized light image I T and the orthogonal polarized light image I PV by imaging the skin of the examinee while emitting light to the unpolarized light-emitting portions 42 A and the polarized light-emitting portions 42 B in a time division manner. The imaging apparatus 11 supplies the resulting captured images to the image processing apparatus 12 .
- Step S 2 the preprocessor 21 performs preprocessing for making it easy to perform processing at the subsequent stages on the unpolarized light image I T and the orthogonal polarized light image I PV that are supplied from the imaging apparatus 11 .
- the preprocessor 21 adjusts luminance values according to Expressions (1) and (2) such that the average luminance of the images take an optimal value.
- the unpolarized light image I T P and the orthogonal polarized light image I PV P after preprocessing are supplied to the specular reflection image generator 22 .
- the unpolarized light image I T P after preprocessing is also supplied to the evaluation result presentation unit 24 .
- Step S 3 the gain multiplier 51 of the specular reflection image generator 22 multiplies the gain Gain PP with the orthogonal polarized light image I PV P according to Expression (3) above.
- the orthogonal polarized light image I PV P, G after the gain multiplication is supplied to the subtractor 52 .
- Step S 4 the subtractor 52 generates a difference image I Diff1 by subtracting from the orthogonal polarized light image I PV the orthogonal polarized light image I PV P, G after the gain multiplication. Specifically, the subtractor 52 performs the calculation of Expression (4) above for each of pixels of the orthogonal polarized light image I PV .
- Step S 5 the clipper 53 performs clipping processing of clipping the difference image I Diff1 calculated by the subtractor 52 such that the image is included in an appropriate range. Then, the clipper 53 outputs the image after clipping processing to the shine image calculator 31 and the shine value calculator 32 as the specular reflection image I S .
- the shine image calculator 31 of the shine analyzer 23 calculates the shine image I G based on the specular reflection image I S supplied from the specular reflection image generator 22 . More specifically, as shown in FIG. 6 , the shine image calculator 31 calculates the shine image I G by mapping a range of from the luminance value Is_th_min to the luminance value Is_th_max of the specular reflection image I S to the luminance values of from 0 to 255.
- Step S 7 the shine value calculator 32 of the shine analyzer 23 calculates the shine value Gloss_val according to Expression (5) above based on the specular reflection image I S supplied from the specular reflection image generator 22 .
- Step S 8 the shine evaluator 33 of the shine analyzer 23 evaluates the shine based on the shine value Gloss_val calculated by the shine image calculator 31 . Specifically, using the shine evaluation value calculation graph shown in FIG. 7 , the shine evaluator 33 calculates the shine evaluation value Gloss_eval based on the shine value Gloss_val. The calculated shine evaluation value Gloss_eval is supplied from the shine evaluator 33 to the evaluation result presentation unit 24 together with the shine image I G supplied from the shine image calculator 31 .
- the evaluation result presentation unit 24 causes the display apparatus 13 to display the evaluation result of the skin shine of the examinee. More specifically, the evaluation result presentation unit 24 causes the display apparatus 13 to display the evaluation result presentation screen shown in FIG. 8 and the evaluation result presentation screen shown in FIG. 9 , using the unpolarized light image I T P after preprocessing that is supplied from the preprocessor 21 , the shine evaluation value Gloss_eval and the shine image I G that are supplied from the shine evaluator 33 , and the like.
- the skin surface condition can be evaluated using the unpolarized light image I T and the orthogonal polarized light image I PV that are obtained by the imaging apparatus 11 , and hence it is possible to evaluate the skin surface condition with the low-cost configuration.
- the shine value is evaluated without binarizing the unpolarized light image I T and the orthogonal polarized light image I PV that are acquired by the imaging apparatus 11 . Therefore, it is possible to more accurately evaluate the skin surface condition without losing information.
- FIG. 11 is a block diagram showing the imaging system according to the second embodiment of the present disclosure.
- the imaging system 1 of FIG. 11 includes an imaging apparatus 11 , an image processing apparatus 12 , and a display apparatus 13 .
- the image processing apparatus 12 has a configuration partially different from that in the first embodiment.
- the image processing apparatus 12 includes a preprocessor 21 , a parallel polarized light image generator 81 , a texture analyzer 82 , and an evaluation result presentation unit 83 .
- the image processing apparatus 12 evaluates a skin texture as the skin surface condition of the examinee.
- the parallel polarized light image generator 81 uses an unpolarized light image I T P and an orthogonal polarized light image I PV P after preprocessing that are supplied from the preprocessor 21 . Using an unpolarized light image I T P and an orthogonal polarized light image I PV P after preprocessing that are supplied from the preprocessor 21 , the parallel polarized light image generator 81 generates a parallel polarized light image IPP being an image having a parallel polarized component. The parallel polarized light image generator 81 supplies the parallel polarized light image I PP to the texture analyzer 82 .
- the texture analyzer 82 uses the parallel polarized light image I PP calculated by the parallel polarized light image generator 81 to perform texture analysis processing of analyzing the skin texture of the examinee.
- the texture analyzer 82 supplies the resulting analysis result to the evaluation result presentation unit 83 .
- the evaluation result presentation unit 83 uses the analysis result supplied from the texture analyzer 82 to display information indicating the evaluation result of the skin texture condition of the examinee.
- FIG. 12 is a block diagram showing a detailed configuration example of the parallel polarized light image generator 81 .
- the parallel polarized light image generator 81 includes a gain calculator 91 , a gain multiplier 92 , a subtractor 93 , and an offset adder 94 .
- the unpolarized light image I T P after preprocessing that is supplied from the preprocessor 21 is input into the gain calculator 91 and the subtractor 93 and the orthogonal polarized light image I PV P after preprocessing is input into the gain calculator 91 and the gain multiplier 92 .
- the gain calculator 91 calculates the gain Gain PP to be multiplied by the gain multiplier 92 with the orthogonal polarized light image I PV P .
- the gain Gain PP can be calculated based on the following conception. That is, the unpolarized light image I T P consists of a surface reflection component and an internal reflection component based on a dichromatic reflection model.
- the surface reflection component has a minute structure of the surface and the specular reflection component and high-frequency components are relatively dominant.
- the internal reflection component indicates the color of an inside of the skin, and hence low-frequency components are dominant. Therefore, by calculating the gain Gain PP according to Expression (6) below, an image having the surface reflection component can be obtained.
- Gain PP arg ⁇ max ⁇ ⁇ x , y ⁇ ( Contrast ⁇ ( ⁇ ⁇ ⁇ d ⁇ ( x , y ) ) ) ( 6 )
- Argmax ⁇ ⁇ in Expression (6) indicates a function that determines a variable that maximizes a calculated value in ⁇ ⁇ .
- Expression (6) shows that the gain Gain PP that maximizes the sum of local contrasts Contrast ( ⁇ d(x, y)) is calculated.
- ⁇ d(x, y) is an expression that is calculated according to Expression (7) below and expresses processing contents of subtraction performed by the subtractor 93 with the gain Gain being an unknown.
- ⁇ d ( x,y ) I T P ( x,y ) ⁇ Gain PP ⁇ I PV P ( x,y ) (7)
- the local contrast Contrast ( ⁇ d(x, y)) is calculated according to Expression (8) by employing a method of applying a differential filter to ⁇ d(x, y) and obtaining a response as the local contrast.
- Contrast ⁇ ( ⁇ ⁇ ⁇ d ⁇ ( x , y ) ) [
- Expression (8) two kinds of 3*3 differential filters for the horizontal direction and the vertical direction are prepared, a convolution operation in each of the horizontal direction and the vertical direction, and values each taking an absolute value are added, to thereby determine the local contrast Contrast ( ⁇ d(x, y)).
- ⁇ D indicates a universal set of ⁇ d(x, y) corresponding to a region of a difference image after subtraction processing performed by the subtractor 93 and the symbol of the cross (x) enclosed by the circle ( ⁇ ) indicates the convolution operation.
- the local contrast Contrast ( ⁇ d(x, y)) may be calculated by a method different from Expression (8).
- the calculated gain Gain PP is supplied from the gain calculator 91 to the gain multiplier 92 .
- the gain multiplier 92 multiplies, according to Expression (9) below, the gain Gain PP calculated by the gain calculator 91 with the orthogonal polarized light image I PV P after preprocessing.
- the gain multiplier 92 calculates the orthogonal polarized light image I PV P, G after the gain multiplication and supplies the orthogonal polarized light image I PV P, G to the subtractor 93 .
- I PV P,G ( x,y ) Gain PP ⁇ I PV P ( x,y ) (9)
- the subtractor 93 generates the difference image I Diff2 by subtracting, according to the following Expression (10), from the unpolarized light image I T P the orthogonal polarized light image I PV P, G after the gain multiplication.
- the subtractor 93 supplies the difference image I Diff2 to thereby generate the offset adder 94 .
- I Diff2 ( x,y ) I T P ( x,y ) ⁇ I PV P,G ( x,y ) (10)
- the offset adder 94 calculates the parallel polarized light image I PP by adding an offset value std_val to a difference image I Diff2 supplied from the subtractor 93 . That is, the offset adder 94 performs a calculation according to the following Expression (11).
- the calculated parallel polarized light image I PP is supplied to the texture analyzer 82 .
- FIG. 13 is a view showing a concept of a calculation of the specular reflection image I S in the above-mentioned first embodiment.
- a horizontal axis indicates a pixel position x in a predetermined line of an image and a vertical axis indicates a luminance value (pixel value) in the pixel position x.
- a component of light emitted to the skin of the examinee and reflected includes a surface reflection component reflected on the surface of the skin and an internal reflection component reflected after entering the skin.
- a total reflection component is a reflection component including both of the surface reflection component and the internal reflection component.
- the total reflection component can be obtained as the unpolarized light image I T captured while causing the unpolarized light-emitting portions 42 A to emit light.
- the internal reflection component can be obtained as the orthogonal polarized light image I PV captured while causing the polarized light-emitting portions 42 B to emit light.
- the image processing apparatus 12 calculates the specular reflection image I S being the surface reflection component shown by a solid line in FIG. 13 . Positive regions of the specular reflection image I S calculated at this time, which are shown by hatching in FIG. 13 , correspond to the skin shine portions.
- FIG. 14 is a view showing a concept of a calculation of the parallel polarized light image I PP in the second embodiment.
- the parallel polarized light image generator 81 calculates the parallel polarized light image Ipp being the surface reflection component by adding the offset value std_val to the surface reflection component, which is obtained by subtracting the orthogonal polarized light image I PV being the internal reflection component from the unpolarized light image I T being the total reflection component, to adjust the surface reflection component to the same luminance level as that of the total reflection component or the like.
- the offset value for adjusting the surface reflection component to the same luminance level as that of the total reflection component or the like corresponds to the average value of the total reflection component (unpolarized light image I T ) and the internal reflection component (orthogonal polarized light image I PV )
- the luminance values of the unpolarized light image I T and the orthogonal polarized light image I PV has been adjusted by the preprocessor 21 to the standard value std_val, and hence the standard value std_val can be used as the offset value.
- FIG. 15 is a block diagram showing a detailed configuration example of the texture analyzer 82 .
- the texture analyzer 82 includes an epidermis image processor 101 , an epidermis pattern detector 102 , an acquired-element analyzer 103 , and a texture evaluator 104 .
- the epidermis image processor 101 is supplied with the parallel polarized light image I PP calculated by the parallel polarized light image generator 81 .
- the parallel polarized light image I PP will be also referred to as an epidermis image in the following description.
- the epidermis image processor 101 subjects the epidermis image to predetermined image processing such as correction and noise removal.
- the epidermis image processor 101 supplies the epidermis image after image processing to the epidermis pattern detector 102 and the acquired-element analyzer 103 .
- the epidermis pattern detector 102 detects a pattern of an epidermis (hereinafter, referred to as epidermis pattern) in the epidermis image, which is formed of furrows (sulcus cutises) and ridges (crista cutises) on an epidermis.
- the epidermis pattern detector 102 detects a detection result (hereinafter, referred to as epidermis pattern detection result) to the acquired-element analyzer 103 .
- the acquired-element analyzer 103 analyzes, based on the epidermis image after image processing and the epidermis pattern detection result, acquired elements out of elements indicating the skin texture condition.
- the acquired-element analyzer 103 supplies an analysis result to the texture evaluator 104 .
- the texture evaluator 104 evaluates the skin texture condition of the examinee based on the analysis result by the acquired-element analyzer 103 .
- the texture evaluator 104 supplies an evaluation result to the evaluation result presentation unit 83 ( FIG. 11 ).
- FIG. 16 is a block diagram showing a function configuration example of the epidermis image processor 101 and the epidermis pattern detector 102 .
- the epidermis image processor 101 includes an image corrector 121 , a single channel extractor 122 , and a noise remover 123 .
- the epidermis pattern detector 102 includes a binarizer 131 and a labeling processing unit 132 .
- the image corrector 121 performs predetermined image correction such as distortion correction and reduction of the epidermis image.
- the image corrector 121 supplies the epidermis image after the correction to the single channel extractor 122 .
- the single channel extractor 122 extracts a signal component of a predetermined channel from the corrected epidermis image.
- the single channel extractor 122 supplies an epidermis image of the extracted signal component (hereinafter, referred to as single-channel epidermis image) to the noise remover 123 .
- the noise remover 123 removes noise from the single-channel epidermis image.
- the noise remover 123 supplies the single-channel epidermis image after the noise removal (hereinafter, referred to as noise-removed epidermis image) to the binarizer 131 of the epidermis pattern detector 102 and the acquired-element analyzer 103 .
- the binarizer 131 performs binarization processing on the noise-removed epidermis image.
- the binarizer 131 supplies the resulting binarized image (hereinafter, referred to as binarized epidermis image) to the labeling processing unit 132 .
- the labeling processing unit 132 detects the epidermis pattern by performing labeling processing on the binarized epidermis image. More specifically, the labeling processing unit 132 detects regions of ridges (hereinafter, referred to as ridge regions) in the epidermis image as the epidermis pattern. Further, the labeling processing unit 132 counts the number of ridge regions in the epidermis image. Further, the labeling processing unit 132 supplies the acquired-element analyzer 103 with an epidermis pattern detection result indicating a detection result of the ridge regions and the number of ridges.
- FIG. 17 is a block diagram showing a function configuration example of the acquired-element analyzer 103 .
- the acquired-element analyzer 103 includes an epidermis size distribution analyzer 151 , an epidermis shape distribution analyzer 152 , an epidermis shape distribution analyzer 153 , and an epidermis directivity analyzer 154 .
- the epidermis size distribution analyzer 151 analyzes a distribution of sizes of the epidermis pattern. More specifically, the epidermis size distribution analyzer 151 analyzes a distribution of sizes of the ridge regions and calculates an epidermis size distribution evaluation value indicating uniformity of the sizes of the ridge regions. The epidermis size distribution analyzer 151 supplies the calculated epidermis size distribution evaluation value to the texture evaluator 104 .
- the epidermis shape distribution analyzer 152 analyzes a distribution of shapes of the epidermis pattern. More specifically, the epidermis shape distribution analyzer 152 analyzes a distribution of shapes of the ridge regions and calculates an epidermis shape distribution evaluation value indicating uniformity of the shapes of the ridge regions. The epidermis shape distribution analyzer 152 supplies the calculated epidermis shape distribution evaluation value to the texture evaluator 104 .
- the epidermis shape distribution analyzer 153 analyzes a distribution of shapes of the epidermis pattern from a perspective different from that of the epidermis shape distribution analyzer 152 . More specifically, the epidermis shape distribution analyzer 153 compares each ridge region with a predetermined reference shape and determines epidermis shape distribution information indicating a ratio at which the ridge regions have shapes similar to the reference shape. The epidermis shape distribution analyzer 153 supplies the determined epidermis shape distribution information to the texture evaluator 104 .
- the epidermis directivity analyzer 154 analyzes directivity of the epidermis pattern. More specifically, the epidermis directivity analyzer 154 analyzes a distribution of edge directions of the ridge regions by applying edge filters of four directions of, for example, 0 degrees, 45 degrees, 90 degrees, and 135 degrees to the noise-removed epidermis image, and calculates an epidermis directivity evaluation value indicating uniformity of the distribution of the edge directions of the ridge regions. The epidermis directivity analyzer 154 supplies the calculated epidermis directivity evaluation value to the texture evaluator 104 .
- the epidermis size distribution evaluation value, the epidermis shape distribution evaluation value, the epidermis shape distribution information, and the epidermis directivity evaluation value are indexes for evaluating acquired characteristics of the skin texture condition.
- Step S 21 the imaging apparatus 11 captures a skin image. Specifically, the imaging apparatus 11 generates two kinds of (two) images of the unpolarized light image I T and the orthogonal polarized light image I PV by imaging the skin of the examinee while causing the unpolarized light-emitting portions 42 A and the polarized light-emitting portions 42 B to emit light in a time division manner. The imaging apparatus 11 supplies the resulting captured images to the image processing apparatus 12 .
- Step S 22 the preprocessor 21 performs preprocessing for making it easy to perform processing at the subsequent stages on the unpolarized light image I T and the orthogonal polarized light image I PV supplied from the imaging apparatus 11 . Specifically, the preprocessor 21 adjusts luminance values according to Expressions (1) and (2) such that the images take an optimal average luminance value.
- the unpolarized light image I T P and the orthogonal polarized light image I PV P after preprocessing are supplied to the parallel polarized light image generator 81 .
- Step S 23 the gain calculator 91 of the parallel polarized light image generator 81 calculates the gain Gainpp to be multiplied by the gain multiplier 92 with the orthogonal polarized light image I PV P .
- Step S 24 the gain multiplier 92 multiplies the gain Gainpp calculated by the gain calculator 91 with the orthogonal polarized light image I PV P after preprocessing, and supplies the orthogonal polarized light image I PV P, G after the gain multiplication to the subtractor 93 .
- the gain multiplier 92 performs the calculation according to Expression (9) above.
- Step S 25 the subtractor 93 generates the difference image I Diff2 by subtracting from the unpolarized light image I T P the orthogonal polarized light image I PV P, G after the gain multiplication and supplies the difference image I Diff2 to the offset adder 94 . That is, the subtractor 93 performs the calculation according to Expression (10) above with each pixel of the unpolarized light image I T P .
- Step S 26 the offset adder 94 generates the parallel polarized light image I PP by adding the offset value std_val to the difference image I Diff2 according to Expression (11). Then, the offset adder 94 supplies the generated parallel polarized light image I PP to the texture analyzer 82 .
- Step S 27 the texture analyzer 82 executes texture analysis processing of analyzing the skin texture of the examinee using the parallel polarized light image I PP (epidermis image) generated in Step S 26 and supplies the resulting analysis result to the evaluation result presentation unit 83 . Details of texture analysis processing in Step S 27 will be described later with reference to FIG. 19 and the like.
- Step S 28 the evaluation result presentation unit 83 causes the display apparatus 13 to display information indicating the evaluation result of the skin texture condition of the examinee using the analysis result supplied from the texture analyzer 82 and terminates processing.
- FIG. 19 shows a detailed flowchart of texture analysis processing in Step S 27 of FIG. 18 .
- Step S 41 the epidermis image processor 101 performs epidermis image processing of subjecting the parallel polarized light image I PP as the epidermis image to predetermined image processing such as correction and noise removal. Details of epidermis image processing will be described with reference to FIG. 20 .
- Step S 42 the epidermis pattern detector 102 detects an epidermis pattern in the epidermis image, which is formed of ridges or furrows on an epidermis, and performs epidermis pattern detection processing of outputting the epidermis pattern detection result being the detection result. Details of epidermis pattern detection processing will be described later with reference to FIG. 21 .
- Step S 43 the acquired-element analyzer 103 performs, based on the epidermis image after image processing and the epidermis pattern detection result, acquired-element analysis processing of analyzing acquired elements out of elements indicating the skin texture condition. Details of acquired-element analysis processing will be described later with reference to FIG. 22 .
- Step S 44 the texture evaluator 104 evaluates, based on the analysis result by the acquired-element analyzer 103 , the skin texture condition of the examinee and calculates a texture evaluation value as the evaluation result.
- the calculated texture evaluation value is supplied to the evaluation result presentation unit 83 . Texture analysis processing is terminated. Processing proceeds to Step S 28 , referring back to FIG. 18 .
- the skin texture condition can be evaluated based on the uniformity of the texture and the shapes of the ridges being the acquired elements indicating the skin texture condition. As a result, it is possible to more accurately evaluate the skin texture condition.
- Step S 41 of FIG. 19 epidermis image processing in Step S 41 of FIG. 19 will be described in detail.
- Step S 61 the image corrector 121 corrects an image.
- the image corrector 121 performs, for example, shading correction and lens distortion correction on the epidermis image or cuts out a center region of the epidermis image.
- the image corrector 121 reduces the image after the correction.
- the epidermis image after the correction has a size of vertical 160 pixels horizontal 120 pixels unless otherwise specifically noted.
- the image corrector 121 supplies the epidermis image after the correction to the single channel extractor 122 .
- Step S 62 the single channel extractor 122 extracts signal components of a predetermined channel from the corrected epidermis image. For example, the single channel extractor 122 extracts signal components of a B (blue) channel from the corrected epidermis image. Then, the single channel extractor 122 supplies the noise remover 123 with the single-channel epidermis image composed of the extracted signal components.
- Step S 63 the noise remover 123 removes noise from the single-channel epidermis image.
- the noise remover 123 applies a smoothing filter to the single-channel epidermis image.
- the noise remover 123 applies an edge-preserving smoothing filter to the single-channel epidermis image.
- the noise remover 123 applies an isolated point removal filter to the single-channel epidermis image.
- this isolated point removal filter a median filter having 3*3 pixels is used, for example.
- the noise remover 123 supplies the noise-removed epidermis image being the single-channel epidermis image after the noise removal to the binarizer 131 of the epidermis pattern detector 102 and the epidermis directivity analyzer 154 of the acquired-element analyzer 103 .
- epidermis image processing is terminated and processing returns to texture analysis processing in FIG. 19 .
- Step S 71 the binarizer 131 performs binarization processing. Specifically, assuming that a bright region of the epidermis image is a ridge on a front side and a dark region of the epidermis image is a furrow on a deep side under a uniform light source, the binarizer 131 binarizes the noise-removed epidermis image in order to perform segmentation of the ridges and the furrows. Then, the binarizer 131 supplies the binarized epidermis image obtained by binarizing the noise-removed epidermis image to the labeling processing unit 132 .
- Step S 72 the labeling processing unit 132 performs 4 or 8-coupled labeling processing on the binarized epidermis image from the outside.
- the labeling processing unit 132 detects a region surrounded by an outermost white outline as one region and ignores a black region or a region surrounded by another while outline within this region even if it is present. With this, for example, a region that is dark due to the presence of a recess within the ridge and the like is ignored, and hence it is possible to accurately detect the ridge regions.
- labelling regions regions labeled in labeling processing will be referred to as labelling regions.
- an interval between furrows of an average human skin is 0.25 to 0.5 mm.
- a ridge has an area of from approximately 0.031 to 0.25 mm 2 .
- the labeling processing unit 132 calculates an appropriate range of a size of a ridge in an epidermis image based on the size or the like of the image sensor 43 of the imaging apparatus 11 . Then, the labeling processing unit 132 detects, from the detected labelling regions, regions each having a size within the calculated appropriate range as the ridge regions.
- the labeling processing unit 132 counts the number of detected ridge regions as the number of ridges N ridge .
- the labeling processing unit 132 supplies an epidermis pattern detection result indicating the detection result of the ridge regions and the number of ridges N ridge to the epidermis size distribution analyzer 151 , the epidermis shape distribution analyzer 152 , and the epidermis shape distribution analyzer 153 of the acquired-element analyzer 103 .
- Step S 81 the epidermis size distribution analyzer 151 analyzes a distribution of sizes of the epidermis pattern.
- the epidermis size distribution analyzer 151 creates a histogram of the size of the ridge region.
- FIG. 23 shows an example of the histogram of the size (area) of the ridge region.
- a horizontal axis indicates a size of the ridge region and a vertical axis indicates a frequency frq n of each bin of a histogram.
- the epidermis size distribution analyzer 151 calculates an average value H avg of sizes of the ridge regions according to the following Expression (12).
- n indicates a median of each bin.
- the epidermis size distribution analyzer 151 calculates a variance H var of the sizes of the ridge regions according to the following Expression (13).
- H var ⁇ n ⁇ ( ( n - H avg ) 2 ⁇ frq n ) ⁇ n ⁇ ( frq n ) ( 13 )
- the epidermis size distribution analyzer 151 calculates, based on a normalization curve shown in FIG. 24 , an epidermis size distribution evaluation value Eeval size where the variance H var is normalized to a range of from 0 to 1.
- Size_th_min and Size_th_max are threshold values each of which determines the normalization curve.
- the epidermis size distribution evaluation value Eeval size increases as the variance H var of the sizes of the ridge regions decreases. In other words, the epidermis size distribution evaluation value Eeval size increases as a variation of the sizes of the ridge regions decreases. Thus, the epidermis size distribution evaluation value Eeval size is an index indicating the uniformity of the sizes of the ridge regions.
- the epidermis size distribution analyzer 151 supplies the epidermis size distribution evaluation value Eeval size to the texture evaluator 104 .
- Step S 82 the epidermis shape distribution analyzer 152 performs epidermis shape distribution analysis processing 1 .
- Step S 82 epidermis shape distribution analysis processing 1 in Step S 82 will be described in detail.
- Step S 101 the epidermis shape distribution analyzer 152 selects a reference region. Specifically, the epidermis shape distribution analyzer 152 selects one ridge region which has not been yet set as the reference region and set the ridge region as the reference region.
- the epidermis shape distribution analyzer 152 selects a comparison region. Specifically, the epidermis shape distribution analyzer 152 selects one ridge region the shape of which has not been yet compared with that of the reference region and sets the ridge region as the comparison region.
- Step S 103 the epidermis shape distribution analyzer 152 calculates a shape difference between the reference region and the comparison region.
- the epidermis shape distribution analyzer 152 digitalizes shapes of the reference region and the comparison region using Hu invariant moments and calculates a shape difference between the reference region and the comparison region based on the digitalized values.
- the method of calculating the difference is not particularly limited, the difference decreases as the shape of the reference region and the shape of the comparison region becomes more similar to each other.
- Step S 104 the epidermis shape distribution analyzer 152 integrates differences. Specifically, the epidermis shape distribution analyzer 152 adds a newly calculated difference to the already calculated integrated value of the differences of the ridge regions.
- Step S 105 the epidermis shape distribution analyzer 152 determines whether or not a ridge region not compared with the reference region remains. If it is determined that the ridge region not compared with the reference region remains, processing returns to Step S 102 .
- Step S 105 until it is determined that the ridge region not compared with the reference region does not remain, processing of from Steps S 102 to S 105 is repeatedly performed.
- Step S 105 If it is determined in Step S 105 that the ridge region not compared with the reference region does not remain, processing proceeds to Step S 106 .
- Step S 106 the epidermis shape distribution analyzer 152 determines whether or not a ridge region not set as the reference region remains. If it is determined that the ridge region not set as the reference region remains, processing returns to Step S 101 .
- Step S 106 After that, until it is determined in Step S 106 that the ridge region not set as the reference region does not remain, processing of from Steps S 101 to S 106 is repeatedly performed. As a result, differences are calculated with respect to all combinations of the ridge regions and a cumulative addition value of the differences is further calculated.
- Step S 106 If it is determined in Step S 106 that the ridge region not set as the reference region does not remain, processing proceeds to Step S 107 .
- Step S 107 the epidermis shape distribution analyzer 152 calculates a difference average Diff avg according to the following Expression (14).
- R i and R j indicate ridge regions of a label i and a label j, respectively.
- a denominator on the right side of Expression (14) becomes the cumulative addition value of the shape differences of all the combinations of the ridge regions.
- N comp is calculated according to the following Expression (15) and indicates the number of comparisons of the shapes of the ridge regions.
- N comp N ridge ⁇ ( N ridge - 1 ) 2 ( 15 )
- the epidermis shape distribution analyzer 152 calculates an evaluation value. Specifically, the epidermis shape distribution analyzer 152 calculates, based on a normalization curve shown in FIG. 26 , an epidermis shape distribution evaluation value Eeval shape where the difference average Diff avg is normalized to a range of from 0 to 1.
- Shape_th_min and Shape_th_max are threshold values each of which determines the normalization curve.
- the epidermis shape distribution evaluation value Eeval shape increases as the difference average Diff avg of the shapes of the ridge regions decreases. In other words, the epidermis shape distribution evaluation value Eeval shape increases as a variation in the shapes of the ridge regions decreases. Thus, the epidermis shape distribution evaluation value Eeval shape is an index indicating uniformity of the shapes of the ridge regions.
- the epidermis shape distribution analyzer 152 supplies the epidermis shape distribution evaluation value Eeval shape to the texture evaluator 104 .
- epidermis shape distribution analysis processing 1 is terminated.
- Step S 83 the epidermis shape distribution analyzer 153 performs epidermis shape distribution analysis processing 2 .
- Step S 83 epidermis shape distribution analysis processing 2 in Step S 83 will be described in detail.
- Step S 121 the epidermis shape distribution analyzer 153 selects a reference shape.
- the ridges In general, it is ideal that the ridges have triangle or diamond shapes. In contrast, a shape branching into two or more parts and an elongated shape are considered as non-ideal shapes.
- the epidermis shape distribution analyzer 153 sets, for example, Shape 0 to Shape 3 shown in FIG. 28 as reference shapes.
- the reference shapes Shape 0 and Shape 1 are a triangle shape and a diamond shape, respectively, which are similar to the ideal shapes of the ridges.
- the reference shapes Shape 2 and Shape 3 are a shape branching into two parts and an elongated shape, respectively, which are similar to the non-ideal shapes of the ridges.
- the epidermis shape distribution analyzer 153 selects one reference shape which has not been yet compared with the ridge region.
- Step S 122 the epidermis shape distribution analyzer 153 selects a comparison region. Specifically, the epidermis shape distribution analyzer 152 selects one ridge region which has not been yet compared with the reference shape and sets the ridge region as the comparison region.
- Step S 123 the epidermis shape distribution analyzer 153 calculates a shape difference between the reference shape and the comparison region. Note that the same method as in the calculation of the difference between the reference region and the comparison region of the ridge region in Step S 103 of FIG. 25 , which is described above, is used for the calculation of the difference at this time.
- Step S 124 the epidermis shape distribution analyzer 153 integrates differences. Specifically, the epidermis shape distribution analyzer 153 adds a newly calculated difference to the previously calculated integrated value of the differences of the ridge regions from the current reference shape.
- Step S 125 the epidermis shape distribution analyzer 153 determines whether or not a ridge region not compared with the current reference shape remains. If it is determined that the ridge region not compared with the current reference shape remains, processing returns to Step S 122 .
- Step S 125 After that, until it is determined in Step S 125 that the ridge region not compared with the current reference shape does not remain, processing of from Steps S 122 to S 125 is repeatedly performed.
- Step S 125 If it is determined in Step S 125 that the ridge region not compared with the current reference shape does not remain, processing proceeds to Step S 126 .
- Step S 126 the epidermis shape distribution analyzer 153 determines whether or not the reference shape not compared remains. If it is determined that the reference shape not compared remains, processing returns to Step S 121 .
- Step S 126 After that, until it is determined in Step S 126 that the reference shape not compared does not remain, processing of from Steps S 121 to S 126 is repeatedly performed. As a result, as shown in the following Expression (16), a cumulative addition value Diff i of the differences of the shapes of the ridge regions from the reference shapes is calculated.
- S i indicates a reference shape with a value of ID being i.
- Step S 126 If it is determined in Step S 126 that the reference shape not compared does not remain, processing proceeds to Step S 127 .
- Step S 127 the epidermis shape distribution analyzer 153 calculates a shape ratio of the ridge regions. Specifically, the epidermis shape distribution analyzer 153 calculates an epidermis shape distribution information ShapeRatio i indicating the shape ratio of the ridge regions according to the following Expression (17).
- N RS indicates the total number of reference shapes.
- the epidermis shape distribution information ShapeRatio i indicates a ratio at which the ridge regions have shapes similar to the reference shape with the value of ID being i.
- the epidermis shape distribution analyzer 152 supplies the epidermis shape distribution information ShapeRatio i to the texture evaluator 104 .
- epidermis shape distribution analysis processing 2 is terminated.
- the epidermis directivity analyzer 154 analyzes the directivity of the epidermis pattern by applying an edge filter of four directions of, for example, 0 degrees, 45 degrees, 90 degrees, and 135 degrees to the noise-removed epidermis image. More specifically, the epidermis directivity analyzer 154 calculates an epidermis directivity evaluation value Eeval direction indicating uniformity of the distribution of the edge directions of the ridge regions, which takes a value smaller than 1 when the edge directions of the ridge regions are not uniformly distributed in the four directions.
- Eeval direction indicating uniformity of the distribution of the edge directions of the ridge regions
- acquired-element analysis processing is terminated and processing returns to texture analysis processing in FIG. 19 and proceeds to the calculation of the texture evaluation value by the texture evaluator 104 in the subsequent Step S 44 .
- the texture evaluator 104 calculates a texture evaluation value eval1 total according to the following Expression (18).
- eval1 total E eval size *E eval shape *E eval direction (18)
- the texture evaluation value eval1 total increases as the uniformity of the sizes of the ridges, uniformity of the shapes of the ridges, and uniformity of the directions of the ridges increases, that is, as the texture is generally smoother (uniformity of texture increases). Further, the uniformity of the sizes of the ridges, the uniformity of the shapes of the ridges, and the uniformity of the directions of the ridges change in an acquired manner due to aging, health condition, skin care, and the like. Thus, the texture evaluation value eval1 total is an index for evaluating the uniformity of the skin texture, which changes in an acquired manner.
- This uniformity of the texture significantly influences the appearance of the skin like the fineness of the texture. That is, if the texture is generally smooth even when the texture is fine, the appearance of the skin is bad. On the other hand, if the texture is generally smooth even when the texture is not fine, the appearance of the skin is good.
- the texture evaluation value eval2 total may be calculated according to the following Expression (19).
- eval2 total E eval size *E eval shape *E eval direction *ShapeRatio ideal (19)
- ShapeRatio ideal is calculated according to the following Expression (20), for example.
- ShapeRatio ideal ShapeRatio 0 *ShapeRatio 1 (20)
- ShapeRatio 0 is ShapeRatio with respect to the reference shape Shape 0 of the triangle in FIG. 28 .
- ShapeRatio 1 is ShapeRatio with respect to the reference shape Shape 1 of the diamond in FIG. 28 . That is, ShapeRatio ideal shows a ratio at which the ridge regions have triangle or diamond shapes considered as the ideal shapes.
- the texture evaluation value eval2 total is an index for evaluating the acquired elements that affect the skin texture condition in more details in comparison with the texture evaluation value eval1 total .
- the texture evaluator 104 supplies the evaluation result of the skin texture condition to the evaluation result presentation unit 83 .
- the texture evaluator 104 supplies the evaluation result presentation unit 83 not only with the texture evaluation value eval1 total and the texture evaluation value eval2 total , but also with the evaluation values used when the texture evaluation value eval1 total and the texture evaluation value eval2 total are calculated.
- the evaluation result presentation unit 83 causes the display apparatus 13 to display a screen shown in FIG. 29 .
- a radar chart showing current evaluation values of the uniformity of the sizes of the ridges, the uniformity of the shapes of the ridges, the uniformity of the distribution of the directions of the ridges, and the fineness of the texture with the current evaluation values being individually compared with preceding evaluation values.
- an epidermis size distribution evaluation value Eeval size As the values in this radar chart, an epidermis size distribution evaluation value Eeval size , an epidermis shape distribution evaluation value Eeval shape , an epidermis directivity evaluation value Eeval direction , and a number-of-ridges evaluation value Eeval num are used.
- a change in comprehensive determination between the preceding skin texture condition and the current skin texture condition is shown.
- This comprehensive determination value is displayed based on a result obtained by comparing the texture evaluation value eval3 total obtained by comprehensively evaluating, for example, the epidermis size distribution evaluation value Eeval size , the epidermis shape distribution evaluation value Eeval shape , the epidermis directivity evaluation value Eeval directio , and the number-of-ridges evaluation value Eevalnum with the preceding texture evaluation value eval3 total .
- the examinee can immediately know the skin condition and also know a change in the skin condition from the preceding one.
- a circle graph indicating a distribution of shapes of the ridges may also be presented.
- the texture analyzer 82 in texture analysis processing by the texture analyzer 82 , it is possible to separately evaluate the acquired characteristics and inherent characteristics of the skin texture condition. In addition, it is possible to evaluate the skin texture condition in more details based on the acquired characteristics and the inherent characteristics.
- FIG. 31 is a block diagram showing an imaging system according to a third embodiment of the present disclosure.
- An imaging system 1 according to the third embodiment is a system serving both a function of evaluating the skin shine in the above-mentioned first embodiment and a function of evaluating the skin texture in the above-mentioned second embodiment.
- an image processing apparatus 12 in FIG. 31 includes a preprocessor 21 , a specular reflection image generator 22 , a shine analyzer 23 , and an evaluation result presentation unit 24 and a parallel polarized light image generator 81 , a texture analyzer 82 , and an evaluation result presentation unit 83 .
- the image processing apparatus 12 is capable of performing both a process of evaluating the skin shine and a process of evaluating the skin texture using the two kinds of (two) images of the unpolarized light image I T and the orthogonal polarized light image I PV that are supplied from the imaging apparatus 11 .
- the image processing apparatus 12 may also perform either one of the process of evaluating the skin shine and the process of evaluating the skin texture according to an instruction selected by an operator, initial settings, or the like.
- the configurations and operations of the image processing apparatus 12 are the same as those of the above-mentioned first and second embodiments, and hence descriptions thereof will be omitted.
- FIG. 32 is a block diagram showing an imaging system according to a fourth embodiment of the present disclosure.
- first to third embodiments are embodiments in which the image processing apparatus 12 placed at a closer distance from the imaging apparatus 11 acquires an image signal via a cable or the like and performs image processing
- the functions performed by the image processing apparatus 12 may be performed by a cloud server or the like.
- the imaging system 1 shown in FIG. 32 represents a configuration example in which the above-mentioned functions performed by the image processing apparatus 12 are performed by the cloud server.
- the imaging system 1 includes an imaging apparatus 201 and a server 202 .
- the imaging apparatus 201 includes an imaging unit 221 , a transmitter/receiver 222 , and a display unit 223 .
- the server 202 includes a transmitter/receiver 241 and an image processing unit 242 .
- the imaging unit 221 has the same functions as those of the imaging apparatus 11 described above.
- the imaging unit 221 captures two kinds of (two) images of an unpolarized light image I T and an orthogonal polarized light image I PV as skin images and supplies these images to the transmitter/receiver 222 .
- the transmitter/receiver 222 transmits the two images supplied from the imaging unit 221 to the server 202 via a network such as a local area network (LAN) and the Internet.
- a network such as a local area network (LAN) and the Internet.
- the transmitter/receiver 222 receives information indicating a skin shine evaluation result or a skin texture evaluation result, which is transmitted from the server 202 , and supplies the information to the display unit 223 .
- the display unit 223 displays, based on the information supplied from the transmitter/receiver 222 , the skin shine evaluation result or the skin texture evaluation result.
- the transmitter/receiver 241 of the server 202 receives the two kinds of (two) images of the unpolarized light image I T and the orthogonal polarized light image I PV that are transmitted from the transmitter/receiver 222 of the imaging apparatus 201 .
- the transmitter/receiver 241 acquires the information indicating the skin shine evaluation result or the skin texture evaluation result, which is obtained by image processing in the image processing unit 242 , and transmits the information to the imaging apparatus 201 .
- the image processing unit 242 has the same functions as those of the image processing apparatus 12 according to any of the above-mentioned first to third embodiments and performs the skin shine evaluation or the skin texture evaluation based on the two kinds of images of the unpolarized light image I T and the orthogonal polarized light image I PV .
- all the functions of the image processing apparatus 12 according to the first to third embodiments are performed by the image processing unit 242 of the server 202 .
- the contents of processing performed by the server 202 may be appropriately set.
- the server 202 may perform some of the functions of the image processing apparatus 12 . In this case, sharing of image processing on the imaging apparatus 11 side and image processing on the server 202 side may be arbitrarily set.
- the series of image processing described above may be performed by hardware or may be performed by software. If the series of image processing are performed by software, programs configuring the software are installed on a computer.
- the computer includes a computer incorporated in dedicated hardware, a general-purpose personal computer, for example, capable of executing various functions by installing various programs, and the like.
- FIG. 33 is a block diagram showing a configuration example of hardware of a computer that executes the series of image processing described above according to the programs.
- a central processing unit (CPU) 301 In the computer, a central processing unit (CPU) 301 , a read only memory (ROM) 302 , and a random access memory (RAM) 303 are connected to one another via a bus 304 .
- CPU central processing unit
- ROM read only memory
- RAM random access memory
- an input/output interface 305 is connected to the bus 304 .
- An input unit 306 , an output unit 307 , a storage unit 308 , a communication unit 309 , and a drive 310 are connected to the input/output interface 305 .
- the input unit 306 includes a keyboard, a mouse, a microphone, and the like.
- the output unit 307 includes a display, a speaker, and the like.
- the storage unit 308 includes a hard disk, a non-volatile memory, and the like.
- the communication unit 309 includes a network interface and the like.
- the drive 310 drives a removable medium 311 such as a magnetic disk, an optical disc, a magneto-optical disk, and a semiconductor memory.
- the CPU 301 by the CPU 301 loading the programs stored in, for example, the storage unit 308 into the RAM 303 via the input/output interface 305 and the bus 304 and executing the loaded programs, the series of image processing described above are performed.
- the programs can be installed on the storage unit 308 via the input/output interface 305 by the removable medium 311 being mounted on the drive 310 .
- the programs may be received by the communication unit 309 via a wired or wireless communication medium such as a local area network, the Internet, and digital satellite broadcasting and installed on the storage unit 308 .
- the programs may be installed on the ROM 302 and the storage unit 308 in advance.
- the steps described in the flowcharts may be performed in time series following the described order, of course, and do not need to be necessarily processed in time series.
- the steps may be performed in parallel or at a necessary timing, for example, when a call is performed.
- system set forth in the present specification means a collection of a plurality of components (apparatuses, modules (components), etc.) and all components may or do not need to be included in a single casing.
- a plurality of apparatuses housed in respective casings and connected to one another via a network and a single apparatus including a plurality of modules housed in a single casing are systems.
- Embodiments of the present disclosure are not limited to the above-mentioned embodiments and various modifications can be made without departing from the gist of the present disclosure.
- the present disclosure may take a cloud computing configuration in which a single function is shared and cooperatively processed by a plurality of apparatuses over a network.
- steps described with reference to the flowcharts may be performed by a single apparatus or may be shared and performed by a plurality of apparatuses.
- a single step includes a plurality of processes
- the plurality of processes included in the single step may be performed by a single apparatus or may be shared and performed by a plurality of apparatuses.
- an unpolarized light-emitting portion configured to emit light having an unpolarized component
- a polarized light-emitting portion configured to emit light having a predetermined polarized component via a first polarization filter
- an imaging element configured to image a subject, which is irradiated with light by one of the unpolarized light-emitting portion and the polarized light-emitting portion, through a second polarization filter, the first polarization filter and the second polarization filter having polarization directions in an orthogonal relationship to each other, the imaging element being further configured to image the subject irradiated with light by the unpolarized light-emitting portion and the subject irradiated with light by the polarized light-emitting portion in a time division manner and output an unpolarized light image and an orthogonal polarized light image that are obtained as a result of imaging.
- the unpolarized light-emitting portion and the polarized light-emitting portion include a plurality of unpolarized light-emitting portions and a plurality of polarized light-emitting portions that are arranged in a point symmetrical manner with the imaging element being a center.
- the unpolarized light-emitting portion and the polarized light-emitting portion include a plurality of unpolarized light-emitting portions and a plurality of polarized light-emitting portions that are arranged in an annular manner with the imaging element being a center.
- an unpolarized light-emitting portion configured to emit light having an unpolarized component
- a polarized light-emitting portion configured to emit light having a predetermined polarized component through the first polarization filter
- an imaging element configured to image a subject, which is irradiated with light by one of the unpolarized light-emitting portion and the polarized light-emitting portion, through a second polarization filter, the first polarization filter and the second polarization filter having polarization directions in an orthogonal relationship to each other, the method including by the imaging apparatus:
- a specular reflection image generator configured to:
- the specular reflection image generator includes
- a shine analyzer configured to analyze skin shine using the specular reflection image generated by the specular reflection image generator.
- the shine analyzer includes
- the shine analyzer includes a shine value calculator configured to calculate, based on the specular reflection image, an amount of sebum of an examinee.
- the shine analyzer further includes a shine evaluator configured to calculate, based on the amount of sebum calculated by the shine value calculator, a shine evaluation value for evaluating shine of an examinee.
- a result presentation unit configured to present an analysis result of shine of skin of an examinee by the shine analyzer
- the result presentation unit is configured to present, as the analysis result, a shine image and an amount of sebum of the skin of the examinee.
- a preprocessor that is provided at a previous stage of the specular reflection image generator and configured to adjust luminance levels of the unpolarized light image and the orthogonal polarized light image and supply the unpolarized light image and the orthogonal polarized light image after the adjustment to the specular reflection image generator.
- an imaging apparatus including
- a specular reflection image being an image having a specular reflection component from the unpolarized light image and the orthogonal polarized light image.
- an unpolarized light-emitting portion configured to emit light having an unpolarized component
- a polarized light-emitting portion configured to emit light having a predetermined polarized component through the first polarization filter
- an imaging element configured to image a subject, which is irradiated with light by one of the unpolarized light-emitting portion and the polarized light-emitting portion, through a second polarization filter, the first polarization filter and the second polarization filter having polarization directions in an orthogonal relationship to each other, the process including
- a specular reflection image being an image having a specular reflection component from the unpolarized light image and the orthogonal polarized light image.
- an imaging apparatus including
- a parallel polarized light image being an image having a parallel polarized component from the unpolarized light image and the orthogonal polarized light image.
- the parallel polarized light image generator includes
- a texture analyzer configured to analyze skin texture using the parallel polarized light image generated by the parallel polarized light image generator.
- a preprocessor that is provided at a previous stage of the parallel polarized light image generator and configured to adjust luminance levels of the unpolarized light image and the orthogonal polarized light image and supply the unpolarized light image and the orthogonal polarized light image after the adjustment to the parallel polarized light image generator.
- an imaging apparatus including
- generating a parallel polarized light image being an image having a parallel polarized component from the unpolarized light image and the orthogonal polarized light image.
- an unpolarized light-emitting portion configured to emit light having an unpolarized component
- a polarized light-emitting portion configured to emit light having a predetermined polarized component through the first polarization filter
- an imaging element configured to image a subject, which is irradiated with light by one of the unpolarized light-emitting portion and the polarized light-emitting portion, through a second polarization filter, the first polarization filter and the second polarization filter having polarization directions in an orthogonal relationship to each other, the process including
- generating a parallel polarized light image being an image having a parallel polarized component from the unpolarized light image and the orthogonal polarized light image.
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Veterinary Medicine (AREA)
- Heart & Thoracic Surgery (AREA)
- Public Health (AREA)
- Animal Behavior & Ethology (AREA)
- Surgery (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Biomedical Technology (AREA)
- General Physics & Mathematics (AREA)
- Radiology & Medical Imaging (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Dermatology (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
- Investigating Or Analysing Materials By Optical Means (AREA)
- Image Input (AREA)
- Studio Devices (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
- Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)
- Optics & Photonics (AREA)
Abstract
Description
- 1. First Embodiment (Imaging System That Evaluates Skin Shine)
- 2. Second Embodiment (Imaging System That Evaluates Skin Texture)
- 3. Third Embodiment (Imaging System That Evaluates Skin Shine and Texture)
- 4. Fourth Embodiment (Imaging System That Performs Image Processing at Server)
dst_val=src_val+(std_val−src_avg)
I PV P,G(x,y)=GainPP ·I PV P(x,y) (3)
I Diff1(x,y)=I T P(x,y)−I PV P,G(x,y) (4)
Δd(x,y)=I T P(x,y)−GainPP ·I PV P(x,y) (7)
I PV P,G(x,y)=GainPP ·I PV P(x,y) (9)
I Diff2(x,y)=I T P(x,y)−I PV P,G(x,y) (10)
I PP(x,y)−I Diff2(x,y)+std_val (11)
<Difference Between First Embodiment and Second Embodiment>
eval1total =Eevalsize *Eevalshape *Eevaldirection (18)
eval2total =Eevalsize *Eevalshape *Eevaldirection*ShapeRatioideal (19)
ShapeRatioideal=ShapeRatio0*ShapeRatio1 (20)
- (1) An imaging apparatus, including:
- (2) The imaging apparatus according to (1), in which
- (3) The imaging apparatus according to (1) or (2), in which
- (4) An imaging method of an imaging apparatus, the imaging apparatus including
- (5) An image processing apparatus, including
-
- acquire an unpolarized light image and an orthogonal polarized light image that are captured by an imaging apparatus including
- an unpolarized light-emitting portion configured to emit light having an unpolarized component,
- a polarized light-emitting portion configured to emit light having a predetermined polarized component through the first polarization filter, and
- an imaging element configured to image a subject, which is irradiated with light by one of the unpolarized light-emitting portion and the polarized light-emitting portion, through a second polarization filter, the first polarization filter and the second polarization filter having polarization directions in an orthogonal relationship to each other, and
- generate, from the unpolarized light image and the orthogonal polarized light image, a specular reflection image being an image having a specular reflection component.
- acquire an unpolarized light image and an orthogonal polarized light image that are captured by an imaging apparatus including
- (6) The image processing apparatus according to (5), in which
-
- a gain multiplier configured to multiply a gain with the orthogonal polarized light image,
- a subtractor configured to subtract the orthogonal polarized light image after the gain multiplication from the unpolarized light image to generate a first difference image, and
- a clipper configured to clip the first difference image to have a luminance value in a predetermined range.
- (7) The image processing apparatus according to (5) or (6), further including
- (8) The image processing apparatus according to (7), in which
-
- a shine image calculator configured to calculate, based on the specular reflection image, a shine image being an image indicating an amount of sebum of an examinee.
- (9) The image processing apparatus according to (7) or (8), in which
- (10) The image processing apparatus according to (9), in which
- (11) The image processing apparatus according to any one of (7) to (10), further including
- (12) The image processing apparatus according to any one of (5) to (11), further including:
- (13) An imaging processing method of an image processing apparatus, including:
-
- an unpolarized light-emitting portion configured to emit light having an unpolarized component,
- a polarized light-emitting portion configured to emit light having a predetermined polarized component through the first polarization filter, and
- an imaging element configured to image a subject, which is irradiated with light by one of the unpolarized light-emitting portion and the polarized light-emitting portion, through a second polarization filter, the first polarization filter and the second polarization filter having polarization directions in an orthogonal relationship to each other; and
- (14) A program that causes a computer to execute a process, the computer being configured to process an unpolarized light image and an orthogonal polarized light image that are captured by an imaging apparatus including
- (15) An image processing apparatus, including a parallel polarized light image generator configured to
-
- an unpolarized light-emitting portion configured to emit light having an unpolarized component,
- a polarized light-emitting portion configured to emit light having a predetermined polarized component through the first polarization filter, and
- an imaging element configured to image a subject, which is irradiated with light by one of the unpolarized light-emitting portion and the polarized light-emitting portion, through a second polarization filter, the first polarization filter and the second polarization filter having polarization directions in an orthogonal relationship to each other, and
- (16) The image processing apparatus according to (15), in which
-
- a gain calculator configured to calculate a gain,
- a gain multiplier configured to multiply the gain with the orthogonal polarized light image,
- a subtractor configured to subtract the orthogonal polarized light image after the gain multiplication from the unpolarized light image to generate a second difference image, and
- an offset adder configured to add a predetermined offset value to the second difference image.
- (17) The image processing apparatus according to (15) or (16), further including
- (18) The image processing apparatus according to any one of (15) to (17), further including:
- (19) An imaging processing method of an image processing apparatus, including:
-
- an unpolarized light-emitting portion configured to emit light having an unpolarized component,
- a polarized light-emitting portion configured to emit light having a predetermined polarized component through the first polarization filter, and
- an imaging element configured to image a subject, which is irradiated with light by one of the unpolarized light-emitting portion and the polarized light-emitting portion, through a second polarization filter, the first polarization filter and the second polarization filter having polarization directions in an orthogonal relationship to each other; and
- (20) A program that causes a computer to execute a process, the computer being configured to process an unpolarized light image and an orthogonal polarized light image that are captured by an imaging apparatus including
Claims (20)
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2013175770A JP6107537B2 (en) | 2013-08-27 | 2013-08-27 | Imaging system and image processing method thereof, image processing apparatus and image processing method thereof, and program |
JP2013-175770 | 2013-08-27 |
Publications (2)
Publication Number | Publication Date |
---|---|
US20150062380A1 US20150062380A1 (en) | 2015-03-05 |
US9345430B2 true US9345430B2 (en) | 2016-05-24 |
Family
ID=52582704
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/446,858 Expired - Fee Related US9345430B2 (en) | 2013-08-27 | 2014-07-30 | Imaging apparatus and imaging method thereof, image processing apparatus and image processing method thereof, and program |
Country Status (4)
Country | Link |
---|---|
US (1) | US9345430B2 (en) |
JP (1) | JP6107537B2 (en) |
KR (1) | KR102336064B1 (en) |
CN (1) | CN104414646B (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10441379B2 (en) | 2017-12-28 | 2019-10-15 | 3Gen, Inc. | Multipurpose medical illuminator with magnification |
US11395714B2 (en) | 2019-11-11 | 2022-07-26 | Dermlite Llc | Medical illuminator with variable polarization |
US11405561B2 (en) | 2017-12-27 | 2022-08-02 | Casio Computer Co., Ltd. | Imaging device and imaging method |
US11800237B2 (en) | 2017-12-27 | 2023-10-24 | Casio Computer Co., Ltd. | Imaging device and imaging method |
Families Citing this family (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6160426B2 (en) * | 2013-10-04 | 2017-07-12 | 富士ゼロックス株式会社 | Image processing apparatus and program |
US9903806B2 (en) * | 2013-12-17 | 2018-02-27 | Nanometrics Incorporated | Focusing system with filter for open or closed loop control |
JP6501868B2 (en) * | 2015-03-26 | 2019-04-17 | マクセル株式会社 | Imaging device, signal processing device, and skin diagnosis system |
BR112018076965A8 (en) * | 2016-06-27 | 2023-03-07 | Koninklijke Philips Nv | SKIN SHINE DETECTION DEVICE, SKIN SHINE DETECTION METHOD, AND COMPUTER PROGRAM |
CN109804621B (en) * | 2016-10-17 | 2020-11-17 | 索尼公司 | Image processing apparatus, image processing method, and image pickup apparatus |
CN106353952A (en) * | 2016-10-28 | 2017-01-25 | 山东鲁能智能技术有限公司 | Polarizer-containing transformer substation indoor inspection robot image acquisition system and method |
CN106725336B (en) * | 2017-01-05 | 2023-10-10 | 北京响臻科技有限公司 | Portable skin photographing device based on cross polarization |
JP6993087B2 (en) * | 2017-01-17 | 2022-02-04 | 花王株式会社 | Skin strain measurement method |
US10542928B2 (en) * | 2017-04-04 | 2020-01-28 | Casio Computer Co., Ltd. | Medical imaging device including first and second light sources having different light emission directions |
EP3384829A1 (en) * | 2017-04-05 | 2018-10-10 | Koninklijke Philips N.V. | Skin gloss measurement for quantitative estimation of skin gloss |
EP3384830A1 (en) | 2017-04-05 | 2018-10-10 | Koninklijke Philips N.V. | Skin gloss measurement which is sensor angle rotation independent |
EP3384831A1 (en) * | 2017-04-05 | 2018-10-10 | Koninklijke Philips N.V. | Skin gloss measurement using brewster's angle |
CN108784647B (en) * | 2017-04-27 | 2021-07-27 | 立特克科技股份有限公司 | Skin detection device and detection method thereof |
WO2019012858A1 (en) | 2017-07-12 | 2019-01-17 | ソニー株式会社 | Imaging device, image generation method, and imaging system |
EP3505048A1 (en) * | 2017-12-28 | 2019-07-03 | Koninklijke Philips N.V. | Optical skin sensor using optimal spectral bands to minimize the effect of probe pressure |
EP3530179A1 (en) * | 2018-02-27 | 2019-08-28 | Koninklijke Philips N.V. | Obtaining images for use in determining one or more properties of skin of a subject |
US10702160B2 (en) * | 2018-05-02 | 2020-07-07 | Canfield Scientific, Incorporated | Skin assessment using image fusion |
CN108606780A (en) * | 2018-05-15 | 2018-10-02 | 北京科莱普云技术有限公司 | Skin detecting method, device, computer equipment and storage medium |
JP7244641B2 (en) * | 2018-11-20 | 2023-03-22 | リーンエイピー・インコーポレイテッド | Sample imaging and image archiving for image comparison |
CN110633662B (en) * | 2019-09-03 | 2022-03-25 | 云南白药集团健康产品有限公司 | Image processing method, device and system |
WO2022010310A1 (en) | 2020-07-09 | 2022-01-13 | Samsung Electronics Co., Ltd. | Electronic device for acquiring image by using light-emitting module having polarizing filter and method for controlling same |
EP4101368A1 (en) * | 2021-06-09 | 2022-12-14 | Koninklijke Philips N.V. | Determining specular reflection information |
KR20230100508A (en) * | 2021-12-28 | 2023-07-05 | 삼성전자주식회사 | An electronic device for providing information on skin oil and/or moisture by using at least one image obtained using visible light and method for controlling the same |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090043363A1 (en) * | 2007-06-19 | 2009-02-12 | Astron Clinica Limited | Method and apparatus for measuring skin texture |
JP2010273737A (en) | 2009-05-26 | 2010-12-09 | Mandom Corp | Method of evaluating surface state of skin |
US20130188023A1 (en) * | 2012-01-23 | 2013-07-25 | Omnivision Technologies, Inc. | Image sensor with optical filters having alternating polarization for 3d imaging |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0835928A (en) * | 1994-07-20 | 1996-02-06 | Unitec Res Kk | Imaging apparatus |
JP3980722B2 (en) * | 1997-04-03 | 2007-09-26 | 株式会社モリテックス | CCD microscope |
JP4133248B2 (en) * | 2002-11-18 | 2008-08-13 | 株式会社コーセー | Skin gloss evaluation method |
JP2004187248A (en) * | 2002-11-29 | 2004-07-02 | Earth Green:Kk | Television camera apparatus for photographing skin |
JP2005000429A (en) * | 2003-06-12 | 2005-01-06 | Kose Corp | Method for evaluating beauty of skin |
JP4667313B2 (en) * | 2006-07-10 | 2011-04-13 | 株式会社モリテックス | Observation device with polarized illumination |
JP5238098B2 (en) * | 2010-11-30 | 2013-07-17 | パナソニック株式会社 | Image processing apparatus and method of operating image processing apparatus |
JP5990905B2 (en) * | 2011-12-19 | 2016-09-14 | ソニー株式会社 | Measuring device, measuring method, program, and recording medium |
-
2013
- 2013-08-27 JP JP2013175770A patent/JP6107537B2/en active Active
-
2014
- 2014-07-24 KR KR1020140093807A patent/KR102336064B1/en active IP Right Grant
- 2014-07-30 US US14/446,858 patent/US9345430B2/en not_active Expired - Fee Related
- 2014-08-20 CN CN201410413099.8A patent/CN104414646B/en not_active Expired - Fee Related
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090043363A1 (en) * | 2007-06-19 | 2009-02-12 | Astron Clinica Limited | Method and apparatus for measuring skin texture |
JP2010273737A (en) | 2009-05-26 | 2010-12-09 | Mandom Corp | Method of evaluating surface state of skin |
US20130188023A1 (en) * | 2012-01-23 | 2013-07-25 | Omnivision Technologies, Inc. | Image sensor with optical filters having alternating polarization for 3d imaging |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11405561B2 (en) | 2017-12-27 | 2022-08-02 | Casio Computer Co., Ltd. | Imaging device and imaging method |
US11800237B2 (en) | 2017-12-27 | 2023-10-24 | Casio Computer Co., Ltd. | Imaging device and imaging method |
US10441379B2 (en) | 2017-12-28 | 2019-10-15 | 3Gen, Inc. | Multipurpose medical illuminator with magnification |
US11395714B2 (en) | 2019-11-11 | 2022-07-26 | Dermlite Llc | Medical illuminator with variable polarization |
Also Published As
Publication number | Publication date |
---|---|
CN104414646A (en) | 2015-03-18 |
JP2015046698A (en) | 2015-03-12 |
US20150062380A1 (en) | 2015-03-05 |
JP6107537B2 (en) | 2017-04-05 |
CN104414646B (en) | 2018-12-14 |
KR102336064B1 (en) | 2021-12-07 |
KR20150024769A (en) | 2015-03-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9345430B2 (en) | Imaging apparatus and imaging method thereof, image processing apparatus and image processing method thereof, and program | |
US9990563B2 (en) | Image processing device, image processing method, program, and recording medium for detection of epidermis pattern | |
US9396531B2 (en) | Systems and methods for image and video signal measurement | |
US9524558B2 (en) | Method, system and software module for foreground extraction | |
US9916666B2 (en) | Image processing apparatus for identifying whether or not microstructure in set examination region is abnormal, image processing method, and computer-readable recording device | |
US20110317924A1 (en) | Image processing apparatus, image processing method, and image processing program | |
US20160338603A1 (en) | Signal processing device, signal processing method, and computer-readable recording medium | |
US10726532B2 (en) | Measurement of non-uniformity noise | |
CN108197546A (en) | Photo-irradiation treatment method, apparatus, computer equipment and storage medium in recognition of face | |
JP2010287948A (en) | Image processing device, method, and program | |
US20180225522A1 (en) | Ir or thermal image enhancement method based on background information for video analysis | |
US10667704B2 (en) | Apparatus and method for measuring the quality of an extracted signal | |
CN109886195B (en) | Skin identification method based on near-infrared monochromatic gray-scale image of depth camera | |
CN106570855A (en) | Method and system for quickly judging pork freshness | |
CN117011250A (en) | Defect detection method, device and storage medium | |
US20190150848A1 (en) | Image processing apparatus, operation method performed by image processing apparatus and recording medium | |
Bhelonde et al. | Flexible wound assessment system for diabetic patient using android smartphone | |
JP2020021314A (en) | Image processing system and image processing method | |
WO2018112979A1 (en) | Image processing method and apparatus, and a terminal device | |
CN113257232B (en) | Method and device for generating spectrogram, electronic equipment and storage medium | |
US20230072179A1 (en) | Temporal metrics for denoising depth image data | |
WO2021045140A1 (en) | Information processing device, pulse wave measurement system, and pulse wave measurement program | |
US20220198614A1 (en) | Image processing apparatus, image processing method, and program | |
KR20230151371A (en) | Apparatus and method for measuring image quality using statistical characteristic analysis | |
CN117115029A (en) | Infrared image edge enhancement method, device and equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SONY CORPORATION, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:NAKAMURA, YUSUKE;GOMI, SHINICHIRO;REEL/FRAME:033423/0510 Effective date: 20140717 |
|
FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20240524 |