JP2018160128A - Image processing apparatus, image processing system, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To achieve high correction precision of an image affected by an influence of external light.SOLUTION: The image processing apparatus includes: a camera 105 for imaging a face image 12 including at least an eye image; a pupil size-related processing unit for acquiring a pupil size P on the basis of the face image 12 imaged by the camera 105; an image correction unit for correcting the face image 12 imaged by the camera 105 on the basis of a pupil size ratio calculated from the pupil size P acquired by a pupil information acquisition unit; and a biological information extraction unit for acquiring biological information, such as pulse and blood pressure, from a skin image 13 in the corrected face image 12.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique of an image processing apparatus, an image processing system, and an image processing method for correcting an image affected by external light.

With the automatic driving of vehicles and the improvement of safety technology, driver state detection technology is required.
In other words, when switching from automatic operation to manual operation, it is necessary to detect the health condition of the driver.
Various institutions such as universities and companies are studying state monitoring technology using cameras and sensors. Here, the monitored states are heartbeat, posture, sleepiness, and the like.

  In addition, as a technique for detecting the state of a person without contact, a technique using image processing with a camera image or a Doppler sensor has been reported. In these techniques, it is possible to acquire human biological information (heartbeat, respiration, pulse wave, blood pressure, etc.) from a human image captured by a camera.

  Patent Document 1 states that “the pulse wave detection device 1 color-converts a moving image frame image from an RGB component to a YIQ component, and uses the eye color of the user prepared in advance as the Q component, Then, the pulse wave detection device 1 detects the brightness of the imaging environment based on the Y value of the eye portion, and then the pulse wave detection device 1 averages the Q value of the skin portion of the frame image. From this, the pulse wave signal Qm is detected, and further, the amount of change in brightness is corrected by subtracting the average value Ye of the Y values of the eyes from this, and the Qm after brightness correction is output. A pulse wave detection device and a pulse wave detection program that can detect a pulse wave satisfactorily even when the brightness changes due to movement by a vehicle or the like are disclosed (see summary).

JP 2006-193021 A

However, the above-described state monitoring technique is not widespread due to determination accuracy, a feeling of burden associated with equipment mounting, and the like.
In particular, there is a problem that biological information cannot be measured correctly due to the influence of external light. That is, since the above-described biometric information is calculated from fluctuations in the specific color component of the face image, it is difficult to distinguish whether the face color of the person has changed or the external environment has changed if the way the light strikes is different. This is an obstacle to improving measurement accuracy.

  The technique described in Patent Document 1 performs skin brightness adjustment using luminance information (Y component of the YIQ color system) of human pupils in an image. However, it cannot be used in dark places or when using sunglasses.

  The present invention has been made in view of such a background, and an object of the present invention is to improve the correction accuracy of an image affected by external light.

In order to solve the above-described problem, the present invention provides an imaging unit that captures an image including at least an eye image, and pupil information that acquires information related to the size of the pupil based on the image captured by the imaging unit. An acquisition unit and an image correction unit that corrects the image captured by the imaging unit based on information about the size of the pupil acquired by the pupil information acquisition unit.
Other solutions will be described as appropriate in the embodiments.

  According to the present invention, it is possible to improve the correction accuracy of an image affected by external light.

It is a figure which shows the outline | summary of the image correction process performed in this embodiment. It is a figure which shows the example of the density | concentration conversion map used by 1st Embodiment. It is a figure which shows the example of the contrast conversion map used by 1 embodiment. It is a figure which shows the example of an image conversion in case an external environment is dark (P / Pr> 1). It is a figure which shows the example of an image conversion in case an external environment is bright (P / Pr <1). 1 is a diagram illustrating a configuration example of an image processing apparatus 1 according to a first embodiment. It is an example of the face image 12 with a shadow on the eyes. It is a figure which shows the detailed structure of the process part 100 used by 1st Embodiment. It is a figure which shows the detailed structure of the image process part 110 used by 1st Embodiment. It is a flowchart which shows the procedure of the image correction process performed in 1st Embodiment. It is a figure which shows the detailed structure of the process part 100a used by 2nd Embodiment. It is a flowchart which shows the procedure of the image correction process performed in 2nd Embodiment. It is a figure which shows the structural example of the abnormality detection system Za used by 3rd Embodiment. It is a figure which shows the structural example of the abnormality detection apparatus 51 used by 3rd Embodiment. It is a flowchart which shows the procedure of the abnormality detection process performed in 3rd Embodiment. It is a figure which shows the structure of the stress environment management system Zb used by 4th Embodiment. It is a figure which shows the structural example of the abnormality detection apparatus 51a used by 4th Embodiment. These are the flowcharts which show the procedure of the abnormality process performed in 4th Embodiment. It is a figure which shows the structure of the home monitoring apparatus 61 used by 5th Embodiment.

  Next, modes for carrying out the present invention (referred to as “embodiments”) will be described in detail with reference to the drawings as appropriate. In addition, in each figure, the same code | symbol is attached | subjected about the same structure and description is abbreviate | omitted.

[First Embodiment]
<Process overview>
FIG. 1 is a diagram showing an outline of image correction processing performed in the present embodiment.
First, the face image 12 is captured by the camera (imaging unit) 105. Thereafter, the pupil size P is extracted from the pupil image 11 in the face image 12.
Based on the pupil size P, the density of the face image 12 captured by the camera 105 is corrected. Then, based on the skin image 13 portion of the corrected face image 12, biological information such as blood pressure and pulse is extracted. The density indicates the color density. Instead of density, brightness, brightness, or the like may be used.
The camera 105 may be a general visible light camera, but by using the infrared camera (imaging unit) 106, the pupil size P can be acquired at night or when wearing sunglasses.
The black eye size K1 and the distance K2 from the top of the eye to the outside of the eye will be described later.

<Image correction>
Next, image correction using the pupil size P will be described with reference to FIGS.
(Concentration conversion map)
FIG. 2 is a diagram illustrating an example of a density conversion map used in the first embodiment.
In FIG. 2, the horizontal axis represents the input density, and the vertical axis represents the output density. Each density is 256 tone.
The density is darker as the value is smaller, and brighter as the value is larger.
With the density conversion map as shown in FIG. 2, the density is converted according to the pupil size ratio P / Pr. Note that P is a current value of the pupil size, and Pr is a reference pupil size (reference pupil size). The reference pupil size is the size of the pupil at a predetermined brightness. The pupil size may be the diameter of the pupil or the area of the pupil.

  As shown in FIG. 2, when the pupil size ratio P / Pr = 1, the conversion line (conversion information) 21 is arranged on the diagonal line of the density conversion map. This means that a certain density is converted to the same density and output.

Next, the case where the pupil size ratio P / Pr> 1 will be described. The case of the pupil size ratio P / Pr> 1 is a case where the current pupil size P is larger than the reference pupil size Pr and a case where the external environment is dark.
The conversion line (conversion information) 22 in such a case is arranged below the conversion line 21. This conversion line 22 means that the density 63 to 255 is converted to the density 0 to 255. That is, only a bright part is extracted from the image. As a result, the entire image is converted to be bright.

A case where the pupil size ratio P / Pr <1 will be described. The case of the pupil size ratio P / Pr <1 is a case where the current pupil size P is smaller than the reference pupil size Pr and a case where the external environment is bright.
The conversion line (conversion information) 23 in such a case is disposed above the conversion line 21. This conversion line 23 means that the density 0 to 157 is converted to the density 0 to 255. That is, only a dark part is extracted from the image. As a result, the entire image is converted to be dark.
Such a density conversion map is created and set in advance.

  Incidentally, the black eye size K1 shown in FIG. 1 may be used instead of the reference pupil size Pr. In addition to this, it is possible to use one having a constant size and distance, such as using a distance K2 from the top of the eye to the outside of the eye instead of the reference pupil size Pr. By doing so, appropriate correction can be performed even if a change occurs in the pupil size P in the face image 12 due to the movement of the face. Further, by using the black eye size K1 or the distance K2 from the eye head to the corner of the eye instead of the reference pupil size Pr, it is not necessary to set the reference pupil size Pr in advance. Thereby, a user's burden can be reduced.

In a state where the pupil size ratio P / Pr> 1, as the pupil size ratio P / Pr increases, the conversion line 21 moves in the direction of the conversion line 22 and the inclination thereof increases.
Similarly, in a state where the pupil size ratio P / Pr <1, the smaller the pupil size ratio P / Pr, the more the conversion line 21 moves in the direction of the conversion line 23 and the inclination thereof increases.

(Contrast conversion map)
FIG. 3 is a diagram illustrating an example of a contrast conversion map used in the first embodiment.
In FIG. 3, the horizontal axis represents the input density, and the vertical axis represents the output density. Each density is 256 tone.
A contrast conversion curve 31 is set as shown in FIG. The contrast conversion curve 31 converts the density of the image so that the dark part becomes darker and the bright part becomes brighter.
This increases the contrast of the image.
Note that image conversion based on the contrast conversion map in FIG. 3 is performed by general image processing.

(Actual image conversion: When the external environment is dark)
FIG. 4 is a diagram illustrating an example of image conversion when the external environment is dark (P / Pr> 1).
FIG. 4 shows a density histogram of the skin image 13 (see FIG. 1), where the horizontal axis indicates density and the vertical axis indicates the number of pixels. The density histograms in FIGS. 4 and 5 are green light density histograms.
The upper part of FIG. 4 shows a density histogram before conversion, and the lower part shows a density histogram after conversion. The conversion includes conversion using the density conversion map of FIG. 2 and conversion using the contrast conversion map of FIG.
In particular, according to the density conversion map of FIG. 2, the density range 41 before conversion (upper stage) is expanded from 0 to 255 after conversion (lower stage). That is, a bright part is extracted. This brightens the entire image.

(Actual image conversion: When the external environment is bright)
FIG. 5 is a diagram illustrating an example of image conversion when the external environment is bright (P / Pr <1).
FIG. 5 shows a density histogram of the skin image 13, where the horizontal axis indicates the density and the vertical axis indicates the number of pixels.
The upper part of FIG. 5 shows a density histogram before conversion, and the lower part shows a density histogram after conversion. The conversion includes conversion using the density conversion map of FIG. 2 and conversion using the contrast conversion map of FIG.
In particular, according to the density conversion map of FIG. 2, the density range 42 before conversion (upper stage) is expanded from 0 to 255 after conversion (lower stage). That is, a dark part is extracted. This darkens the entire image.

Thus, by using the density conversion map shown in FIG. 2, when the external environment is dark, that is, when the image is dark, the entire image is brightened. When the external environment is bright, that is, when the image is bright, the entire image is darkened.
In this way, the brightness of the image can be made appropriate.

  By using the density conversion map in this manner, the density correction of the face image 12 is facilitated.

<Image processing apparatus 1>
FIG. 6 is a diagram illustrating a configuration example of the image processing apparatus 1 according to the first embodiment. Reference is made to FIG. 1 as appropriate.
As illustrated in FIG. 6, the image processing apparatus 1 includes a memory 101, a CPU (Central Processing Unit) 102, and a transmission / reception apparatus 103. The image processing apparatus 1 includes a camera 105 (infrared camera 106) and a storage device 104.
The memory 101 is a ROM (Read Only Memory) or the like, and the storage device 104 is a RAM (Random Access Memory) or the like.
A program is stored in the memory 101, and this program is executed by the CPU 102. As a result, the processing unit 100 and the units 100 to 112, 120 to 122, 131 (described later in FIGS. 8 and 9) constituting the processing unit 100 are embodied.

The camera 105 is a camera for capturing the face image 12. Further, as described above, by using the infrared camera 106 as the camera 105, the pupil size P can be acquired at night, in a dark place such as a tunnel, or when the user wears sunglasses. .
Note that the pupil image 11 may be acquired by the infrared camera 106 in a dark place, and the pupil image 11 may be acquired by a visible light camera in a bright place.

  FIG. 7 will be described later.

<Processing unit 100>
FIG. 8 is a diagram illustrating a detailed configuration of the processing unit 100 used in the first embodiment. Reference is made to FIG. 1 as appropriate.
The processing unit 100 includes a pupil size related processing unit (pupil information acquisition unit) 120, an image processing unit 110, and a biological information extraction processing unit 131.
The pupil size related processing unit 120 performs processing related to the pupil size P, and includes an validity determination processing unit 121 and a pupil size ratio calculation processing unit 122.
The validity determination processing unit 121 determines whether or not the face image 12 captured by the camera 105 is suitable (effective) for calculating the pupil size ratio P / Pr. Details of the determination by the validity determination processing unit 121 will be described later.
The pupil size ratio calculation processing unit 122 calculates the pupil size ratio P / Pr from the pupil image 11 in the face image 12 captured by the camera 105.

Based on the pupil size ratio P / Pr calculated by the pupil size ratio calculation processing unit 122, the image processing unit 110 corrects the density of the face image 12 captured by the camera 105 using the method described with reference to FIGS. Is done. Details of the image processing unit 110 will be described later.
The biological information extraction processing unit 131 extracts biological information such as a pulse value and a blood pressure value from the skin image 13 in the corrected face image 12.

<Image processing unit 110>
FIG. 9 is a diagram illustrating a detailed configuration of the image processing unit 110 used in the first embodiment. Reference is made to FIGS. 1 and 8 as appropriate.
The image processing unit 110 includes a density histogram generation unit 111 and an image correction unit 112.
The density histogram generation unit 111 generates a density histogram of the face image 12 captured by the camera 105.
The image correction unit 112 is shown in FIGS. 2, 4, and 5 based on the pupil size ratio P / Pr calculated by the pupil size related processing unit 120 (pupil size ratio calculation processing unit 122 (see FIG. 8)). The density histogram is converted using a technique. Thereafter, the image correction unit 112 corrects the density of the face image 12 based on the converted density histogram. Further, the image correction unit 112 appropriately corrects contrast, sharpness, and the like.

(flowchart)
FIG. 10 is a flowchart illustrating a procedure of image correction processing performed in the first embodiment. Reference is made to FIGS. 1, 6, 8, and 9 as appropriate.
The process of FIG. 10 is started when the power of the image processing apparatus 1 is turned on, and is repeated until it is turned off.
First, the camera 105 captures the user's face image 12.
Then, the image processing unit 110 determines whether sunglasses are worn based on the captured face image 12 (S101). Whether or not sunglasses are worn can be determined by a general image recognition process.
As a result of step S101, when sunglasses are worn (S101 → Yes), the pupil size related processing unit 120 acquires the pupil size from the pupil image 11 taken in the previous step S101. Then, the pupil size related processing unit 120 sets the pupil size as the reference pupil size Pr (S102).
Thereafter, the processing unit 100 advances the processing to step S111.

As a result of step S101, when sunglasses are not worn (S101 → No), the camera 105 captures the user's face image 12 (S111).
Thereafter, the validity determination processing unit 121 extracts validity determination information from the captured face image 12 (S112). The validity determination information is information on the location of the shadow portion in the face image 12, information on the eyelid state, and information on the light amount change. The light amount change is a light amount change compared to the previous face image 12. Therefore, the light quantity change is not extracted in the first process after the process of FIG. 10 is started.

Then, the validity determination processing unit 121 determines whether or not the face image 12 captured in step S111 is valid based on the validity determination information extracted in step S112 (S113).
The validity determination processing unit 121 determines that the face image 12 is not valid (S113 → No) when at least one of the following conditions (A1) to (A3) is satisfied.
(A1) As shown in FIG. 7, the shadow is on the eyes and the other parts are not shadowed. This is because the pupil size P cannot be measured accurately when a shadow is on the eyes. That is, when the shadow is on the eyes, the amount of light striking the skin and the pupil size P cannot be associated with each other. When the shadow is in the eyes, the accuracy of measurement of the pupil size P can be improved by preventing the processing from being performed.
(A2) The eyelid is closed. This is because the pupil size P cannot be measured if the eyelid is closed.
(A3) The change in the light amount is a predetermined value or more. This is because if the change in the amount of light is large, the reduction / enlargement of the pupil of the amount of light cannot catch up with the change in the amount of light and an appropriate pupil size cannot be acquired.

When the face image 12 is not valid as a result of step S113 (S113 → No), the processing unit 100 returns the process to step S111.
If the result of step S113 is that the face image 12 is valid (S113 → Yes), the pupil size related processing unit 120 extracts the pupil size P from the face image 12 (S121). At this time, the pupil size ratio calculation processing unit 122 may extract one (for example, the left eye) pupil size P, or may be an average of the pupil sizes of both eyes as the pupil size P.

Next, the pupil size ratio calculation processing unit 122 calculates the pupil size ratio P / Pr (S122).
Subsequently, the density histogram generation unit 111 generates a density histogram of the face image 12 (S123).
Then, the image correction unit 112 corrects the density histogram by the method described with reference to FIGS. 2 to 5 based on the calculated pupil size ratio P / Pr (S124).
Then, the image correction unit 112 corrects the face image 12 based on the density histogram corrected in step S124 (S125).

The biological information extraction processing unit 131 acquires biological information such as blood pressure and pulse rate based on the corrected face image 12 (S131). The biological information extraction processing unit 131 may perform extraction of biological information by the method described in Patent Document 1. In the method described in Patent Document 1, biological information such as blood pressure and pulse is extracted from temporal fluctuations in a specific wavelength region (such as green) in the skin image 13. In addition, biological information may be acquired by detecting a pulse wave or the like from a change in wavelength spectrum.
When the camera 105 is an infrared camera, it is possible to extract biological information such as blood pressure and pulse from time variation in the infrared region of the skin image 13.
Further, when an infrared camera and a visible light camera are used as the camera 105, even if biometric information is extracted from the visible light skin image 13 illuminated by the light of another car or the like by the method described above. Good.
The biometric information extraction processing unit 131 stores the extracted biometric information in the storage device 104 (S132), and the processing unit 100 returns the process to step S301.

According to 1st Embodiment, the image processing apparatus 1 correct | amends the face image 12 (skin image 13) imaged with the camera 105 based on the information regarding the size of a pupil. As a result, highly accurate image correction can be performed. In particular, image correction can be performed even when the surroundings are dark or when the user is wearing sunglasses. As a result, the extraction accuracy of biological information can be improved.
Even when the surroundings are dark or when sunglasses are worn, the pupil reacts to the surrounding brightness.
Since the technique described in Patent Document 1 uses the color of the pupil, it cannot be used when the surroundings are dark or when sunglasses are worn.

Moreover, in 1st Embodiment, the extraction accuracy of biometric information can be improved by correcting the density of the skin color.
In the first embodiment, the pulse and blood pressure can be obtained with high accuracy by extracting the pulse and blood pressure based on the corrected image.

  In the present embodiment, the density histogram is flattened (see FIGS. 4 and 5). The image correction unit 112 can sharpen the face image 12 by performing an unsharp mask process on the face image 12 whose density is flattened.

Alternatively, the brightness of the environment and the size of the pupil in the past may be stored, and if the pupil is large relative to the brightness of the environment, it may be determined that the sympathetic nerve is working. Conversely, when the pupil is small relative to the brightness of the environment, it may be determined that the parasympathetic nerve is working.
Further, when the contraction speed of the pupil is larger than the average value so far by a predetermined value or more, it may be determined that the parasympathetic nerve is working. On the contrary, when the dilation speed of the pupil is larger than the average value so far by a predetermined value or more, it may be determined that the sympathetic nerve is working.

[Second Embodiment]
<Processing unit 100a>
FIG. 11 is a diagram illustrating a detailed configuration of the processing unit 100a used in the second embodiment.
In FIG. 11, the same components as those in FIG.
The processing unit 100a illustrated in FIG. 11 is different from the processing unit 100 illustrated in FIG. 8 in that a skin information processing unit 132 is included.
The skin information processing unit 132 calculates the difference (base difference value) between the skin information base value stored in advance in the storage device 104 (see FIG. 6) and the skin information in the captured skin image 13. Then, the skin information processing unit 132 passes the calculated base difference value to the image processing unit 110.

<Flowchart>
FIG. 12 is a flowchart illustrating a procedure of image correction processing performed in the second embodiment.
In FIG. 12, the same processes as those in FIG. 10 are denoted by the same step numbers and description thereof is omitted.
After the pupil size ratio P / Pr is calculated in step S122, the skin information processing unit 132 extracts skin information (S201). Skin information is information about skin color. Specifically, each RGB value of skin is extracted.

Next, the skin information processing unit 132 calculates a difference (base difference value) between the skin information base value acquired in advance and the skin information in the face image 12 captured in step S111 (S202).
Then, the image correcting unit 112 corrects the density histogram based on the pupil size ratio P / Pr and the base difference value of the skin information calculated in step S102 (S124a).
By performing the following processing, the density histogram is corrected based on the difference from the skin information base value.
For example, when the R (Red) value is small, the skin information processing unit 132 calculates a base difference value for the R value. Then, the image correction unit 112 increases the density of the base difference value by the R value of the face image 12 (skin image 13).
After step S125, the same processing as that of FIG. 10 is performed.

According to the second embodiment, the density of the face image 12 (skin image 13) can be made appropriate even when anemia is caused and the skin is white.
Moreover, according to 2nd Embodiment, the influence by the difference in the color of the skin by an individual difference can be suppressed.

[Third Embodiment]
<System outline>
FIG. 13 is a diagram illustrating a configuration example of an abnormality detection system Za used in the third embodiment.
The abnormality detection system (image processing system) Za shown in the third embodiment performs the following processing. That is, when the abnormality detection device 51 mounted on the vehicle 52 detects a driver abnormality, the management company server 53 and the insurance company server 54 are notified of the abnormality being detected via the cloud service 55.
The management company server 53 is a server installed in the management company. The management company is a company that, when notified of a driver's abnormality detection, calls an operator or services an ambulance.
The insurance company server 54 is a server installed in the insurance company.

<Abnormality detection device 51>
FIG. 14 is a diagram illustrating a configuration example of the abnormality detection device 51 used in the third embodiment.
14, the same components as those in FIG. 6 are denoted by the same reference numerals as those in FIG.
The abnormality detection device 51 includes the image processing device 1b and the vehicle device 2. Here, the image processing device 1b and the vehicle device 2 are connected to each other, but the image processing device 1b may be mounted on the vehicle device 2.

(Image processing apparatus 1b)
The image processing apparatus 1b is different from FIG. 6 in that an abnormality detection processing unit 141 that detects an abnormality of the driver is executed by the CPU 102 based on the biological information extracted by the processing unit 100. The processing performed by the abnormality detection processing unit 141 will be described later. Note that the abnormality detection processing unit 141 may be mounted on the vehicle device 2.

(Vehicle equipment 2)
The vehicle device 2 is a device that is normally mounted on the vehicle 52 (see FIG. 13). The vehicle device 2 includes a CPU 201, a speaker 202, and a display device 203. The vehicle device 2 includes a clock 204, a GPS (Global Positioning System) signal reception device 205, a gyro sensor 206, a vehicle control device 207, and a transmission / reception device 208.
Among these, the GPS signal receiving device 205 receives a GPS signal transmitted from a GPS artificial satellite (not shown).
The gyro sensor 206 detects the attitude of the vehicle 52.
When the image processing apparatus 1b detects an abnormality of the driver, the vehicle control device 207 stops the vehicle 52 at a safe place by autonomous driving.

<Flowchart>
FIG. 15 is a flowchart illustrating a procedure of abnormality detection processing performed in the third embodiment. Reference is made to FIGS. 13 and 14 as appropriate.
The abnormality detection processing unit 141 calculates the average value of the past N biological values (S301). The biological value is a blood pressure value, a pulse value, or the like included in the biological information.
Next, the processing unit 100 performs the processing of FIG. 10 or FIG. 12 to acquire biological information (S302).
Then, the abnormality detection processing unit 141 determines whether or not the biological value is normal (S303). Whether or not the biological value is normal is whether or not the biological value acquired in step S302 is a predetermined value or more away from step S301. The predetermined value is, for example, 2σ (σ is a standard deviation).

If the result of step S303 is normal (S303 → Yes), the abnormality detection processing unit 141 returns the process to step S301.
If the result of step S303 is not normal (S303 → No), the abnormality detection processing unit 141 stores the living body obtained in step S302 as an NG value in the storage device 104 (S311).
Then, the abnormality detection processing unit 141 determines whether it is N times consecutively determined as not normal in step S303 (S312).
As a result of step S312, when it is determined that it is not normal in step S303 for N consecutive times (S312 → No), the abnormality detection processing unit 141 returns the process to step S301.

As a result of step S312, when it is determined that it is not normal in step S303 for N consecutive times (S312 → Yes), the abnormality detection processing unit 141 notifies the vehicle device 2 that the state of the driver is abnormal. .
The CPU 201 of the vehicle device 2 causes the display device 203 to output (warning output) that the driver is in an abnormal state (S321). The output may be a call from the speaker 202 or the like.
Further, the CPU 201 causes the vehicle control device 207 to perform emergency stop processing on the vehicle 52 (S322). The emergency stop process is a process of stopping after moving to the road shoulder or the like.
Then, the CPU 201 transmits abnormality occurrence information to the management company server 53 and the insurance company server 54 via the cloud via the transmission / reception device 208 (S323).
Thereafter, the status is checked by the operator of the management company (S324). The state confirmation by the operator means that the operator talks to the driver via a video phone or the like.

  In addition, the insurance company reviews insurance settings based on the information on the occurrence of abnormalities.

According to the third embodiment, the accuracy of abnormality detection can be improved by using the image processing apparatus 1b. In step S312, when the image processing apparatus 1b continues N times, it can prevent abnormality detection from being output when biological information is accidentally abnormal by outputting abnormality detection.
In addition, by mounting the image processing apparatus 1b on the vehicle 52, it is possible to monitor the health condition of the driver.

[Fourth Embodiment]
<Stress environment management system Zb>
FIG. 16 is a diagram showing a configuration of a stress environment management system Zb used in the fourth embodiment.
The stress environment management system (image processing system) Zb collects stress information detected by the abnormality detection device 51a mounted on each of the plurality of vehicles 52 in the management company server 53 and the insurance company server 54 via the cloud service 55. It is done.
Here, the stress information is associated with the position information, and the management company and the insurance company can grasp where many people feel stress. Such information is used for setting warning intervals and insurance premiums.

<Abnormality detection device 51a>
FIG. 17 is a diagram illustrating a configuration example of an abnormality detection device 51a used in the fourth embodiment.
In FIG. 17, the same components as those in FIG. 14 are denoted by the same reference numerals as those in FIG.
The abnormality detection device 51a includes an image processing device 1c and a vehicle device 2. Here, the image processing device 1c and the vehicle device 2 are connected to each other, but the image processing device 1c may be mounted on the vehicle device 2.

(Image processing apparatus 1c)
The image processing apparatus 1 c is different from FIG. 6 in that the stress level processing unit 142 and the external environment processing unit 143 are executed by the CPU 102. Further, the image processing apparatus 1c is different from FIG. 14 in that the CPU 102 is provided with a stress degree processing unit 142 and an external environment processing unit 143 instead of the abnormality detection processing unit 141 (see FIG. 14).
Based on the biological information extracted by the processing unit 100, the stress level processing unit 142 calculates a stress level such as a driver's fatigue level and tension level.
The external environment processing unit 143 detects the position of the vehicle 52 when the stress level processing unit 142 calculates the stress level.
The processing performed by the stress level processing unit 142 and the external environment processing unit 143 will be described later.
Moreover, since the vehicle apparatus 2 is the same as that shown in FIG. 14, description here is abbreviate | omitted.
The stress level processing unit 142 and the external environment processing unit 143 may be mounted on the vehicle device 2.

<Flowchart>
FIG. 18 is a flowchart illustrating the procedure of the abnormality process performed in the fourth embodiment.
First, biometric information is acquired by performing the processing of FIG. 10 or FIG. 12 (S401).
Next, the external environment processing unit 143 specifies the current information of the vehicle 52 together with the GPS signal received by the GPS signal receiving device 205, the gyro signal received from the gyro sensor 206, and the like (S402). The current information is information including the current time, the current position of the vehicle 52, the direction of the vehicle 52, and the like. Note that the current position and orientation of the vehicle 52 are the current position and orientation of the person imaged by the camera 105.
Then, the stress degree processing unit 142 determines whether or not the processes of Step S401 and Step S402 have been performed M times or more (S403).
As a result of step S403, when the processes of step S401 and step S402 have not been performed M times or more (S403 → No), the stress degree processing unit 142 returns the process to step S401.

As a result of step S403, when the processes of step S401 and step S402 are performed M times or more (S403 → Yes), the stress level processing unit 142 calculates the stress level (S411). The stress level is calculated from the accumulated biometric value based on the fatigue level, the tension level, and the like. The calculation of the stress level assumes that the driver feels a strong stress when the pulse and / or blood pressure exceeds a predetermined value.
Then, the external environment processing unit 143 specifies an external environment with high fatigue and tension by matching the current position specified in step S402 with the map data (S412).
Thereafter, the stress level processing unit 142 transmits information related to the external environment (external environment information) identified in step S412 and the stress level to the management company server 53 and the insurance company server 54 via the cloud (S413). This transmission may be performed by the transmission / reception device 103 of the image processing device 1c or the transmission / reception device 208 of the vehicle device 2.
And the management company server 53 outputs the external environment which tends to have high fatigue and tension as an information sharing report based on the fatigue and high external environment collected from other vehicles 52 (S414).

For example, the management company collects from the abnormality detection devices 51 a mounted on the plurality of vehicles 52. Then, the management company specifies a section where many drivers feel strong stress based on the collected stress level and the location information where the stress is felt. Then, the management company distributes this section to the driver, and can thereby promote alertness to the section where stress is strongly felt.
In addition, insurance companies set high insurance premiums for drivers who run a lot in stress-sensitive sections.
Thus, according to the fourth embodiment, it is possible to specify a section where stress is felt.

[Fifth Embodiment]
<Home monitoring device 61>
FIG. 19 is a diagram showing a configuration of a home monitoring device 61 used in the fifth embodiment.
In FIG. 19, the same components as those in FIG.
The at-home monitoring device (image processing system) 61 includes an image processing device 1 b and a home appliance 3. Here, the image processing device 1b and the home appliance 3 are connected, but the home appliance 3 may have a format in which the image processing device 1b is mounted.
Here, since the image processing apparatus 1b is the same as that shown in FIG. 14, the description thereof is omitted here.

(Household appliances 3)
The home appliance 3 is a home appliance such as an air conditioner or an interphone.
The home appliance 3 includes a memory 301, a CPU 302, a camera 303, a clock 304, a speaker 305, a display device 306, a device control device 307, and a transmission / reception device 308. Note that the household electrical appliance 3 may not include all of the camera 303, the clock 304, the speaker 305, and the display device 306. When the home appliance 3 is provided with the camera 303, the camera 105 (infrared camera 106) of the image processing apparatus 1b may be omitted.

Note that the processing of the home monitoring device 61 of the fifth embodiment is the processing in FIG. 15 in which the processing in step S322 is omitted and the processing in the vehicle device 2 (see FIG. 14) is replaced with the home appliance 3. The description here is omitted.
When the image processing apparatus 1b detects an abnormality, the home appliance 3 performs notification to the management company.

  According to the fifth embodiment, it is possible to monitor the health of the resident using the home appliance 3. Further, for example, if the home appliance 3 is air-conditioned, the device control device 307 may perform processing such as temperature adjustment when the image processing device 1b detects an abnormality.

The biological information may include a respiratory state.
Note that the correction is preferably performed every predetermined time. For example, when the correction information is obtained in step S125 of FIG. 10, the image correction unit 112 continues to apply the obtained correction information to the face image 12 acquired thereafter. When new correction information is obtained after a predetermined time has elapsed, the image correction unit 112 continues to apply the obtained correction information to the face image 12 acquired thereafter.

  In the present embodiment, the pupil size ratio P / Pr is used as information regarding the size of the pupil, but the present invention is not limited to this. For example, Pr / P may be used, or the current pupil size P may be used. Further, P (t) −P (t−1) may be used as information regarding the size of the pupil. Here, P (t) is the size of the pupil at time t.

  Further, in step S113 of FIG. 10 and FIG. 12, the validity determination processing unit 121 determines whether or not the face image 12 acquired by “whether or not the eyelid is closed” is valid. In addition to this, The following determination may be made. That is, the validity determination processing unit 121 may determine whether or not the driver (user) is drowsy by determining whether or not the eyelid remains closed for a predetermined time. When it is determined that the driver (user) is drowsy, processing such as increasing the volume of the speaker 202 or the like may be performed.

  The present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail for easy understanding of the present invention, and is not necessarily limited to having all the configurations described. In addition, a part of the configuration of a certain embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of a certain embodiment. Moreover, it is possible to add / delete / replace other configurations for a part of the configurations of the embodiments.

Each of the above-described configurations, functions, units 1 to 3, storage unit 5, etc. may be realized by hardware by designing a part or all of them, for example, with an integrated circuit. Further, as shown in FIG. 4, the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by a processor such as the CPU 102. In addition to storing information such as programs, tables, and files for realizing each function in an HD (Hard Disk), a memory 101, a recording device such as an SSD (Solid State Drive), or an IC (Integrated Circuit) card It can also be stored in a recording medium such as an SD (Secure Digital) card or a DVD (Digital Versatile Disc).
In each embodiment, control lines and information lines are those that are considered necessary for explanation, and not all control lines and information lines are necessarily shown on the product. In practice, it can be considered that almost all configurations are connected to each other.

1, 1b, 1c Image processing device 2 Vehicle device 3 Home appliance 11 Pupil image 12 Face image 13 Skin image 21-23 Conversion line (conversion information)
51, 51a Anomaly detection device 52 Vehicle 53 Management company server 54 Insurance company server 55 Cloud service 61 Home monitoring device (image processing system)
DESCRIPTION OF SYMBOLS 100 Processing part 104 Storage device 105,303 Camera (imaging part)
106 Infrared camera (imaging part)
110 Image processing unit 111 Density histogram generation unit 112 Image correction unit 120 Pupil size related processing unit (pupil information acquisition unit)
121 Pupil Size Ratio Calculation Processing Unit 122 Validity Determination Processing Unit 131 Biometric Information Extraction Unit 132 Skin Information Processing Unit 202, 305 Speaker 203, 306 Display Device 205 GPS Signal Receiving Device 206 Gyro Sensor 207 Vehicle Control Device 307 Device Control Device Za, Zb Anomaly detection system (image processing system)

Claims (15)

An imaging unit that captures an image including at least an eye image;
Based on the image captured by the imaging unit, a pupil information acquisition unit that acquires information related to the size of the pupil;
An image correction unit that corrects the image captured by the imaging unit based on information on the size of the pupil acquired by the pupil information acquisition unit;
An image processing apparatus comprising: The image includes a skin image,
The image processing apparatus according to claim 1, further comprising: a biological information extraction unit that extracts biological information based on the corrected skin image. Information relating to the size of the pupil and pixel density conversion information are associated with each other and stored in the storage unit,
The image correction unit
The pixel density of the image is converted based on the information on the pupil size and the pixel density conversion information based on the acquired information on the pupil size. The image processing apparatus according to 1. Information regarding the density of the skin color acquired in advance is stored in the storage unit,
The image processing apparatus according to claim 1, wherein the density of the image is corrected based on information on the skin color density and information on the skin density of the image. The image processing apparatus according to claim 1, wherein when the eyelid is closed based on the image, the pupil information acquisition unit and the image correction unit do not perform processing. A storage unit for storing the biological information detected in the past;
An abnormality detection unit that detects an abnormality of the biological information by comparing the past biological information and the acquired biological information;
The image processing apparatus according to claim 2, further comprising: The abnormality detection unit
The image processing apparatus according to claim 6, wherein when the abnormality of the biological information is detected continuously for a predetermined number of times, the detection of the abnormality is output. A position information acquisition unit that acquires position information of a person to be imaged by the imaging unit;
A position storage unit that stores the position information when the abnormality is detected in the storage unit;
The image processing apparatus according to claim 6. The image processing apparatus according to claim 1, wherein the imaging unit is an infrared camera. The image processing apparatus according to claim 2, wherein the biological information includes at least a pulse and a blood pressure. An imaging unit that captures an image including at least an eye image;
Based on the image captured by the imaging unit, a pupil information acquisition unit that acquires information related to the size of the pupil;
An image correction unit that corrects the image captured by the imaging unit based on information on the size of the pupil acquired by the pupil information acquisition unit;
An image processing system comprising: The image includes a skin image,
A biometric information extracting unit that extracts biometric information based on the corrected skin image;
A storage unit for storing the biological information detected in the past;
An abnormality detection unit that detects an abnormality of the biological information by comparing the past biological information and the acquired biological information;
A notification unit for notifying the detected abnormality;
The image processing system according to claim 11, further comprising: The image processing system according to claim 11, wherein the imaging unit is provided in a vehicle. The image processing system according to claim 11, wherein the imaging unit is provided in a home appliance. An image processing apparatus having an imaging unit that captures an image including at least an eye image,
Based on the image captured by the imaging unit, obtain information on the size of the pupil,
An image processing method comprising: correcting the image captured by the imaging unit based on the acquired information regarding the size of the pupil.

Images