WO2023053769A1

WO2023053769A1 - Image processing device, imaging device, camera system, image processing method, and program

Info

Publication number: WO2023053769A1
Application number: PCT/JP2022/031320
Authority: WO
Inventors: 哲也藤川; 智大島田; 臣一下津; 智行河合
Original assignee: 富士フイルム株式会社
Priority date: 2021-09-29
Filing date: 2022-08-19
Publication date: 2023-04-06

Abstract

Provided are an image processing device, an imaging device, a camera system, an image processing method, and a program that are capable of accurately identifying a defective region. An image processing device (30) is provided with a processor for identifying a defective part of a structure, wherein the processor: calculates a first evaluation value relating to a change in brightness of a plurality of regions in an image, for two or more first images of the structure at a first wavelength including an absorption band of water, acquired at different times; and identifies a defective region including the defective part from among the plurality of regions on the basis of the first evaluation value.

Description

Image processing device, imaging device, camera system, image processing method, and program

The present invention relates to an image processing device, an imaging device, a camera system, an image processing method, and a program, and more particularly to an image processing device, an imaging device, a camera system, an image processing method, and a program that identify defective areas.

　Conventionally, water leaks have been inspected by irradiating the subject with infrared light and photographing the reflected infrared light.

In Patent Document 1, the intensity distribution of the infrared light reflected from the inspection site when there is no water leakage and the intensity distribution of the infrared light reflected from the inspection site when there is water leakage are received by the light receiving means. A technique for detecting water leaks by comparing is described.

JP-A-11-160185

An embodiment according to the technology of the present disclosure provides an image processing device, an imaging device, a camera system, an image processing method, and a program that can accurately identify a defective area.

An image processing apparatus according to one aspect of the present invention is an image processing apparatus including a processor for identifying a defective portion of a structure, wherein the processor comprises a first wavelength of the structure having a water absorption band and a first wavelength of the structure. In at least two or more first images that are images acquired at different times, a first evaluation value regarding changes in brightness in a plurality of regions in the image is calculated, and based on the first evaluation value, a plurality of Identify the defective area including the defective portion among the areas of .

Preferably, the first evaluation value is the brightness change rate in the plurality of areas.

Preferably, at least one of the rate of change in brightness changes in a direction that increases over time.

Preferably, the brightness change rate is a weighted average brightness change rate calculated by weighted averaging the brightness change rates in a plurality of regions.

Preferably, the processor calculates an image average brightness change rate by averaging the brightness change rates of the first image, and the weighted average brightness change rate lower than the image average brightness change rate by a first threshold or more is the first image brightness change rate. Areas that persist for a period of time are identified as defective areas.

Preferably, the processor changes the first threshold over time.

Preferably, the processor ends identifying the fault location when all the first evaluation values continue to be less than the second threshold.

Preferably, the processor is a second image of a structure with a second wavelength different from the first wavelength, and in at least two or more second images corresponding to the first image, a plurality of images corresponding to the first image A second evaluation value is calculated for changes in brightness in the region, and the defect region is identified based on the first evaluation value and the second evaluation value.

Preferably, the second image is acquired under the same shooting conditions as the corresponding first image, and the second image is taken so that the brightness is lower than that of the first image.

Preferably, the processor divides the first image into a plurality of regions, and changes the division number and division size of the plurality of regions.

Preferably, the processor identifies the defect area based on the first evaluation values of the plurality of areas.

An imaging device according to another aspect of the present invention includes the image processing device described above.

A camera system according to another aspect of the present invention includes the image processing device described above. Preferably, a light projector for irradiating the structure with light is provided.

An image processing method, which is another aspect of the present invention, is an image processing method for an image processing apparatus having a processor for specifying a defective portion of a structure, wherein the processor detects a structure of a first wavelength having a water absorption band. calculating a first evaluation value regarding changes in brightness in a plurality of regions in the first images of the object, which are at least two or more first images acquired at different times; and a step of identifying a defective region including a defective portion among the plurality of regions based on the value.

A program, which is another aspect of the present invention, is a program for causing an image processing device having a processor to perform an image processing method for identifying a defective portion of a structure, wherein Calculating a first evaluation value relating to changes in brightness in a plurality of regions in the first images of the structure of at least two or more obtained at different times; and a step of specifying a defective region including a defective portion among the plurality of regions based on one evaluation value.

FIG. 1 is a block diagram showing an embodiment of the internal configuration of an imaging device. FIG. 2 is a diagram showing main functional blocks of the defect identification processing unit. FIG. 3 is a diagram illustrating calculation of the first evaluation value. FIG. 4 is a diagram for explaining identification of a defective area performed by the defective area identification unit. FIG. 5 is a diagram showing the reflectance relationship between the defective area and the normal area. FIG. 6 is a flow chart showing an image processing method using the defect identification processor. FIG. 7 is a diagram conceptually showing the first image and the second image. FIG. 8 is a diagram showing temporal changes in the reflectance of one region of the first image and the change rate of the reflectance. FIG. 9 is a flow chart showing an imaging method. FIG. 10 is a diagram illustrating division of the first image into a plurality of regions. FIG. 11 is a diagram illustrating division of the first image into a plurality of regions. FIG. 12 is a diagram showing, as an example of the first threshold that fluctuates, the first threshold that fluctuates according to the rate of change. FIG. 13 is a diagram for explaining calculation of the rate of change. FIG. 14 is a flowchart when correction processing is performed. FIG. 15 is a diagram for explaining correction processing. FIG. 16 is a flow chart for starting use of the projector. FIG. 17 is a diagram explaining the timing when the projector is used. FIG. 18 is a diagram for explaining a case in which brightness changes within an image. FIG. 19 is a flow chart for starting use of the projector.

Preferred embodiments of an image processing device, an imaging device, a camera system, an image processing method, and a program according to the present invention will be described below with reference to the accompanying drawings.

Conventionally, light with a wavelength in the absorption band of water has been used when investigating trouble spots that cause rain leaks in buildings. For example, when investigating water leaks on the wall surface of a building, the location of the water leak has been detected by measuring the reflectance of the wall surface (brightness obtained by a camera).

However, the surface of the object to be investigated is not necessarily a smooth surface with a uniform reflectance. For example, if the surface of the wall being investigated has unevenness, if the way the light hits the wall is uneven, or if the wall is made of different materials, the brightness (or reflectance) ) may vary even though there is no defect. In such a case, it is difficult to accurately identify the defect location based on the change in brightness.

Therefore, the present invention, which will be described below, proposes a technique that can accurately identify a defective location based on changes in brightness.

<First embodiment>
FIG. 1 is a block diagram showing an embodiment of an internal configuration of an imaging device equipped with an image processing device (defect identification processing section) of the present invention. Note that the image processing device functions as the defect identification processing unit 30 within the imaging device.

The imaging device 10 captures an image of a structure to be investigated as a subject. Then, by processing the obtained image with the defect identification processing unit 30, a defect area including the defect location is specified. Here, the defective location in the present disclosure includes a location where water leakage (or rain leakage) occurs and a failure location that may cause water leakage (or rain leakage).

The imaging device 10 is centrally controlled by a CPU (Central Processing Unit) (processor) 40 .

The imaging device 10 is provided with an operation unit 38 such as a power switch. A signal from the operation unit 38 is input to the CPU 40, and the CPU 40 controls each circuit of the imaging device 10 based on the input signal. In addition to driving control of the (mechanical shutter) 15, driving control of the diaphragm 14 by the diaphragm driving unit 34, and driving control of the imaging lens 12 by the lens driving unit 36, imaging operation control, image processing control, image data recording/ Playback control, etc.

An external device connected to the imaging device 10 is connected to the external device connection terminal 20 . In this example, a projector 21 is connected as an external device to be connected to the external device connection terminal 20 . The light projector 21 irradiates the object to be investigated with light (for example, light containing a first wavelength and/or a second wavelength, which will be described later), thereby compensating for a shortage of light in photographing by the imaging device 10 . The control of the projector 21 is performed by the CPU 40 . An example using the light projector 21 in the present invention will be described in Modified Example 6. The external device connection terminal 20 also includes a terminal for wireless connection. For example, the external device connection terminal 20 includes terminals for wireless LAN (Local Area Network), Bluetooth (registered trademark), UWB (Ultra Wide Band), and the like.

The luminous flux that has passed through the imaging lens 12, the diaphragm 14, the mechanical shutter (mechanical shutter) 15, etc. is imaged on the imaging element 16, which is a known image sensor. The imaging element 16 can receive a first wavelength having an absorption band of water to obtain a first image. The first wavelength consists of infrared wavelength bands including wavelengths of 1450 nm, 1940 nm, and 2900 nm that are absorbed by water. Further, the imaging element 16 can receive a second wavelength, which will be described later, to obtain a second image. A band-pass filter (not shown) that transmits the first wavelength is inserted in the optical path of the imaging device 10 when acquiring the first image, and transmits the second wavelength when acquiring the second image. A bandpass filter (not shown) is inserted in the optical path of the imaging device 10 to allow the light to pass through.

The imaging element 16 has a large number of light receiving elements (photodiodes) arranged two-dimensionally, and the subject image formed on the light receiving surface of each photodiode has an amount of signal charge corresponding to the amount of light incident on each light receiving element. The voltage is converted into a voltage by an amplifier, converted into a digital signal through an A/D (Analog/Digital) converter in the imaging device 16, and output.

The image signal (image data) read from the imaging element 16 when shooting a moving image or still image is temporarily stored in a memory (SDRAM (Synchronous Dynamic Random Access Memory) 48 via the image input controller 22. The SDRAM is an example, and any memory such as ReRAM (Resistive Random Access Memory), PCM (Phase-change memory), MRAM (Magnetoresistive Random Access Memory) can be selected as necessary.

The ROM 47 is a flash memory (Flash Memory), ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), or MRAM (Magnetoresistive Random Access Memory) and the like.

The image processing unit 24 reads unprocessed image data (RAW data) acquired via the image input controller 22 when shooting a moving image or still image and temporarily stored in the memory 48 . The image processing unit 24 performs offset processing, pixel interpolation processing (interpolation processing for phase difference detection pixels, defective pixels, etc.), white balance correction, gain control processing including sensitivity correction, gamma correction processing, and It performs demosaic processing, luminance and color difference signal generation processing, edge enhancement processing, color correction processing, and the like.

The image processed by the image processing unit 24 is encoded by the video encoder 28 and output to the display unit (not shown). Image data processed by the image processing unit 24 and processed as still images or moving images for recording are stored in the memory 48 again.

Although the image input controller 22, the image processing unit 24, the video encoder 28, etc. have been described as independent processing units (circuits), the CPU 40 can also execute software processing using a program stored in the memory 48 (or ROM 47). is.

As described above, the imaging device 10 captures an image of an object to be investigated, and the image processed by the image processing unit 24 (a still image and a frame image that constitutes a moving image) is processed by the defect identification processing unit 30. , and processing is performed to identify the defect area to be investigated. Note that the imaging apparatus 10 described above is a specific example and is not limited to this. Any imaging device that can capture the first image and/or the second image can be applied to the present invention. Also, a camera system may be used instead of the imaging device described above.

FIG. 2 is a diagram showing main functional blocks of the defect identification processing unit 30. As shown in FIG. Note that each function of the defect identification processing unit 30 may be realized by executing a program by the CPU 40 of the imaging device 10, or may be implemented by one or more CPUs (processors) (unused) installed in the defect identification processing unit 30. shown).

The defect identification processing unit 30 is composed of an image acquisition unit 101 , an evaluation value calculation unit 103 , and a defect area identification unit 105 .

The image acquisition unit 101 acquires at least two first images captured at different times. Further, when the temporal average of the brightness change rate (for example, the weighted average brightness change rate) is calculated, the image acquisition unit 101 obtains at least the first images captured at different times. Get 3 or more. Here, the first image is an image captured using a first wavelength having a water absorption band with a structure to be investigated as an object. Structures include constructions such as buildings, houses, bridges, etc., and are provided indoors or outdoors. Note that the number of first images acquired by the image acquisition unit 101 is appropriately set depending on the investigation target and the investigation environment.

The evaluation value calculation unit 103 calculates a first evaluation value regarding changes in brightness in a plurality of regions in the plurality of first images acquired by the image acquisition unit 101 .

FIG. 3 is a diagram explaining calculation of the first evaluation value in the evaluation value calculation unit 103. FIG.

The first image P1 is an image acquired by the imaging device 10 at time t1. The first image P2 is an image captured by the imaging device 10 at time t2. The first image P3 is an image captured by the imaging device 10 at time t3. In the chronological order, time t1, time t2, and time t3 have elapsed.

The evaluation value calculation unit 103 divides the first image P1 into a plurality of areas (areas S1 to S9). Also, the evaluation value calculation unit 103 divides the first image P2 and the first image P3 into a plurality of regions (regions S1 to S9) so as to correspond to the first image P1.

Then, the evaluation value calculation unit 103 calculates the brightness of the regions S1 to S9 of the first images P1 to P3. Here, the evaluation value calculation unit 103 calculates the brightness of the regions S1 to S9 of the first images P1 to P3 using luminance as one index of brightness. For example, the evaluation value calculation unit 103 calculates the brightness of the area S1 by averaging the brightness of the area S1 of the first image P1 (average brightness). Similarly, the evaluation value calculation unit 103 calculates the brightness (average brightness) of the regions S2 to S9 of the first image P1, the regions S1 to S9 of the first image P2, and the regions S1 to S9 of the first image P3. Calculate

After that, the evaluation value calculation unit 103 calculates a first evaluation value regarding changes in brightness in the regions S1 to S9 of the first images P1 to P3. In the illustrated case, the evaluation value calculation unit 103 calculates, as the first evaluation value, the luminance change rate in the regions S1 to S9 of the first images P1 to P3.

In the case shown in FIG. 3, the evaluation value calculation unit 103 calculates the rate of change between the average luminance of the area S1 of the first image P1 and the average luminance of the area S1 of the first image P2, and the average luminance of the area S1 of the first image P2. A weighted average brightness change rate R1 of the rate of change between the average brightness and the average brightness of the region S1 of the first image P3 is calculated as a first evaluation value.

Similarly, the evaluation value calculation unit 103 calculates the weighted average brightness of the average brightness of the regions S2 to S9 of the first image P1, the regions S1 to S9 of the first image P2, and the regions S1 to S9 of the first image P3. The rate of change in thickness is calculated as the first evaluation value. Note that the weight for calculating the weighted average brightness change rate is appropriately determined.

Also, the first image P1 to the first image P3 are, for example, images of the wall of the house captured by the imaging device 10. In addition, before the imaging device 10 takes an image (before time t1), the wall to be inspected is evenly sprinkled with water to investigate water leakage. On the survey day (the day on which the first image P1 to the first image P3 were captured by the imaging device 10), the weather was fine, the water on the wall surface evaporated and decreased with the passage of time, and the reflectance of the wall surface was getting higher. Therefore, at least one of the weighted average brightness change rate R1 to the weighted average brightness change rate R9 increases.

Returning to FIG. 2, based on the first evaluation value calculated by the evaluation value calculation unit 103, the defect area identifying unit 105 identifies the defect area including the defect location among the plurality of areas. Specifically, the defective area identifying unit 105 determines that the weighted average brightness change rate R1 to the weighted average brightness change rate R9 are lower than the image average brightness change rate by a first threshold or more for a first period. Then, the area corresponding to the weighted average brightness change rate is identified as the defective area. Here, the image average brightness change rate is a change rate obtained by averaging all the weighted average brightness change rates of the regions S1 to S9 in the first image P1 to the first image P3. Note that the first threshold value and the first period are appropriately set according to the investigation target and the investigation environment.

FIG. 4 is a diagram for explaining identification of a defective area performed by the defective area identification unit 105. FIG. In the case shown in FIG. 4, water leakage occurs due to a crack C in the region S5. Note that the crack is an example of a defective portion.

FIG. 4 shows a first image P1 captured at time t1, a first image P2 captured at time t2, and a first image P3 captured at time t3 acquired by the image acquisition unit 101. FIG. Then, the evaluation value calculation unit 103 calculates the weighted average brightness change rate R1 to the weighted average brightness change rate R9 of the regions S1 to S9 in the first image P1 to the first image P3 as the first evaluation values.

The defect area identifying unit 105 acquires the weighted average brightness change rate R1 to the weighted average brightness change rate R9 (see symbol H). In addition, the defect area specifying unit 105 acquires an image average brightness change rate R(ALL) by averaging the weighted average brightness change rate R1 to the weighted average brightness change rate R9.

Then, the defective area identifying unit 105 determines that the weighted average brightness change rate R1 to the weighted average brightness change rate R9 are lower than the image average brightness change rate R(ALL) by a first threshold or more for the first period. detect areas where Here, since the crack C exists in the area S5 and water leakage occurs, the weighted average brightness change rate R5 is lower than the image average brightness change rate R(ALL) by the first threshold or more in the first period. continue. Therefore, the defective area identifying unit 105 identifies the area S5 as a defective area having a defective portion.

Below, the relationship between the reflectance of the defective area and the normal area will be explained.

FIG. 5 is a diagram showing the reflectance relationship between the defective area and the normal area. In FIG. 5, the vertical axis indicates reflectance, and the horizontal axis indicates time. Also shown are times t1 to t3 when the first images P1 to P3 shown in FIG. 4 were captured. A line L1 indicates the reflectance in the region S5 explained in FIG. 4, and a line L2 shows the reflectance in the regions S1 to S4 and the regions S6 to S9 explained in FIG. ing. In the line L1, water seeps into the surface of the object to be investigated from the cracks C and water continues to remain on the surface, so the increase in reflectance is gradual even when time elapses. On the other hand, in the line L2, the water on the surface of the investigation object evaporates with the passage of time, and the reflectance sharply increases. Then, in L2, when the surface under investigation becomes dry, the reflectance remains constant at a high value. As described above, the reflectance of the region S5 having the defective portion gradually increases with the passage of time, and the reflectance of the normal regions S1 to S4 and the regions S6 to S9 sharply increases with the passage of time.

Therefore, the weighted average brightness change rate R5 of the cracked region S5 is the weighted average brightness change rate R1 to the weighted average brightness change rate R4 of the other regions, the weighted average brightness change rate R6 to the weighted average brightness change less than rate R9. Furthermore, the weighted average brightness change rate R5 is lower than the image average brightness change rate R(ALL) by the first threshold or more.

FIG. 6 is a flow chart showing an image processing method using the defect identification processing unit 30 mounted on the imaging device 10. FIG. In this image processing method, each step is executed by executing a program by the CPU 40 or a CPU (not shown) installed in the defect identification processing section 30 . Further, the following description will be made along the specific example described with reference to FIG. 4 .

First, the image acquisition unit 101 acquires the first images P1 to P3 of the first wavelength having the absorption band of water (step S10). Note that the first image P1 to first image P3 are acquired at time t1 to time t3, respectively. Next, the evaluation value calculation unit 103 divides the first image P1 to the first image P3 into regions S1 to S9, and calculates the weighted average brightness change rates R1 to R9 of each region (the first evaluation value is Calculation step: step S11). After that, the defective area identifying unit 105 determines whether there is an area in which the weighted change rate is lower than the average brightness change rate of the image by a first threshold value (referred to as a threshold value in the drawing) or more for a first period of time. (step of identifying the defective area: step S12). In this example, the weighted average brightness change rate R5 of the region S5 is lower than the image average brightness change rate by the first threshold or more, and this state continues for the first period. Therefore, the defect area identifying unit 105 identifies the area S5 as a defect area (step S13).

As described above, according to the defect identification processing unit 30 installed in the imaging device 10, in at least two or more first images acquired at different times, changes in brightness in a plurality of regions within the image is calculated, and based on the first evaluation value, the defective area including the defective portion is specified among the plurality of areas. Therefore, the defective area can be accurately specified.

<Second embodiment>
Next, a second embodiment of the invention will be described. In this embodiment, the second image is shot separately from the first image, and the exposure amount is adjusted for each shot. As a result, although the first images are captured at different times, all the first images can be captured with uniform brightness. In addition, in the present embodiment, if the rate of change of all areas of the first image continues to be less than the set second threshold, it is determined that the investigation target is dry. , terminates the process of identifying the defective portion. As a result, when there is no defective area, the process can be automatically terminated and an efficient investigation can be performed.

<<Capturing the second image>>
First, the shooting of the second image performed by the imaging device 10 will be described. The second image is an image of the structure under investigation at a second wavelength. Preferably, the second wavelength is a wavelength different from the absorption band of the defect site under investigation. Specifically, the second image is preferably captured at a wavelength lower than the first wavelength with respect to the absorbance of the leak (water) to be detected. By capturing the second image at such a second wavelength, it is possible to suppress exposure adjustment from being affected by absorption of light by the water leakage to be detected.

FIG. 7 is a diagram conceptually showing the first image and the second image captured by the imaging device 10. FIG. Although FIG. 7 schematically shows the case where the first image and the second image are acquired at time t1 and time t2, more first images and second images may be acquired. .

At time t1, the imaging device 10 captures the second image Q1 and the first image P1. Also, the second image Q2 and the first image P2 are captured by the imaging device 10 at time t2.

At time t1, the first image P1 is captured immediately after the second image Q1 is captured or substantially immediately from the viewpoint of exposure adjustment. Then, the second image Q1 and the first image P1 are shot under the same shooting conditions. Similarly, at time t2, the first image P2 is shot immediately after the second image Q2 is shot, or substantially immediately from the viewpoint of exposure adjustment. Then, the second image Q2 and the first image P2 are shot under the same shooting conditions. Note that it is preferable to adjust the exposure between the second image Q1 and the second image Q2 by adjusting the gain of the diaphragm 14 and the image sensor 16 while keeping the exposure time constant. As a result, it is possible to suppress the influence of variation in the sensitivity of the imaging element 16 due to the exposure time.

The second image Q1 and the second image Q2 are shot with uniform brightness after exposure adjustment. Also, the second image Q1 and the first image P1 are shot under the same shooting conditions, and the second image Q2 and the first image P2 are shot under the same shooting conditions. Therefore, since the first image P1 and the first image P2 are shot with the same brightness, the first evaluation value can be obtained with high accuracy, and the defect area can be specified more accurately.

<< Determination of dry state >>
Next, determination of the dry state will be described. The defective area identifying unit 105 determines the dry state based on the first evaluation value of each area, and ends the process of identifying the defective area when determining that the structure to be inspected is in a dry state. Specifically, if all the first evaluation values continue to be less than the second threshold value, the defect area identification unit 105 terminates the defect area detection processing performed by the defect identification processing unit 30. .

FIG. 8 is a diagram showing temporal changes in the reflectance of one region of the first image and the change rate of the reflectance. In this example, the object of investigation is a structure installed outdoors, and the water on the surface of the structure gradually evaporates over time.

A line L11 indicates the reflectance, and a line L12 indicates the rate of change. As indicated by line L11, the water on the surface of the structure evaporates over time and the reflectance increases. Moreover, as indicated by line L11, after a certain amount of time has passed, the amount of water on the surface of the structure decreases due to evaporation, and the increase in reflectance also slows down.

As shown by line L12, after a certain amount of time has passed, the rate of change continues to be less than the second threshold. In such a state, the defective area identifying unit 105 determines that the area is dry. The defect identification processing unit 30 terminates the defect identification processing performed by the defect identification processing unit 30 when determining that all of the plurality of areas of the first image are dry. As a result, when all the areas are normal, the defect area specifying unit 105 can automatically determine that there is no defect, and can perform an efficient investigation.

FIG. 9 is a flowchart showing an imaging method using the imaging device 10 of this embodiment. In this image processing method, each step is executed by executing a program by the CPU 40 or a CPU (not shown) installed in the defect identification processing section 30 . Further, the following description will be made along the specific example described with reference to FIG.

First, the imaging device 10 captures the second image Q1 at time t1 (step S20). After that, the imaging device 10 shoots the first image P1 under the same shooting conditions as when the second image Q1 was shot (step S21). After that, the imaging device 10 captures the second image Q2 at time t2 (step S22). The imaging device 10 adjusts the exposure so that the second image Q2 has the same brightness as the second image Q1, and shoots the second image Q2. After that, the imaging device 10 shoots the first image P2 under the same shooting conditions as when the second image Q2 was shot (step S23).

After that, the defect identification processing unit 30 acquires the first image P1 and the first image P2 (step S24). Further, the defect identification processing unit 30 acquires the second image Q1 and the second image Q2 (step S25).

After that, the evaluation value calculation unit 103 divides the obtained first image P1, first image P2, second image Q1, and second image Q2 into a plurality of regions, and calculates the luminance change rate of each region (step S26 ). Specifically, the evaluation value calculation unit 103 calculates luminance change rates (first evaluation values) of a plurality of regions of the first image P1 and the first image P2. The evaluation value calculation unit 103 also calculates luminance change rates (second evaluation values) of a plurality of regions of the second image Q1 and the second image Q2. Then, the defect area identifying unit 105 identifies the defect area based on the rate of change of the first image and the rate of change of the second image (step S27). Specifically, the defect area identifying unit 105 calculates the difference or ratio between the rate of change of the first image and the rate of change of the second image, and the calculated rate of change (correction rate of change) is the image average brightness change. It is determined whether or not there is an area where the rate is lower than the rate by a first threshold or more for a first period. Then, if there is an area in which the corrected change rate is lower than the image average brightness change rate by the first threshold value or more for the first period of time, that area is identified as a defective area (step S28).

After that, it is determined whether or not the corrected rate of change continues to be less than the second threshold for the second period (fixed period in the figure) in all regions (step S29). If the corrected rate of change continues to be less than the second threshold for the second period in all regions, the defect region identifying unit 105 determines that all regions are normal regions, and performs processing to identify the defect region. terminate. On the other hand, if the corrected rate of change is greater than or equal to the second threshold, then a second image is taken to continue the investigation. Note that the second threshold and the second period are values arbitrarily set by the user.

In the above description, an example was explained in which the corrected change rate calculated from the first evaluation value and the second evaluation value is used as the change rate. The example of the present embodiment is not limited to this, and as described in the first embodiment, the defect area identification process is performed using the first evaluation value.

As described above, in the present embodiment, the first image can be captured with uniform brightness by capturing the second image and adjusting the brightness before capturing the first image. As a result, the present embodiment can identify the defect area with higher accuracy. Further, in the present embodiment, when the rate of change of all regions of the first image continues to be less than the second threshold value, it is determined that the investigation target is dry, and the process of identifying the defective portion is performed. terminate. As a result, when there is no defective area, the process can be automatically terminated and an efficient investigation can be performed.

<Modification 1>
Next, Modification 1 of the present invention will be described. Depending on the material, the way the survey target spreads when it absorbs water may change. Therefore, in this example, when the evaluation value calculation unit 103 divides the first image to set a plurality of areas, the number of divisions and the division size are changed according to the member to be investigated. As a result, the first evaluation value is calculated in consideration of the difference in how water spreads due to the member when flooded, and the process of identifying the defective area can be performed, thereby improving the accuracy of identifying the defective area.

FIG. 10 is a diagram explaining how the evaluation value calculation unit 103 divides the first image into a plurality of regions.

The first image P11 is the first image when a concrete structure is targeted for investigation. Concrete structures are members that are difficult to be flooded. Therefore, even when water leakage occurs, the area D1 where water is flooded is narrow. In such a case, the evaluation value calculation unit 103 divides the first image P11 by setting the region S to be small.

The first image P12 is the first image when a wooden structure is targeted for investigation. A wooden structure is a member that is easily flooded. Therefore, even when water leakage occurs, the area D2 where water is flooded is wide. In such a case, the evaluation value calculation unit 103 divides the first image P12 by setting the region S to be large.

It should be noted that the number of divisions and the division size of the area are input and set in advance by the user according to the member to be investigated.

As described above, in this example, the number of divided regions of the first image performed by the evaluation value calculation unit 103 or the size of the divided regions can be changed according to the members constituting the investigation target. . As a result, it is possible to specify a defective area in consideration of the difference in how water spreads when flooded due to the member to be investigated.

<Modification 2>
Next, Modification 2 of the present invention will be described. In this example, the evaluation value calculation unit 103 changes the size of the area S according to the size of the defective portion, the subject distance, and the focal length.

FIG. 11 is a diagram explaining how the evaluation value calculation unit 103 divides the first image into a plurality of regions.

The first image P13 is the first image captured when the subject distance is long or when the focus is short. Also, the first image P13 is the first image captured when the crack to be detected is small. In such a case, the defective portion D3 is photographed small, so the evaluation value calculation unit 103 sets the region S small. Specifically, the area S is set so that the defective portion D3 to be detected is circumscribed.

The first image P14 is the first image captured when the subject distance is short or when the focus is long. Also, the first image P14 is the first image captured when the crack to be detected is large. In such a case, the defective portion D4 is photographed small, so the evaluation value calculation unit 103 sets the region S large. Specifically, the area S is set so as to include the defective portion D4 to be detected.

It should be noted that the number of divisions and the division size of the region are input and set in advance by the user according to the size of the defect location to be detected.

As described above, in this example, the evaluation value calculation unit 103 changes the size of the area S according to the size of the defect location, the subject distance, and the focal length. As a result, the defect area identifying unit 105 can accurately identify the defect area.

<Modification 3>
Next, Modification 3 of the present invention will be described. In this example, the first threshold used when the defect area identifying unit 105 identifies the defect area is dynamically changed.

If the first threshold, which is the threshold for the rate of change in brightness, is constant, and the rate of change in each area fluctuates non-linearly, there is a possibility that the defect area identification process will not be performed appropriately. Further, when the first threshold is constant, it may take time to specify the defect area. Therefore, in this example, the first threshold value is not set to a constant value, but is dynamically changed to identify the defective portion in an appropriately short time.

FIG. 12 is a diagram showing, as an example of the first threshold that fluctuates, the first threshold that fluctuates according to the rate of change. The left vertical axis indicates reflectance, the right vertical axis indicates rate of change, and the horizontal axis indicates time.

A line L110 and a line L111 are diagrams showing the reflectance of the region S. A line L110 indicates the temporal change in reflectance of the region identified as the defective region, and a line L111 indicates the temporal change of the reflectance of the normal region. As indicated by lines L110 and L111, it takes time for the reflectance of the region identified as the defective region to increase compared to the reflectance of the normal region.

A line L120 and a line L121 are diagrams showing the change rate of the reflectance of the region S. A line L120 indicates the temporal change in the rate of change in the area identified as the defective area, and a line L121 indicates the temporal change in the rate of change in the normal area. As indicated by lines L120 and L121, the rate of change in the reflectance of the area identified as the defect area gradually increases with the lapse of time, and then increases sharply.

A line L101 is a diagram showing the first threshold. The fluctuating first threshold value may be the average of the rate of change of all areas of the first image (image average brightness change rate) or the median value of the rate of change of all areas of the first image. As the first threshold value, a value based on the average of the rate of change of all regions of the first image (image average brightness change rate) or the median value of the rate of change of all regions of the first image is adopted. may Further, as the first threshold, the average of the rate of change of all regions of the first image (image average brightness change rate), or the median value of the rate of change of all regions of the first image, which is greater than 0 and greater than 1 A value multiplied by a small factor may be adopted.

As described above, it is possible to efficiently identify a defective area by varying the first threshold value used when the defective area identifying unit 105 identifies the defective area.

<Modification 4>
Next, Modification 4 of the present invention will be described. In this example, the evaluation value calculation unit 103 may calculate the rate of change using a plurality of adjacent areas instead of the rate of change of one area.

FIG. 13 is a diagram explaining the calculation of the rate of change in the evaluation value calculation unit 103 of this example.

In the first image P15, the evaluation value calculation unit 103 calculates the rate of change in brightness in the areas S11, S12, and S13 in accordance with the area of the defect location D. In this manner, the evaluation value calculation unit 103 may calculate the rate of change in brightness not only for one region but also for adjacent regions. Note that the evaluation value calculation unit 103 may calculate the rate of change in brightness of vertically elongated regions such as the adjacent regions S11 to S13 as described above, or calculate the rate of change in brightness of the horizontally elongated region. Alternatively, the rate of change in brightness of five vertically and horizontally adjacent regions may be calculated, or the brightness of nine regions obtained by adding a diagonally adjacent region to the five vertically and horizontally adjacent regions. A rate of change may be calculated.

As described above, the evaluation value calculation unit 103 calculates the rate of change in the brightness of adjacent regions, and if there is a tendency in the shape of the defective portion, setting the region in accordance with the shape can improve the brightness. Robust failure area identification can be performed.

<Modification 5>
Next, Modification 5 of the present invention will be described. In this example, when the imaging apparatus 10 is provided with a LOG (logarithmic) mode, the evaluation value calculation unit 103 performs correction processing.

FIG. 14 is a flowchart when correction processing is performed.

The evaluation value calculation unit 103 determines whether the LOG mode is activated in the imaging device 10 (step S30). When the evaluation value calculation unit 103 determines that the LOG mode is activated, it executes the correction processing described below (step S31).

FIG. 15 is a diagram for explaining correction processing performed by the evaluation value calculation unit 103 of this example.

The imaging device 10 may have a LOG (logarithmic) mode and a LINEAR (linear) mode. In the LOG mode, as shown in FIG. 15A, the output value is nonlinear with respect to the amount of incident light. If the process of specifying the defect area of the present invention is performed based on such an output value, the calculation of the first evaluation value by the evaluation value calculation unit 103 and the process by the defect area identification unit 105 may become complicated. . Therefore, when the LOG mode is set in the imaging apparatus 10, the evaluation value calculation unit 103 performs processing to cancel the LOG mode. Specifically, as shown in FIG. 15B, correction is performed such that the corrected value is output with respect to the output value from the camera. As a result, an output value that is linear with respect to the amount of light incident on the camera is output as shown in FIG. 15(C). As a result, even when the LOG mode is activated, it is possible to suppress the influence of the non-linear image processing as shown in FIG. is improved.

<Modification 6>
Next, Modification 6 of the present invention will be described. In this example, using the light projector 21 provided in the imaging device 10, processing for identifying the defective area is performed.

If the rate of change in brightness remains lower than the third threshold (denoted as the threshold in the figure) during the third period (denoted as the constant period in the figure), drying may occur due to factors such as high humidity or low temperature. It is considered a situation in which no progress can be made. Therefore, in this case, drying is accelerated by irradiating the survey object with the light projector 21 . Thus, by using the light projector 21, it is possible to identify the defective area in a short time.

FIG. 16 is a flow chart for starting to use the projector 21. FIG.

The evaluation value calculation unit 103 determines whether or not the brightness change rate is less than the third threshold during the third period (step S41). If the rate of change remains below the third threshold, the evaluation value calculator 103 turns on the light projector 21 via the CPU 40, and the light projector 21 is used (step S42).

FIG. 17 is a diagram explaining the timing when the light projector 21 is used. The left vertical axis is the reflectance, the right vertical axis is the rate of change, and the horizontal axis is time.

As shown in FIG. 17, when the rate of change lower than the third threshold continues during the third period, the projector 21 is started to be used. By starting to use the light projector 21, the reflectance and rate of change abruptly improve (see lines L130 and L131). This makes it possible to identify the defective area in a short period of time. It should be noted that the illustrated third threshold value and third period are appropriately set according to the investigation target and the investigation environment.

Next, another aspect of using the light projector 21 will be described.

For example, if the amount of light illuminating the investigation target is lower than the fourth threshold, the amount of light received by illuminating the investigation target with the light projector 21 is increased. As a result, the lack of sensitivity of the imaging element 16 and the influence of noise can be suppressed, and the accuracy of identifying the defective area can be improved. Therefore, the evaluation value calculation unit 103 starts using the light projector 21 via the CPU 40 when the light amount in the acquired first image is lower than the fourth threshold. Note that the fourth threshold is a value appropriately set by the user.

Next, another aspect of using the light projector 21 will be described.

When shooting the second image at different times, the brightness may change depending on the location in the image as described below due to changes in the lighting conditions.

FIG. 18 is a diagram explaining a case where the brightness changes in the image.

The second image P31 is an image captured at time t1. A second image P32 is an image captured at time t2. The second image P32 is an image after exposure adjustment, but since the shooting time is different, the sunshine conditions are different, and there is an area with a certain amount or more of difference in the image. In such a case, the projector 21 is used.

FIG. 19 is a flow chart for starting to use the projector 21. FIG.

The evaluation value calculation unit 103 determines whether or not there is an area having a difference equal to or greater than a fifth threshold (referred to as a threshold in the figure) in the second image (step S50). If there is an area with a difference equal to or greater than the fifth threshold in the second image, the projector 21 is used (step S51). This makes it possible to maintain the accuracy of specifying the defect area even when the sunshine conditions change due to different shooting times. In addition, the fifth threshold value described above is a value appropriately set by the user.

In the above embodiment, the hardware structure of the processing unit (for example, the image acquisition unit 101, the evaluation value calculation unit 103, and the defect area identification unit 105) (processing unit) that executes various processes is as follows. various processors. For various processors, the circuit configuration can be changed after manufacturing, such as CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), which is a general-purpose processor that executes software (program) and functions as various processing units. Programmable Logic Device (PLD), which is a processor, ASIC (Application Specific Integrated Circuit), etc. be

One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types (eg, multiple FPGAs, or combinations of CPUs and FPGAs). may Also, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units in a single processor, first, as represented by a computer such as a client or server, a single processor is configured by combining one or more CPUs and software. There is a form in which a processor functions as multiple processing units. Second, as typified by System On Chip (SoC), etc., there is a form of using a processor that realizes the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip. be. In this way, the various processing units are configured by using one or more of the above various processors as a hardware structure.

Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

Each configuration and function described above can be appropriately realized by arbitrary hardware, software, or a combination of both. For example, a program that causes a computer to execute the above-described processing steps (procedures), a computer-readable recording medium (non-temporary recording medium) recording such a program, or a computer capable of installing such a program However, it is possible to apply the present invention.

Although the examples of the present invention have been described above, it goes without saying that the present invention is not limited to the above-described embodiments, and that various modifications are possible without departing from the gist of the present invention.

10 : Imaging device 12 : Imaging lens 14 : Aperture 16 : Imaging element 20 : External device connection terminal 21 : Projector 22 : Image input controller 24 : Image processing unit 28 : Video encoder 30 : Defect identification processing unit 32 : Sensor driving unit 33 : Shutter drive unit 34 : Aperture drive unit 36 : Lens drive unit 38 : Operation unit 40 : CPU
47: ROM
48: memory 101: image acquisition unit 103: evaluation value calculation unit 105: defect area identification unit

Claims

An image processing device, comprising a processor, for identifying a defective portion of a structure,
The processor
A first image of the structure at a first wavelength having a water absorption band, wherein at least two or more of the first images taken at different times relate to changes in brightness in a plurality of regions within the image. Calculate the first evaluation value,
Based on the first evaluation value, identifying a defect area including the defect location among the plurality of areas;
Image processing device.
The image processing apparatus according to claim 1, wherein the first evaluation value is a brightness change rate in the plurality of areas.
The image processing apparatus according to claim 2, wherein at least one of the rate of change in brightness changes in an increasing direction over time.
The image processing apparatus according to claim 2 or 3, wherein the rate of change in brightness is a weighted average rate of change in brightness calculated by weighted averaging the rates of change in brightness in the plurality of regions.
The processor
calculating an image average brightness change rate by averaging the brightness change rate of the first image;
5. The image processing apparatus according to claim 4, wherein the area in which the weighted average brightness change rate lower than the image average brightness change rate by a first threshold or more continues for a first period is specified as the defective area.
The image processing apparatus according to claim 5, wherein the processor changes the first threshold over time.
The image processing apparatus according to claim 5 or 6, wherein the processor terminates identifying the defective portion when all the first evaluation values continue to be less than the second threshold value.
The processor
In at least two or more second images of the structure corresponding to the first image, which are second images of the structure at a second wavelength different from the first wavelength,
8. Calculating a second evaluation value regarding changes in brightness in a plurality of regions corresponding to the first image, and specifying the defect region based on the first evaluation value and the second evaluation value. The image processing device according to any one of .
The second image is acquired under the same shooting conditions as the corresponding first image,
9. The image processing apparatus according to claim 8, wherein the second image is shot so as to be less bright than the first image.
The processor
dividing the first image into the plurality of regions;
10. The image processing apparatus according to any one of claims 1 to 9, wherein the division number and division size of the plurality of regions are changed.
The processor
The image processing apparatus according to any one of claims 1 to 10, wherein the defect area is specified based on the first evaluation values of the plurality of areas.
An imaging device comprising the image processing device according to any one of claims 1 to 11.
A camera system comprising the image processing device according to any one of claims 1 to 11.
The camera system according to claim 13, comprising a projector for irradiating the structure with light.
An image processing method for an image processing device having a processor for identifying a defective portion of a structure,
by the processor,
A first image of the structure at a first wavelength having a water absorption band, wherein at least two or more of the first images taken at different times relate to changes in brightness in a plurality of regions within the image. a step of calculating a first evaluation value;
a step of identifying a defective region including the defective portion among the plurality of regions based on the first evaluation value;
An image processing method in which
A program for causing an image processing device having a processor to perform an image processing method for identifying a defective portion of a structure,
to the processor;
A first image of the structure at a first wavelength having a water absorption band, wherein at least two or more of the first images taken at different times relate to changes in brightness in a plurality of regions within the image. a step of calculating a first evaluation value;
a step of identifying a defective region including the defective portion among the plurality of regions based on the first evaluation value;
A program that allows you to do
A recording medium that is non-temporary and computer-readable, in which the program according to claim 16 is recorded.