JP2018056786A - Image processing device, imaging apparatus, movable body and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable detection of abnormality in an image due to damage on an image pick-up device following burn-in of a far infrared ray from the sun.SOLUTION: An image processing device 3 being the image processing device 3 mounted on a movable body 6 includes: an acquisition unit 31 being the acquisition unit 31 which acquires an image captured by a far infrared camera 1, acquires a first image at the first time point when the movable body 6 is stopped, and acquires a second image at the second time point before the movable body 6 starts moving again after the first time point; and a processor 35 which detects abnormality in the second image due to the trajectory of a sun image formed on the image pick-up device 22 of the far infrared camera 1 on the basis of the comparison between the first image and the second image, and corrects an area where the abnormality is detected in the second image with respect to an image acquired from the acquisition unit 31 subsequently to the second time point.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing device, an imaging device, a moving body, and an image processing method.

  2. Description of the Related Art Conventionally, cameras that do not include a mechanical shutter that adjusts an exposure time (shutter speed) are known. Such a camera includes an electronic shutter that generates an image signal based on the electric charge accumulated in a predetermined time after discharging the electric charge of the image sensor instead of the mechanical shutter.

JP 2007-201807 A

  However, when a camera that does not include a mechanical shutter is placed outdoors as in the prior art described above, the imaging surface of the image sensor is not sunlit regardless of whether or not there is an operation for imaging by the camera. May receive far infrared rays.

  In particular, if the camera continues to be placed outdoors at the same position and direction for a long time, only a part of the image sensor continues to image far infrared rays from the sun, and these regions burn in far infrared rays from the sun. May cause damage. In this case, the imaging device cannot normally convert the light received in the damaged area into an image signal, and the pixel value of the image corresponding to the damaged area is not damaged. It is different from the pixel value. That is, an abnormality occurs in a part of an image captured by a camera including such an image sensor.

  Therefore, the object of the present invention made by paying attention to these points is an image processing device, an imaging device, a moving body, which can detect an abnormality in an image due to damage of an imaging device due to burn-in of far infrared rays from the sun, And providing an image processing method.

  An image processing apparatus that solves the above-described problem is an image processing apparatus mounted on a moving body, and is an acquisition unit that acquires an image captured by a far-infrared camera, and the moving body is stopped first. An acquisition unit that acquires a first image at a time point (a), and acquires a second image at a second time point after the first time point and before the moving body starts moving again; Based on the comparison between the image and the second image, an abnormality of the second image due to the locus of the sun image formed on the imaging device of the far-infrared camera is detected, and after the second time point, A processor that corrects an area in which an abnormality is detected in the second image with respect to the image acquired from the acquisition unit.

  An imaging device that solves the above-described problem is an imaging device that is mounted on a moving body, and is picked up by an imaging unit that captures an image using far-infrared rays and a first time point when the moving body is stopped. An acquisition unit that acquires the first image that has been acquired and acquires a second image captured by the imaging unit at a second time after the first time and before the moving body starts moving again And detecting the abnormality of the second image due to the trajectory of the sun image formed on the imaging element of the imaging unit based on the comparison between the first image and the second image, And a processor that corrects a region where an abnormality is detected in the second image with respect to the image acquired from the acquisition unit.

  A moving body that solves the above problem acquires an image capturing unit that captures an image using far infrared rays, and a first image captured by the image capturing unit at a first time point when the moving body is stopped. An acquisition unit that acquires a second image captured by the imaging unit at a second time point after the time point before the moving body starts moving again; and the first image and the second image Based on the comparison with the above, the abnormality of the second image due to the trajectory of the sun image formed on the image sensor of the imaging unit is detected, and the image acquired from the acquisition unit after the second time point is detected. And an image processing apparatus that includes a processor that corrects a region where an abnormality is detected in the second image.

  An image processing method for solving the above-described problem is an image processing method executed by an image processing device mounted on a moving body, and the image processing device detects a far infrared ray at a first time point when the moving body is stopped. A first image captured by the camera is acquired, and after the first time point, a second image captured by the far-infrared camera at a second time point before the moving body starts moving again. The image processing device acquires the second image based on a trajectory of the sun image formed on the image sensor of the far-infrared camera based on the comparison between the first image and the second image. An abnormality is detected, and after the second time point, an area where the abnormality is detected in the second image is corrected with respect to the acquired image.

  According to an embodiment of the present invention, it is possible to detect an abnormality in an image due to damage to an image sensor due to burn-in of far infrared rays from the sun.

FIG. 1 is a functional block diagram illustrating a schematic configuration of the imaging apparatus according to the first embodiment. FIG. 2 is a diagram illustrating an example of a moving object on which the imaging apparatus illustrated in FIG. 1 is mounted. 3 is a schematic diagram showing an example of an image processed by the image processing apparatus shown in FIG. 1, FIG. 3 (a) is an example showing a first image, and FIG. 3 (b) is a second image. It is an example which shows an image. FIG. 4 is a flowchart showing a processing flow of the image processing apparatus shown in FIG. FIG. 5 is a flowchart showing details of the first determination process in the flowchart shown in FIG. FIG. 6 is a flowchart showing details of the second determination process in the flowchart shown in FIG. FIG. 7 is a flowchart showing details of the second determination process in the image processing apparatus according to the second embodiment.

  A first embodiment of the present invention will be described below with reference to the drawings.

  As shown in FIG. 1, a far-infrared camera (imaging device) 1 includes an imaging unit 2, an image processing device 3, and the like. The far-infrared camera 1 is connected to an ECU (Engine Control Unit) 4, a display device 5, and the like via a communication cable and transmits and receives information.

  Further, the far-infrared camera 1 is mounted on a moving body 6 as shown in FIG. The far-infrared camera 1 mounted on the moving body 6 images, for example, the front side of the moving body 6. The imaging direction of the far-infrared camera 1 is not limited to the front of the moving body 6, and the side and rear of the moving body 6 may be imaged. Here, the “moving body” means, for example, a vehicle traveling on a road such as a passenger car, a truck, and a bus.

  The far-infrared camera 1 is a camera that captures a far-infrared image (hereinafter referred to as “image”) using far-infrared rays. Far-infrared rays are electromagnetic waves having a wavelength of about 4 μm to 1 mm emitted from an object according to temperature. As the far-infrared camera 1, for example, a camera that detects light in a wavelength band of about 8 μm to 14 μm can be used. The wavelength band of the far-infrared camera 1 is not limited to this, and light in other bands other than this band or in other bands including this band may be imaged. As an optical member such as a lens constituting the far-infrared camera 1, for example, germanium, silicon, zinc sulfide or the like is used. As the material of the optical member, other materials may be used according to the wavelength of far infrared rays to be used.

  As shown in FIG. 1, the imaging unit 2 includes an imaging lens 21, an imaging element 22, a signal processing unit 23, and the like.

  The imaging lens 21 is a lens that collects light so that incident light forms an image on the imaging element 22. The imaging lens 21 may be configured by, for example, a fisheye lens or an ultra wide angle lens. The imaging lens 21 may be composed of a single lens or a plurality of lenses.

  The imaging element 22 is an imaging element that captures an image formed by the imaging lens 21. The image sensor 22 can be a microbolometer type detector. As the image sensor 22 of the far-infrared camera 1, other thermal infrared sensors such as a thermoelectric element and a pyroelectric element, a quantum infrared image sensor, or the like can be used.

  The signal processing unit 23 is a processor that processes an image, for example, a dedicated microprocessor formed to execute a specific function or a processor that executes a specific function by reading a specific program.

  The signal processing unit 23 generates an image signal representing an image formed by the image sensor 22. Further, the signal processing unit 23 may perform arbitrary processing such as distortion correction, gain adjustment, contrast adjustment, and gamma correction on the image.

  The image processing apparatus 3 includes an acquisition unit 31, an image memory 32, a solar information memory 33, a communication interface 34, a processor 35, and the like.

  The acquisition unit 31 is an interface that acquires the image captured by the imaging unit 2 by acquiring the image signal generated by the signal processing unit 23. The acquisition unit 31 can employ a physical connector and a wireless communication device. The physical connector includes an electrical connector that supports transmission using an electrical signal, an optical connector that supports transmission using an optical signal, and an electromagnetic connector that supports transmission using electromagnetic waves. For electrical connectors, connectors conforming to IEC60603, connectors conforming to USB standards, connectors corresponding to RCA terminals, connectors corresponding to S terminals defined in EIAJ CP-1211A, D terminals defined in EIAJ RC-5237 , A connector conforming to the HDMI (registered trademark) standard, and a connector corresponding to a coaxial cable including BNC. The optical connector includes various connectors conforming to IEC 61754. The wireless communication device includes a wireless communication device that complies with each standard including Bluetooth (registered trademark) and IEEE802.11. The wireless communication device includes at least one antenna.

  The acquisition unit 31 acquires an image captured by the imaging unit 2. Specifically, the acquisition unit 31 acquires the first image at the first time point when the moving body 6 is stopped, and after the first time point, the first time before the moving body 6 starts moving again. A second image is acquired at time 2.

  For example, the acquisition unit 31 is captured by the far-infrared camera 1 with the first time point when the stop signal indicating that the communication interface 34 has started control for stopping the moving body 6 is received from the ECU 4. A first image is acquired. In addition, the imaging unit 2 sets the time point when the communication interface 34 receives the start signal indicating that the control for starting the operation of the moving body 6 from the stopped state is performed from the ECU 4 as the second time point, and the second image. To get.

  From the first time point to the second time point, the moving body 6 continues to be stopped. Therefore, in the first image and the second image, substantially the same direction is imaged at substantially the same position, and the same thing in the image is the same in the stationary object such as a building, a road, and the sky. The image is taken at the position. On the other hand, there are cases where the first image and the second image are different for movable subjects such as people and other moving objects.

  With reference to the example shown in FIG. 3, the first image and the second image will be described. In the first image illustrated in FIG. 3A, a building, a road, the sky, and a tree are captured. In addition, in the second image illustrated in FIG. 3B, buildings, roads, sky, and trees are imaged as in FIG. 3A, and further, images are captured in FIG. 3A. A person who is not photographed in the area b1. Further, the region b2 is an abnormality in which an abnormality is detected because the part corresponding to the region b2 of the imaging surface of the image sensor 22 directly receives far-infrared rays emitted from the sun and the image sensor 22 is damaged. It is an area. The pixel values of the pixels constituting the abnormal region are different from the pixel values obtained under the same imaging conditions when the image sensor 22 is not damaged.

  In addition, the acquisition unit 31 can deliver the image captured by the imaging unit 2 to the processor 35.

  The image memory 32 can store the first image captured by the imaging unit 2 and acquired by the acquisition unit 31. As the image memory 32, volatile memory such as DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), and non-volatile memory such as flash memory can be used.

  The sun information memory 33 stores the diameter of the sun image captured by the far-infrared camera 1. The sun image is an image of the sun that appears in the image space when far infrared rays from the sun are directly imaged on the imaging surface of the image sensor 22. As the solar information memory 33, a volatile memory such as a DRAM (Dynamic Random Access Memory) and an SRAM (Static Random Access Memory), and a nonvolatile memory such as a flash memory can be used.

  The communication interface 34 is an interface for the image processing device 3 to transmit / receive information to / from the ECU 4 and the display device 5.

  The communication interface 34 receives from the ECU 4 a stop signal indicating that control for stopping the moving body 6 has started. Further, the communication interface 34 receives from the ECU 4 a start signal indicating that control for starting the operation of the moving body 6 from the stopped state has been performed.

  In addition, the communication interface 34 transmits images captured after the second time point to the display device 5.

  The communication interface 34 can be a physical connector, a wireless communication device, or the like. Physical connectors include electrical connectors, optical connectors, and electromagnetic connectors. The wireless communication device includes a wireless communication device that conforms to each standard including Bluetooth (registered trademark) and IEEE802.11, and an antenna.

  The processor 35 processes the image acquired by the acquisition unit 31. The processor 35 includes one or more processors. The processor 35 may include one or a plurality of memories that store programs for various processes and information being calculated. The memory includes volatile memory and nonvolatile memory. The memory includes a memory independent of the processor 35 and a built-in memory of the processor 35. The processor 35 includes a general-purpose processor that reads a specific program and executes a specific function, and a dedicated processor specialized for specific processing. The dedicated processor includes an application specific integrated circuit (ASIC). The processor 35 includes a programmable logic device (PLD). The PLD includes a field-programmable gate array (FPGA). The processor 35 may be one of SoC (System-on-a-Chip) and SiP (System In a Package) in which one or more processors cooperate.

  Based on the comparison between the first image and the second image, the processor 35 detects an abnormality of the second image due to the sun image formed on the imaging element 22 of the imaging unit 2, and after the second time point The region where the abnormality is detected in the second image is corrected with respect to the image acquired from the acquisition unit 31.

  Specifically, the processor 35 determines whether or not the second image has an abnormality due to burn-in based on the first image and the second image captured by the imaging unit 2. 1 determination processing is performed.

In the first determination process, the processor 35 determines the pixel values of the pixels constituting the first image and the pixels at the positions on the second image corresponding to the positions of the pixels of the first image, respectively. Are respectively calculated. Further, the processor 35 may associate the difference value calculated in this way with the position of each pixel to obtain a difference image. The processor 35 determines whether or not the difference in the difference image is greater than or equal to the _first threshold value TH1. The processor 35, when the difference is determined to be the first threshold value TH ₁ or more, according to the difference of the first threshold value TH ₁ or more, the correction candidate pixels the pixels constituting the second image is a possibility of abnormal Extract as The first and the threshold TH _1, in the first image and the second image, a threshold value set to a difference value larger than the pixel values may be caused by an object other than the sun.

When both the first image and the second image have N × M resolution, the processor 35 determines that the pixel located at the coordinates (1, 1) of the first image and the coordinates (1, 1, It calculates the difference _{D 11} pixels located in 1). Similarly, the processor 35 sets the pixel located at the coordinates (j, k) (j = 1 to N, k = 1 to M) of the first image and the coordinates (j, k) of the second image. The difference D _jk from the pixel located is calculated. Then, the processor 35, the difference _{D jk} for all j and k determines whether the first threshold value TH ₁ or more. Then, the processor 35 extracts a pixel of the second image related to the difference D _jk that is _equal to or greater than the first threshold TH ₁ as a correction candidate pixel.

  As described above, when an abnormality occurs when the second image is captured in a portion of the image sensor 22 that is not abnormal due to burn-in when the first image is captured, the pixels corresponding to the portion where the image sensor 22 is burned are corrected. Extracted as candidate pixels.

  Further, the processor 35 extracts a correction candidate area and performs a second determination process for determining whether the correction candidate area is abnormal due to burn-in based on the shape of the correction candidate area and the diameter of the sun image. The correction candidate area is an area configured by a set of correction candidate pixels of the second image. The shape of the correction candidate area includes the width of the correction candidate area.

  Specifically, the processor 35 extracts a continuous portion in which the number of correction candidate pixels determined in the second image per unit area is equal to or greater than a predetermined density threshold, and the extracted portion is used as a correction candidate region. May be set as The density threshold value is a value at which correction candidate pixels are considered to be gathered in a region where the number of correction candidate pixels per unit area is equal to or greater than this value. Further, the processor 35 extracts the diameter of the sun image stored in the sun information memory 33.

Then, the processor 35 determines whether or not to correct the pixels constituting the correction candidate area based on the width of the correction candidate area and the diameter of the sun image. Specifically, processor 35, the difference between the diameter of the width and the sun image correction candidate region is equal to or a predetermined threshold (second threshold) TH ₂ or more. Here, the second threshold TH ₂ is that when the difference between the width of the correction candidate region and the diameter of the sun image is less than this value, the image sensor 22 related to the correction candidate region has an abnormality due to burn-in. It is a value determined by experimentation that is considered to be.

The processor 35, when the difference between the diameter of the width and the sun image correction candidate region has a second less than the threshold value TH _2, it is determined that there is an abnormality due to seizure of the sun in the correction candidate region, corrected The candidate area is detected as an abnormal area. The processor 35, when the difference was the second threshold value TH ₂ or more, determines that there is no abnormality caused by seizure of the sun to the correction candidate region.

  When a plurality of correction candidate areas are extracted in the first determination process, the processor 35 burns in the correction candidate areas for each correction candidate area based on the size of the correction candidate area and the diameter of the sun image. It is determined whether or not it is abnormal due to.

  If the processor 35 determines that there is an abnormality in the correction candidate area, that is, detects the correction candidate area as an abnormality in the second image, the processor 35 corrects the area where the abnormality is detected (abnormal area). Specifically, the processor 35 extracts the pixel value of the pixel in the first image corresponding to each pixel constituting the abnormal region of the second image. Then, the processor 35 converts each pixel constituting the abnormal region of the second image based on the pixel extracted in the first image. At this time, the processor 35 may replace each pixel constituting the abnormal area of the second image with the pixel value extracted in the first image. Further, the processor 35 changes and replaces the gain of the pixel value extracted from the first image based on the non-abnormal region of the second image and the region of the first image corresponding to the abnormal region. Also good.

  Further, when the processor 35 determines that there is an abnormality in the correction candidate area, that is, when an abnormality is detected in the second image, the moving body 6 starts moving after the second image is captured. After that, the area corresponding to the abnormal area in the image captured by the imaging unit 2 is corrected. For example, the processor 35 can replace the pixel value so that the luminance of the pixels in the abnormal region is the average luminance of the pixels around the abnormal region.

  Next, an image processing method of the image processing apparatus 3 according to the first embodiment will be described with reference to FIG.

  First, the acquisition unit 31 acquires a first image at a first time point when the moving body 6 is stopped. For example, the communication interface 34 receives a stop signal indicating that control for stopping the moving body 6 has been started from the ECU 4 (step S1). When the stop signal is received in step S1, the acquisition unit 31 acquires the first image generated by the imaging unit 2, and the image memory 32 stores the first image (step S2).

  And the acquisition part 31 acquires a 2nd image at the 2nd time before the mobile body 6 starts a movement again after a 1st time. For example, the communication interface 34 receives from the ECU 4 a start signal indicating that control for starting the operation of the moving body 6 from the stopped state has been performed (step S3). When the start signal is received in step S3, the acquisition unit 31 acquires the second image generated by the imaging unit 2 (step S4).

  When the second image is acquired in step S4, the processor 35 determines whether or not there is a possibility of an abnormality due to the burn-in in the second image based on the first image and the second image. A first determination process is performed (step S5).

  Here, the first determination process in step S5 will be described in detail with reference to FIG.

  As illustrated in FIG. 5, the processor 35 calculates a difference between pixel values of the pixels constituting the second image and the pixels constituting the first image corresponding to the pixels (step S51).

When the difference is calculated in step S51, the processor 35 determines whether or not the calculated difference is greater than or equal to the _first threshold value TH1 (step S52).

If it is determined in step S52 that the difference is greater than or equal to the _first threshold value TH1, the processor 35 extracts pixels of the second image related to the difference greater than or equal to the _first threshold value TH1 as correction candidate pixels ( Step S53)

In step S52, when the difference is determined to be less than a first threshold value TH _1, the processor 35 is not correct candidate pixel pixel of the second image according to the first threshold value TH ₁ or more difference (step S54)

  When the process of step S53 or S54 ends, the processor 35 determines whether or not the processes of steps S51 to S54 have been performed for all the pixels of the second image (step S55).

  If it is determined in step S55 that all pixel values have not been processed, the processor 35 repeats the processes in steps S51 to S54 for pixels that have not been processed yet. When it is determined that all the pixel values have been processed, the processor 35 ends the first determination process.

  In the first determination process, in step S51, for all the pixels constituting the second image, a difference from the corresponding pixel in the first image may be calculated. In this case, in step S55, it is determined whether or not the processing in steps S52 to S54 has been performed for all the differences. If it is determined that there is a difference that has not been processed, the processor 35 has not yet processed. Steps S52 to S54 are repeated.

  Returning to FIG. 4, when the first determination process ends, the processor 35 determines whether there is a correction candidate pixel in the second image (step S <b> 6).

  If it is determined in step S6 that there is no correction candidate pixel in the second image, the processor 35 displays the second image without correction (step S11).

  If it is determined in step S6 that there are correction candidate pixels in the second image, the processor 35 determines whether or not there is an abnormality due to burn-in in the correction candidate region formed by the set of correction candidate pixels of the second image. A second determination process is performed (step S7).

  Specifically, as shown in FIG. 6, the processor 35 calculates the width of the correction candidate area in the second image (step S71).

  When the width of the correction candidate area is calculated in step S71, the processor 35 extracts the diameter of the sun image stored in the sun information memory 33 (step S72).

  When the width of the correction candidate area is calculated in step S71 and the diameter of the sun image is extracted in step S72, the processor 35 calculates the difference between the width of the correction candidate area and the diameter of the sun image (step S73).

When the difference is calculated in step S73, the processor 35, the difference between the diameter of the width and the sun image correction candidate region is equal to or the second threshold value TH ₂ or more (step S74).

In step S74, the case difference between the diameter of the width and the sun image correction candidate region has a second less than the threshold value TH _2, the processor 35 determines that the correction candidate region is abnormal (step S75).

If the difference between the width of the correction candidate area and the diameter of the sun image is equal to or larger than the second threshold TH ₂ in step S74, the processor 35 determines that there is no abnormality in the correction candidate area (step S76). ).

  Returning to FIG. 4, when the second determination process in step S7 is completed, the processor 35 branches the process based on the result of the second determination process (step S8).

  When it is determined that the correction candidate area is not an abnormal area (No in step S8), the processor 35 displays the second image without correction (step S11).

  When it is determined that the correction candidate area is an abnormal area (Yes in step S8), that is, when an abnormality is detected from the correction candidate area of the second image, the processor 35 captures the second image. The region where the abnormality of the image captured at the second time point is detected is corrected (step S9). Specifically, the processor 35 extracts the pixel value of the pixel in the first image corresponding to each pixel constituting the region corresponding to the region where the abnormality of the image captured after the second image is detected. To do. Then, the processor 35 converts each pixel in the region where the abnormality of the second image is detected based on the pixel value extracted in the first image. In addition, the processor 35 determines the luminance of the pixels in the abnormal area for the image captured by the imaging unit 2 after the second image is captured and after the moving body 6 starts moving. The pixel value can be replaced so as to obtain the average luminance of surrounding pixels.

  In step S <b> 9, when the image captured after the second image is captured for the region where abnormality is detected in the second image is corrected, the communication interface 34 transmits the corrected image to the display device 5. (Step S10).

  When the corrected image is output in step S10, the display device 5 displays the image output from the image processing device 3 (step S11).

  In the first embodiment, the image processing apparatus 3 is based on the first image in which the moving body 6 stops operating and the second image before the moving body 6 starts moving again. An abnormal region in the second image due to burn-in is corrected. Therefore, the image processing apparatus 3 refers to the image in which the abnormality is reduced even when the image sensor 22 of the far-infrared camera 1 is damaged by the far-infrared radiation emitted by the sun. be able to. In addition, the image processing device 3 outputs an inappropriate processing result that is affected by the abnormality of the image even when various image processing devices perform processing using the image captured after the second time point. Can be reduced.

  In the first embodiment, the processor 35 detects an abnormality in the second image based on the diameter of the sun image. That is, the image processing apparatus 3 can reduce erroneous detection of a difference between the second image and the first image, which is not caused by an abnormality due to sun burning, as being abnormal. Therefore, the image processing device 3 can accurately determine the abnormality due to the sun image in the second image.

  In the first embodiment, when it is determined by the first determination that the second image has a correction candidate area, the processor 35 performs the second determination. However, the present invention is not limited to this. For example, when the processor 35 determines that there is a correction candidate area in the second image by the first determination, the processor 35 may determine that the correction candidate area is an abnormal area without performing the second determination. .

  In the first embodiment, the processor 35 may determine the correction candidate area based on the direction of the far-infrared camera 1 in addition to the determination based on the diameter of the sun image. In this case, the communication interface 34 receives information indicating the direction of the moving body 6 from an orientation sensor or the like of the moving body 6, and the processor 35 attaches the far-infrared camera 1 to the direction of the moving body 6 and the moving body 6. The direction of the far-infrared camera 1 is calculated based on the direction. For example, when the direction of the far-infrared camera 1 is northward, it is assumed that the image sensor 22 is not damaged by image sticking because it does not directly receive far-infrared rays from the sun. Therefore, the processor 35 can determine that there is no correction candidate area in the second image. Thereby, the processor 35 can detect the abnormality of the second image with higher accuracy.

  In the first embodiment, the processor 35 corrects the image captured after the second time point based on the abnormal region determined by the first determination and the second determination. However, the present invention is not limited to this. For example, the processor 35 may store information indicating the abnormality correction area in a memory. In this case, the processor 35 corrects the abnormal region related to the first determination and the second determination made after the predetermined time and the abnormal region related to the first determination and the second determination made at the present time. can do. The predetermined time may be a time until the portion of the image sensor 22 damaged by the burn-in returns to an undamaged state, and can be determined by an experiment or the like.

  Specifically, the image processing device 3 includes a correction memory that stores the positions of the pixels in the abnormal region detected by the first determination and the second determination. Then, the processor 35 corrects the pixel at the position of the pixel in the abnormal region stored in the correction memory for the image acquired from the acquisition unit 31.

  Hereinafter, a second embodiment of the present invention will be described with reference to the drawings.

  The far-infrared camera 1 according to the second embodiment includes an imaging unit 2, an image processing device 3, and the like, similarly to the far-infrared camera 1 according to the first embodiment. The imaging unit 2 includes an imaging lens 21, an imaging element 22, a signal processing unit 23, and the like. The image processing apparatus 3 includes an acquisition unit 31, an image memory 32, a solar information memory 33, a communication interface 34, a processor 35, and the like.

  The communication interface 34 according to the second embodiment receives information indicating the position, direction, and inclination of the moving body 6 from a GPS receiver, a direction sensor, and the like that the moving body 6 has. In addition, the communication interface 34 may receive information indicating weather from any information device included in the mobile body 6. The communication interface 34 may receive information indicating the weather from a device external to the mobile body 6 via a communication network using information indicating the position of the mobile body 6.

  The processor 35 according to the second embodiment is the same as the processor 35 according to the first embodiment, but the processor 35 according to the second embodiment further calculates based on the correction candidate area and the environment information. It is determined that the correction candidate region has an abnormality due to the burning of the sun image when the locus of the sun image at least partially matches. The environmental information is information including at least one of the position, direction, and tilt of the far-infrared camera 1, the weather, and the time when the first image and the second image are captured.

  The trajectory of the sun image is a set of pixels corresponding to a portion where the far infrared ray from the sun is directly imaged in the image sensor 22. When the image sensor 22 is damaged by directly imaging the far infrared rays from the sun, a part of the imaging surface of the image sensor 22 directly images the far infrared rays from the sun, and the image sensor 22 As a result of being damaged, the second image area b2 (see FIG. 3B), which is a corresponding part, becomes an abnormal area. The abnormal region is a region where the pixel values of the pixels constituting the abnormal region are different from the pixel values when the image sensor 22 is not damaged. The trajectory of the solar image can be calculated based on a predetermined parameter.

  Specifically, when information indicating the position, direction, and inclination of the moving body 6 and information indicating the weather are received by the communication interface 34, the processor 35 determines the position, direction, and inclination of the moving body 6 and the moving body. 6, the position, direction, and tilt of the far infrared camera 1 are calculated. Then, based on the environmental information including the position, direction, and tilt of the far-infrared camera 1 and the weather, and at least one of the times when the first image and the second image are captured, the processor 35 The trajectory of the solar image after the time is calculated.

Further, in the second determination process, the processor 35 further includes an abnormality due to burn-in in the correction candidate area based on the similarity between the shape of the correction candidate area and the trajectory of the solar image calculated based on the environment information. It is determined whether or not. Here, “the similarity between the shape of the correction candidate area and the locus of the solar image calculated based on the environment information” is based on the feature amount indicating the shape of the correction candidate target area and the environment information. the difference between the feature amount indicating the trajectory of the calculated sun image representing that a predetermined threshold (third threshold) TH ₃ below. The third threshold TH ₃ is corrected when the difference between the feature amount indicating the shape of the correction candidate target region and the feature amount indicating the trajectory of the solar image calculated based on the environment information is less than this value. The part of the image sensor 22 related to the candidate area is a value determined by experiments or the like that is considered to be abnormal due to burn-in.

  Here, an example of a method in which the processor 35 determines whether or not the shape of the correction candidate area is similar to the trajectory of the solar image will be described.

  The processor 35 is based on the line segment representing the locus of the center of the solar image calculated from the position, direction, inclination, and time of the far-infrared camera 1 as the feature amount, and the representative line segment representing the correction candidate area. To determine similarity. The representative line segment is a line segment along the length direction of the correction candidate region. For example, a line passing through the middle of two approximate lines each approximating two boundaries that separate the correction candidate region and other than the correction candidate region. It can be. The representative line segment is not limited to this, and the processor 35 can determine the representative line segment by an arbitrary method.

  An example of how the processor 35 determines the similarity will be described. The processor 35 determines that the first end point of the line segment representing the locus of the center of the solar image calculated based on the environment information and the first end point of the representative line segment are within a predetermined distance and are included in the environment information. A second end point opposite to the first end point of the line segment representing the locus of the center of the solar image calculated based on the second end point opposite to the first end point of the representative line segment is predetermined. If it is within the distance, it is determined that there is similarity.

  In addition, the processor 35 represents a line segment represented as a trajectory of the solar image calculated based on the environment information in the xy coordinate system of the second image by a linear function (first linear function), and represents the representative line segment. May be represented by a linear function (second linear function), and similarity may be determined based on these linear functions. In this case, the processor 35 is configured such that the difference between the first coefficient of the first linear function and the first coefficient of the second linear function is equal to or smaller than a predetermined threshold value, and the zeroth coefficient of the first linear function and the first coefficient. When the difference from the 0th-order coefficient of the linear function of 2 is equal to or smaller than a predetermined threshold value, it is determined that the shape of the correction candidate area is similar to the locus of the solar image calculated based on the environmental information.

  Since other configurations and operations in the second embodiment are the same as those in the first embodiment, the same or corresponding components are denoted by the same reference numerals and description thereof is omitted.

  Next, an image processing method of the image processing apparatus 3 according to the second embodiment will be described with reference to FIG.

  The image processing method of the second embodiment is different only in step S7 of the image processing method of the first embodiment. Therefore, step S7 of the image processing method of the second embodiment will be described.

  As shown in FIG. 7, the processor 35 calculates the width of the correction candidate area in the second image (step S171).

  On the other hand, the communication interface 34 receives one or more of the position, direction, and inclination of the moving body 6 from the external device or the like, the date and time, and the weather, and the processor 35 receives the position, direction, and position of the far-infrared camera 1. And environmental information including any one or more of inclination, date / time, and weather (step S172).

  Steps S173 to S175 of the image processing method of the second embodiment are the same as steps S72 to S74 of the image processing method of the first embodiment.

In step S175, the difference between the diameter of the width and the sun image correction candidate area is determined to be the second threshold value TH ₂ or more, the processor 35 determines that there is no abnormality in the correction candidate region (step S178 ).

In step S175, the difference between the diameter of the width and the sun image correction candidate area is determined to be a second less than the threshold TH _2, the processor 35, the shape of the correction candidate region obtained in step S172 environment It is determined whether or not there is similarity with the locus of the sun image calculated based on the information (step S176).

  If it is determined in step S176 that the shape of the correction candidate region is similar to the locus of the sun image calculated based on the environmental information, the processor 35 determines that the correction candidate region is abnormal (step) S177). Further, when it is determined in step S85 that the shape of the correction candidate region is not similar to the trajectory of the sun image calculated based on the environment information, the processor 35 determines that there is no abnormality in the correction candidate region. (Step S178).

  As described above, in the second embodiment, the image processing apparatus 3 includes the first image in which the moving body 6 has stopped operating and the second image before the moving body 6 starts moving again. Based on the image, a region where an abnormality is detected in the second image due to the locus of the sun image is corrected. Therefore, the image processing apparatus 3 refers to the image in which the abnormality is reduced even when the image sensor 22 of the far-infrared camera 1 is damaged by the far-infrared radiation emitted by the sun. The same effect as the first embodiment can be obtained. Further, even when another image processing apparatus performs processing using an image picked up after the second time point, the output of an inappropriate processing result affected by an image abnormality is reduced. The same effect as the first embodiment can be obtained.

  In the second embodiment, the image processing device 3 detects the abnormality of the second image based on the trajectory of the sun image calculated based on the environmental information. That is, it is possible to reduce erroneous detection of a difference between the second image and the first image that is not caused by an abnormality due to image sticking of the image sensor 22 as being abnormal. Therefore, it is possible to accurately determine an abnormality in the second image.

  Although the above embodiment has been described as a representative example, it will be apparent to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments and examples, and various modifications and changes can be made without departing from the scope of the claims. For example, a plurality of constituent blocks described in the embodiments and examples can be combined into one, or one constituent block can be divided.

  In the above-described embodiment, the processor 35 performs the first determination for determining the correction candidate area by the first determination process from the entire area of the second image, and performs the second determination for the correction candidate area. This is not the case. For example, the processor 35 performs a recognition process on an object other than the sun for the second image, and determines an area where the object is recognized as an area for detecting an abnormality, that is, a target of the first determination process and the second determination process. May be excluded. As a result, it is possible to reduce erroneous detection of the difference between the objects captured in the first image and the second image as being abnormal due to the trajectory of the sun image.

DESCRIPTION OF SYMBOLS 1 Far-infrared camera 2 Imaging part 3, 7 Image processing apparatus 4 ECU
DESCRIPTION OF SYMBOLS 5 Display apparatus 6 Moving body 21 Imaging lens 22 Image pick-up element 23 Signal processing part 31 Acquisition part 32 Image memory 33 Solar information memory 34 Communication interface 35 Processor

Claims (11)

An image processing apparatus mounted on a moving body,
It is an acquisition part which acquires the image imaged by the far-infrared camera, Comprising: A 1st image is acquired at the 1st time which the said moving body has stopped, and the said moving body is after the said 1st time. An acquisition unit for acquiring the second image at a second time point before starting the movement again;
Based on the comparison between the first image and the second image, an abnormality of the second image due to a trajectory of the sun image formed on the imaging device of the far-infrared camera is detected, and the second image A processor that corrects a region where an abnormality is detected in the second image with respect to the image acquired from the acquisition unit after the time point;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the processor detects an abnormality in the second image based on a difference image generated by subtracting the first image from the second image.   The processor generates the difference image by subtracting a pixel value of a corresponding pixel of the first image from a pixel value of each pixel of the second image, and the pixel value of the difference image is a predetermined first value. 3. The method according to claim 2, wherein an area including pixels that are equal to or greater than a threshold value of 1 is extracted as a correction candidate area, and the correction candidate area is detected as an abnormality of the second image when the correction candidate area is determined to be based on a locus of the sun image. Image processing apparatus.   The image processing apparatus according to claim 3, wherein the processor determines that the correction candidate area is based on a locus of the sun image based on a shape of the correction candidate area.   The processor uses any one or more of the position and direction of the far-infrared camera, the date and time of the first time point and the second time point, and the weather from the first time point to the second time point. The image processing apparatus according to claim 1, wherein an abnormality of the second image is detected.   The processor calculates a trajectory of the sun image based on the position and direction of the far-infrared camera, the date and time of the first time point, and the second time point, and calculates the correction candidate region and the calculated sun image. The image processing apparatus according to claim 1, wherein when the locus coincides at least partially, the correction candidate region is determined to be based on the locus of the sun image.   The processor according to any one of claims 1 to 6, wherein the processor performs recognition processing of an object other than the sun on the second image, and excludes an area where the object is recognized from an area where the abnormality is detected. The image processing apparatus described. A correction memory for storing a table for associating the position and correction value for each pixel to be corrected;
The processor updates the position and correction value for each pixel stored in the table stored in the correction memory based on the detection result of the abnormality in the second image, and based on the updated table The image processing apparatus according to claim 1, wherein the image acquired from the acquisition unit is corrected. An imaging device mounted on a moving body,
An imaging unit that captures an image using far infrared rays;
A first image captured by the imaging unit is acquired at a first time point when the moving body is stopped, and after the first time point, a second time before the moving body starts moving again An acquisition unit that acquires a second image captured by the imaging unit at a time point;
Based on the comparison between the first image and the second image, an abnormality of the second image is detected due to the locus of the sun image formed on the image sensor of the imaging unit, and after the second time point. A processor that corrects a region where an abnormality is detected in the second image with respect to the image acquired from the acquisition unit;
An image processing apparatus comprising: an image processing apparatus. An imaging unit that captures an image using far infrared rays;
A first time point acquired by the imaging unit at a first time point when the moving body is stopped, and a second time point after the first time point and before the moving body starts moving again An acquisition unit for acquiring a second image captured by the imaging unit;
Based on the comparison between the first image and the second image, an abnormality of the second image is detected due to the locus of the sun image formed on the image sensor of the imaging unit, and after the second time point. A processor that corrects a region where an abnormality is detected in the second image with respect to the image acquired from the acquisition unit;
And a moving body comprising an imaging device including the image processing device. An image processing method executed by an image processing apparatus mounted on a moving body,
The image processing apparatus acquires a first image captured by a far-infrared camera at a first time point when the moving body is stopped, and the moving body starts moving again after the first time point. Acquiring a second image captured by the far-infrared camera at a second time before
The image processing device detects an abnormality of the second image due to a trajectory of a sun image formed on the imaging device of the far-infrared camera based on a comparison between the first image and the second image. An image processing method for correcting a region where an abnormality is detected in the second image with respect to the acquired image after the second time point.

Images