JP2021164116A - Imaging apparatus for vehicle, image processing method, and image processing program - Google Patents

Abstract

To provide an imaging apparatus for a vehicle that can obtain appropriate front, rear, right-side, and left-side images according to a direction of sunlight incident on the vehicle.SOLUTION: A high-dynamic-range image synthesis unit 133 synthesizes a front image, a rear image, a right-side image, and a left-side image generated on the basis of a long-time exposure image and a front image, a rear image, a right-side image, and a left-side image generated on the basis of a short-time exposure image, respectively. A sunlight incident direction determination unit (image analysis unit 14) determines the direction of sunlight incident on an imaging apparatus 100 for a vehicle. The high-dynamic-range image synthesis unit 133 increases the proportion of a direction image based on the long-time exposure image selected according to the direction of sunlight incidence determined by the sunlight incident direction determination unit among the front image, the rear image, the right-side image, and the left-side image.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image pickup device for a vehicle, an image processing method, and an image processing program for capturing a 360-degree subject in front, back, left, and right of the vehicle using a fisheye lens.

Patent Document 1 describes a vehicle imaging device (so-called drive recorder) that uses a fisheye lens to image a 360-degree subject in the front-rear and left-right directions of the vehicle.

JP-A-2018-195348

The vehicle image pickup device as described above analyzes a 360-degree captured image to determine the front and rear of the vehicle, and extracts region images corresponding to the front image region, the rear image region, and the left and right side image regions. Generates front image, rear image, right side image, and left side image in each direction. If the direction of sunlight incident on the vehicle is different, the brightness will be different on the front, rear, right side, and left side of the vehicle. It is desirable to obtain an appropriate image in each direction regardless of the incident direction of sunlight.

The present invention is a vehicle image pickup device, an image processing method, and an image capable of obtaining appropriate front image, rear image, right side image, and left side image according to the direction of sunlight incident on the vehicle. The purpose is to provide a processing program.

The present invention is a vehicle image pickup device mounted in a vehicle, in which a long-exposure image in which light is incident through a fisheye lens and a 360-degree subject is imaged in the first exposure time and the first exposure time are described. The front / rear and left / right sides of the vehicle are determined based on the imaging unit that generates a short-time exposure image of a 360-degree subject in a shorter second exposure time, the long-exposure image, and the short-time exposure image. The front / rear / left / right determination unit for determining the circumferential positions of the front image region, the rear image region, the right side image region, and the left side image region of the long-exposure image and the short-time exposure image, and the length. From the time-exposure image and the short-time exposure image, each region image of the front image region, the rear image region, the right side image region, and the left side image region is extracted, and the front image, the rear image, and the right side are extracted. An image extraction unit that generates a direction image of a square image and a left side image, and a front image, a rear image, a right side image, and a left side image that the image extraction unit generates based on the long-exposure image. The high dynamic range image synthesizing unit that the image extracting unit synthesizes the front image, the rear image, the right side image, and the left side image generated based on the short-time exposure image, and the vehicle imaging device. The high dynamic range image synthesizing unit includes a sunlight incident direction determining unit for determining the incident direction of incident sunlight, and the high dynamic range image synthesizing unit includes the front image, the rear image, the right side image, and the left side image. The ratio of the directional image based on the long-exposure image selected according to the incident direction of sunlight determined by the sunlight incident direction determination unit is calculated from the ratio of the directional image based on the non-selected long-exposure image. To provide a vehicle imaging device that also increases the number of images.

The present invention is an image processing method for an image pickup device for a vehicle mounted in a vehicle, and a long-exposure image in which an image pickup unit in which light is incident through a fisheye lens captures a 360-degree subject in a first exposure time. To generate a short-time exposure image of a 360-degree subject in a second exposure time shorter than the first exposure time, and based on the long-exposure image and the short-time exposure image, the front and rear of the vehicle. And left and right are determined, and the positions of the front image region, the rear image region, the right side image region, and the left side image region of the long exposure image and the short exposure image in the circumferential direction are determined, and the length is determined. From the time-exposure image and the short-exposure image, each region image of the front image region, the rear image region, the right side image region, and the left side image region is extracted, and the front image, the rear image, and the right side are extracted. Each direction image of the square image and the left side image is generated, and the front image, the rear image, the right side image, and the left side image generated based on the long exposure image are generated based on the short exposure image. A high dynamic range composite image is generated by synthesizing the front image, the rear image, the right side image, and the left side image, respectively, and the incident direction of the sunlight incident on the vehicle image pickup device is determined to determine the high dynamic range. When generating the range composite image, it is based on the long-exposure image selected according to the determined incident direction of sunlight among the front image, the rear image, the right side image, and the left side image. Provided is an image processing method for increasing the proportion of directional images to a proportion of directional images based on the unselected long-exposure image.

The present invention is a long-exposure image in which a 360-degree subject is imaged in a first exposure time by an imaging unit in which light is incident on a computer mounted on a vehicle imaging device mounted in a vehicle through a fisheye lens. Based on the step of generating a short-exposure image in which a 360-degree subject is imaged in a second exposure time shorter than the first exposure time, and the long-exposure image and the short-exposure image, the vehicle. The step of determining the front, back, left and right of the long-exposure image and the short-exposure image to determine the circumferential positions of the front image region, the rear image region, the right image region, and the left image region. Then, from the long-exposure image and the short-exposure image, each region image of the front image region, the rear image region, the right side image region, and the left side image region is extracted to obtain a front image. The step of generating each direction image of the rear image, the right side image, and the left side image, the front image, the rear image, the right side image, and the left side image generated based on the long exposure image, and the short time. A step of combining the front image, the rear image, the right side image, and the left side image generated based on the exposed image to generate a high dynamic range composite image, and the incident of sunlight incident on the vehicle imaging device. When the step of determining the direction and the high dynamic range composite image are generated, depending on the incident direction of the determined sunlight among the front image, the rear image, the right side image, and the left side image. Provided is an image processing program that executes a step of increasing the proportion of directional images based on the selected long-exposure image from the proportion of directional images based on the unselected long-exposure image.

According to the vehicle image pickup device, the image processing method, and the image processing program of the present invention, each direction image of the front image, the rear image, the right side image, and the left side image suitable according to the direction of sunlight incident on the vehicle. Can be obtained.

It is a perspective view which shows an example of the vehicle in which the vehicle image pickup apparatus of each embodiment is installed in a vehicle. It is a block diagram which shows the specific configuration example of the vehicle image pickup apparatus of each embodiment. It is a 360-degree image taken by the image pickup unit of the vehicle image pickup apparatus of each embodiment, and is a conceptual diagram which shows the state which correctly determined the front of the vehicle. FIG. 5 is a conceptual diagram showing an example of a state in which the front of the vehicle is erroneously determined, which is a 360-degree captured image generated by the imaging unit of the vehicle imaging device of each embodiment. It is a figure which shows an example of the front image of the state which erroneously determined the front of the vehicle shown in FIG. It is a figure which shows an example of the rear image of the state which erroneously determined the front of the vehicle shown in FIG. It is a figure which shows an example of the right side image of the state which erroneously determined the front of the vehicle shown in FIG. It is a figure which shows an example of the left side image of the state which erroneously determined the front of the vehicle shown in FIG. It is a figure which shows the motion vector detected in the front image, the rear image, the right side image, and the left side image in the state which correctly determined the front of the vehicle shown in FIG. It is a figure which shows the motion vector detected in the front image, the rear image, the right side image, and the left side image in the state where the front of the vehicle shown in FIG. 4 is erroneously determined. It is a flowchart which shows the process executed in 1st Embodiment. It is a flowchart which shows the process which is executed following the flowchart shown in FIG. 8 in 1st Embodiment. It is a figure which shows an example of the front image in the state which adjusted so that the direction of the front image is matched with the front of a vehicle. It is a figure which shows an example of the rear image in the state which adjusted so that the direction of the front image is matched with the front of a vehicle. It is a figure which shows an example of the right side image in the state which adjusted so that the direction of the front image is matched with the front of a vehicle. It is a figure which shows an example of the left side image in the state which adjusted so that the direction of the front image coincides with the front of a vehicle. FIG. 5 is a conceptual diagram showing blurring of an image of a subject in a right-side image or a left-side image in the vehicle image pickup device of the second embodiment. It is a conceptual diagram which shows the state which corrected the blur of the image of the subject in the right side image or the left side image in the vehicle image pickup apparatus of 2nd Embodiment. It is a flowchart which shows the process executed in 2nd Embodiment. It is a figure which shows an example of the right-hand side image which detects a plurality of motion vectors which have different movement directions in the vehicle image pickup apparatus of 3rd Embodiment. It is a figure which shows an example of the left side image in which a single motion vector is detected in the vehicle image pickup apparatus of 3rd Embodiment. It is a flowchart which shows the process executed in 3rd Embodiment. A state in which the vehicle is traveling in front of and behind the own vehicle, the vehicle is traveling in the right and left lanes of the lane in which the own vehicle is traveling, and sunlight is illuminating each vehicle from the direction of the white arrow. It is a conceptual diagram which shows. It is a figure which shows the state which divided the incident direction of sunlight into 8 in 4th Embodiment. It is a partial flowchart which shows the process executed in 4th Embodiment. FIG. 18A is a partial flowchart showing the process executed in the fourth embodiment, following FIG. 18A. It is a conceptual diagram showing a state in which a vehicle approaches a tunnel, enters the tunnel, travels in the tunnel, and exits the tunnel. It is a partial flowchart which shows the process executed in 5th Embodiment. FIG. 20A is a partial flowchart showing the process executed in the fifth embodiment following FIG. 20A. It is a block diagram which shows the specific configuration example of the vehicle image pickup apparatus of 6th Embodiment. It is a flowchart which shows the process executed in 6th Embodiment. It is a conceptual diagram which shows the instrument panel area and the non-instrument panel area set in the vehicle interior image area in 7th Embodiment. It is a flowchart which shows the process executed in 7th Embodiment. It is a conceptual diagram which shows the 1st exposure time when generating a long-exposure image and the 2nd exposure time when generating a short-exposure image.

Hereinafter, the vehicle imaging device, the image processing method, and the image processing program of each embodiment will be described with reference to the accompanying drawings.

<First Embodiment>
In FIG. 1, a vehicle imaging device 100 is attached to the upper part of the windshield 201 in the vehicle 200. The vehicle image pickup device 100 is a so-called drive recorder. The mounting position of the vehicle image pickup device 100 is not limited to the windshield 201. The vehicle image pickup device 100 includes a fisheye lens 11 and images 360 degrees of the front and rear and left and right of the vehicle 200 including the inside of the vehicle as viewed from above.

FIG. 2 shows a specific configuration example of the vehicle imaging device 100. The vehicle imaging device 100 includes a fisheye lens 11, an imaging unit 12, an image processing unit 13, an image analysis unit 14, a communication unit 15, a control unit 16, a recording / playback unit 17, and a monitor 18, and the imaging units 12 to 18 are buses. It is connected at 19.

The image processing unit 13 includes an image rotation unit 131, an image extraction unit 132, and a high dynamic range image composition unit 133. Hereinafter, the high dynamic range is abbreviated as HDR. The image analysis unit 14 includes a motion vector detection unit 141 and a front / rear / left / right determination unit 142. The control unit 16 includes a measurement unit 161. The recording / reproducing unit 17 has a removable memory card 170 as a recording medium (storage medium). The recording medium is not limited to the memory card.

The image pickup unit 12 includes an image pickup device that is a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging unit 12 generates an image (all-around image) in which a 360-degree subject is imaged by light indicated by a dashed-dotted line from the subject incident through the fisheye lens 11.

Taking the case where the vehicle image pickup device 100 generates a captured image at 60 frames / second as an example, the imaging unit 12 images a subject at 120 frames / second. At this time, the imaging unit 12 exposes one of the two adjacent frames with the first exposure time (long time) and the other with the second exposure time (short time) shorter than the first time. .. The imaging unit 12 includes an electronic shutter, and the exposure time can be arbitrarily changed by being controlled by the control unit 16. The imaging unit 12 generates a long-exposure image and a short-exposure image as a 360-degree image.

The 360-degree long-exposure image and the short-exposure image generated by the imaging unit 12 are supplied to the image processing unit 13 and the image analysis unit 14.

As conceptually shown in FIG. 3, the 360-degree captured image generated by the image pickup unit 12 is generated by a circular pixel region of a part of the rectangular pixel region of the image pickup device. The captured image generated by the circular pixel region includes an in-vehicle image area viewed from above inside the vehicle 200, a front image area around the in-vehicle image area, a rear image area, a right side image area, and a left side image area. including. Since the 360-degree captured image is an captured image obtained by the imaging unit 12 capturing the subject through the fisheye lens 11, the images in the front image region, the rear image region, the right side image region, and the left side image region are greatly distorted.

The positions of the front image area, the rear image area, the right side image area, and the left side image area shown in FIG. 3 in the circumferential direction are merely examples. Which position in the circumferential direction is to be the front image area, the rear image area, the right side image area, or the left side image area is based on the motion vector detected by the motion vector detection unit 141 and the determination by the front / rear / left / right determination unit 142. It will be finally decided. It is assumed that the front image area shown in FIG. 3 is an image showing the actual front of the vehicle 200, which is correctly determined and is indicated by a broken line rectangle in the direction of the windshield 201 of the vehicle 200.

When the vehicle imaging device 100 is first activated, the HDR image compositing unit 133 synthesizes a 360-degree captured image of a long-exposure image and a short-exposure image, and synthesizes 60 frames / sec with an expanded dynamic range. Generate an image (HDR composite image). The HDR image synthesis unit 133 mixes the long-exposure image and the short-exposure image at a predetermined ratio according to the brightness of the image analyzed by the image analysis unit 14 under the control of the control unit 16. However, in the first embodiment, the HDR image compositing unit 133 may be omitted.

The front / rear / left / right determination unit 142 determines the front / rear and left / right of the vehicle 200 based on the composite image generated by the HDR image composition unit 133 based on the control by the control unit 16, and determines the front / rear image area, the rear image area, and the right side. Temporarily determine the position of the image area and the left image area in the circumferential direction. Since the front-back, left-right directions may be corrected by the processing described later, the front-back, left-right directions are tentatively determined here.

The front / rear / left / right determination unit 142 determines the front of the vehicle 200 based on, for example, an image of the steering wheel 202 of the vehicle 200. However, the state of the image of the steering wheel 202 changes depending on the mounting position, mounting angle, and orientation of the vehicle imaging device 100. The mounting position is whether the vehicle image pickup device 100 is mounted in front of the steering wheel 202 or is mounted in the rear. The mounting angle is what kind of angle the image pickup center of the vehicle image pickup device 100 is with respect to the vertical direction. The orientation is the direction in which the vehicle imaging device 100 is mounted in the circumferential direction.

The front / rear / left / right determination unit 142 may erroneously determine the front of the vehicle 200 depending on the state of the image of the steering wheel 202. The front / rear / left / right determination unit 142 may determine the front of the vehicle 200 based on an image other than the steering wheel 202 of the vehicle 200. In this case as well, the front / rear / left / right determination unit 142 may erroneously determine the front of the vehicle 200.

As shown in FIG. 4, it is assumed that the front / rear / left / right determination unit 142 erroneously determines that the direction deviated from the traveling direction of the vehicle 200 to the left by a predetermined angle is forward. In this case, when the front / rear / left / right determination unit 142 determines the positions of the front image area, the rear image area, the right side image area, and the left side image area in the circumferential direction, the front image area is the actual vehicle 200 shown by the broken line rectangle. It does not match the image showing the front.

Here, the case where the front image region is deviated by a predetermined angle from the traveling direction of the vehicle 200 to the left is shown. Depending on the mounting position of the vehicle image pickup device 100, the rear of the vehicle 200 may be erroneously determined to be the front, or the right side or the left side may be erroneously determined to be the front. It is also possible to mistakenly determine the position as the forward image area.

As will be described later, the image rotation unit 131 of the image processing unit 13 may rotate the captured image at 360 degrees based on the control by the control unit 16. The image extraction unit 132 has each area image based on the front image area, the rear image area, the right side image area, and the left side image area in which the front / rear / left / right determination unit 142 determines the position in the circumferential direction based on the control by the control unit 16. Is extracted. The image extraction unit 132 extracts a greatly distorted front image area, a rear image area, a right side image area, and a left side so that a person approaches the state in which the front, rear, left side, and right side of the vehicle 200 are normally viewed. The distortion caused by the fisheye lens 11 of each region image of the square image region is corrected.

5A to 5D conceptually show an example of a front image 20F, a rear image 20B, a right side image 20R, and a left side image 20L in a state where the image extraction unit 132 extracts each region image and corrects distortion. There is. 5A to 5D show a state in which the front / rear / left / right determination unit 142 erroneously determines that the direction deviated from the traveling direction of the vehicle 200 to the left by a predetermined angle is forward, and does not correct the angle deviation. ..

As shown in FIG. 5A, the front image 20F is shifted to the left A pillar 203AL side. As shown in FIG. 5B, the rear image 20B includes the driver 300 and is offset to the right of the vehicle 200. As shown in FIG. 5C, the right side image 20R is shifted to the front side. As shown in FIG. 5D, the left side image 20L is shifted to the rear side.

The motion vector detection unit 141 detects motion vectors of each of the front image region, the rear image region, the right image region, and the left image region. The motion vector detection unit 141 basically detects the motion vector based on the short-exposure image. The motion vector detection unit 141 detects the motion vector based on the long-exposure image when the motion vector cannot be detected based on the short-exposure image or when it is difficult to detect the motion vector. The motion vectors of the front image area, the rear image area, the right side image area, and the left side image area are referred to as MVF, MVB, MVR, and MVL, respectively.

FIG. 6 shows motion vectors MVF, MVB, MVR, and MVL in a state where the front of the vehicle 200 is correctly determined as shown in FIG. 3 and the front image 20F coincides with the front of the actual vehicle 200. In FIG. 6 and FIG. 7 described later, the right-angled image 20R and the left-sided image 20L are rotated by 90 degrees. The motion vector MVF in the forward image 20F diverges radially with one point at which divergence starts as a vanishing point Pv1. The motion vector MVB in the rear image 20B converges with one point that converges as a vanishing point Pv2. The motion vector MVR of the right image 20R and the motion vector MVL of the left image 20L go from the front to the rear.

If the front image 20F coincides with the front of the actual vehicle 200, the vanishing point Pv1 of the front image 20F is located at approximately the center Hctr in the left-right direction of the front image 20F.

In FIG. 7, as shown in FIG. 4, the front of the vehicle 200 is erroneously determined, and the motion vectors MVF, MVB, MVR, and MVL in a state where the front image 20F is shifted to the left by a predetermined angle from the actual front of the vehicle. Is shown. Since the front image 20F does not match the front of the actual vehicle 200 and is shifted to the left by a predetermined angle, the vanishing point Pv1 of the front image 20F is larger than the substantially central Hctr in the left-right direction of the front image 20F. Located on the right side.

In the vehicle imaging device 100 of the first embodiment, FIG. 4 shows the image rotating unit 131 so that the vanishing point Pv1 coincides with the central Hctr in order to eliminate the deviation between the front image 20F and the actual vehicle front. The 360-degree captured image shown is rotated to the right. Then, as shown in FIG. 3, the front image area coincides with the front of the actual vehicle 200 shown by the broken line rectangle, and the front image 20F is an image showing the front of the vehicle 200.

The image extraction unit 132 extracts each area image of the front image area, the rear image area, the right side image area, and the left side image area in a state where the deviation between the front image 20F and the actual vehicle front is eliminated. A front image 20F, a rear image 20B, a right side image 20R, and a left side image 20L with distortion corrected are generated.

When the deviation between the front image 20F and the actual vehicle front is eliminated, the front image 20F, the rear image 20B, the right image 20R, and the left image 20L are in a state equivalent to that in FIG. 6 in which the front of the vehicle 200 is correctly determined. Become.

Here, as an example, the image rotating unit 131 eliminates the deviation in a state where the front image 20F is displaced from the traveling direction of the vehicle 200 to the left by a predetermined angle, and the image extracting unit 132 extracts each region image. There is. As described above, a position in the circumferential direction completely different from the front of the vehicle 200 may be erroneously determined as the front image area. In this case, the vanishing point Pv1 does not exist in the front image 20F extracted based on the erroneous front image region. Even if the vanishing point Pv1 does not exist in the front image 20F, the image rotating unit 131 may rotate the captured image at 360 degrees so that the vanishing point Pv1 coincides with the central Hctr of the front image 20F. ..

As described above, the image rotating unit 131 rotates the captured image of 360 degrees to eliminate the deviation between the front image 20F and the actual vehicle front, and the image extracting unit 132 extracts each region image and corrects the distortion. do. The HDR image compositing unit 133 synthesizes the front image 20F, the rear image 20B, the right side image 20R, and the left side image 20L in which the distortion of the long exposure image and the short exposure image is corrected.

The control unit 16 controls to record the front image 20F, the rear image 20B, the right side image 20R, and the left side image 20L after the image composition by the HDR image composition unit 133 on the memory card 170. Even if the control unit 16 controls to record the 360-degree captured image of the long-exposure image and the short-exposure image generated by the imaging unit 12 as shown in FIG. 4 in the long-exposure image and the short-exposure image. good. The control unit 16 may control to record both the front image 20F, the rear image 20B, the right side image 20R, the left side image 20L, and the 360-degree captured image.

The control unit 16 may control the monitor 18 to display a quadrant image in which the front image 20F, the rear image 20B, the right side image 20R, and the left side image 20L are arranged in the frame. The control unit 16 may control the monitor 18 to display a two-divided image in which the rear image 20B and the right side image 20R are arranged in the frame. The control unit 16 may control the monitor 18 to display only one selected image (for example, the front image 20F) of the front image 20F, the rear image 20B, the right side image 20R, and the left side image 20L.

The communication unit 15 may transmit the image captured by the vehicle imaging device 100 or the image recorded by the recording / reproducing unit 17 to the outside based on the control by the control unit 16. The communication unit 15 may be omitted.

The processing executed by the image processing unit 13, the image analysis unit 14, or the control unit 16 will be described with reference to the flowcharts shown in FIGS. 8 and 9. In FIG. 8, when the process is started, the front / rear / left / right determination unit 142 tentatively determines the front / rear / left / right directions based on the captured image (composite image output from the HDR image composition unit 133) in step S1.

In step S2, the image analysis unit 14 determines whether or not the vehicle 200 has moved. The image analysis unit 14 can determine that the vehicle 200 has moved when the motion vector detection unit 141 detects the motion vector. The control unit 16 may determine whether or not the vehicle 200 has moved based on the information supplied from the CAN (Controller Area Network) of the vehicle 200.

If the vehicle 200 does not move (NO), the image analysis unit 14 (or control unit 16) repeats the process of step S2. If the vehicle 200 moves (YES), the image rotating unit 131 rotates the captured image of 360 degrees based on the vanishing point Pv1 of the diverging motion vector in step S3 to adjust the direction of the front image 20F. And end the process once.

In FIG. 9, when the process is started, the motion vector detection unit 141 detects the vanishing point Pv1 of the diverging motion vector and the vanishing point Pv2 of the converging motion vector in step S11. In step S12, the measuring unit 161 measures the generation time of the vanishing point Pv1 and the generation time of the vanishing point Pv2 located in the front image 20F.

In step S13, the control unit 16 determines whether or not a predetermined time has elapsed. The predetermined time is, for example, 10 minutes. If the predetermined time has not elapsed (NO), the control unit 16 repeats the process of step S13. If the predetermined time has elapsed (YES), the control unit 16 determines in step S14 whether or not the generation time of the vanishing point Pv1 is longer than the generation time of the vanishing point Pv2. The occurrence time is the total occurrence time within a predetermined time.

If the occurrence time of the vanishing point Pv1 is longer than the occurrence time of the vanishing point Pv2 in step S14 (YES), the front image 20F whose direction is adjusted in step S3 of FIG. 8 coincides with the front of the actual vehicle 200. It is determined that there is. Therefore, the control unit 16, the image processing unit 13, and the image analysis unit 14 end the processing as they are.

In this case, the HDR image compositing unit 133 synthesizes the front image 20F, the rear image 20B, the right side image 20R, and the left side image 20L of the long-exposure image and the short-time exposure image whose directions are adjusted in step S3. Output.

If the occurrence time of the vanishing point Pv1 is not longer than the occurrence time of the vanishing point Pv2 in step S14 (NO), the front image 20F whose direction is adjusted in step S3 of FIG. 8 is not in front of the actual vehicle 200. It is determined to be backward. This is because the vehicle 200 retracted in step S2 of FIG. 8 and the direction of the front image 20F was adjusted based on the vanishing point Pv1 temporarily generated in the image behind the vehicle 200.

Therefore, if the generation time of the vanishing point Pv1 is not longer than the generation time of the vanishing point Pv2 in step S14, the front-rear / left-right determination unit 142 reverses the front-rear direction in step S15. That is, the front / rear / left / right determination unit 142 inverts the front image area and the rear image area, and inverts the right side image area and the left side image area. In step S16, the image rotation unit 131 rotates the captured image at 360 degrees based on the vanishing point Pv1, adjusts the direction of the front image 20F, and ends the process.

In this case, the HDR image synthesizing unit 133 reverses the front-back direction in step S15 and adjusts the direction in step S16. , The left side image 20L is image-combined and output.

It is not essential to provide the process of step S16, but it is preferable to provide it.

10A to 10D show the front image 20F, the rear image 20B, the right side image 20R, and the left side image 20L in a state where the direction of the front image 20F is adjusted to match the front of the vehicle 200 by the above processing. An example is conceptually shown. As shown in FIG. 10A, the deviation of the front image 20F toward the A pillar 203AL on the left side has been eliminated. As shown in FIG. 10B, the deviation of the rear image 20B to the right side of the vehicle 200 has been eliminated. As shown in FIG. 10C, the deviation of the right side image 20R toward the front side is eliminated. As shown in FIG. 10D, the shift of the left side image 20L to the rear side is eliminated.

In the first embodiment, the generation of the right side image 20R and the left side image 20L may be omitted, and only the front image 20F and the rear image 20B may be generated.

As described above, the vehicle image pickup device 100 of the first embodiment includes an image rotation unit 131, an image extraction unit 132, a front / rear / left / right determination unit 142, a motion vector detection unit 141, and a measurement unit 161. The motion vector detection unit 141 detects the motion vector of the captured image obtained by the imaging unit 12 capturing a 360-degree subject. The front / rear / left / right determination unit 142 determines the front / rear and left / right of the vehicle 200 based on the captured image, and determines the positions of at least the front image region and the rear image region of the captured image in the circumferential direction.

The image rotation unit 131 is based on the vanishing point Pv1 (first vanishing point) of a plurality of motion vectors diverging radially among the motion vectors detected by the motion vector detection unit 141 immediately after the vehicle 200 starts moving. And rotate the captured image. As a result, the image rotating unit 131 adjusts the positions of the front image region and the rear image region in the circumferential direction.

The measurement unit 161 measures the first occurrence time of the vanishing point Pv1 generated in the front image region within a predetermined time in a state where the image rotation unit 131 adjusts the positions of the front image region and the rear image region in the circumferential direction. do. Further, the measurement unit 161 measures the second generation time of the vanishing point Pv2 (second vanishing point) of the plurality of converging motion vectors that occurs in the front image region within the predetermined time.

The image rotation unit 131 maintains the positions of the front image region and the rear image region in the circumferential direction when the first generation time is measured longer than the second generation time by the measurement unit 161. The image rotation unit 131 rotates the captured image so that the front image region and the rear image region are inverted when the first generation time is not measured longer than the second generation time by the measurement unit 161.

The image extraction unit 132 extracts each region image of the front image region and the rear image region maintained or inverted by the image rotation unit 131 from the captured images to generate the front image 20F and the rear image 20B. do.

As described above, according to the vehicle imaging device 100 of the first embodiment, the front image region and the rear image region of the vehicle are correctly extracted from the 360-degree captured image, and the front image 20F and the rear image 20B are generated. can do.

The image rotation unit 131 rotates the captured image so as to invert the front image region and the rear image region, and then rotates the captured image based on the vanishing point Pv1 generated in the region image newly set as the front image region. Therefore, it is preferable to adjust the positions of the front image region and the rear image region in the circumferential direction.

The image extraction unit 132 corrects the distortion of the extracted front image area and the rear image area so as to bring the person closer to the state of looking at the front and the rear of the vehicle 200, and generates the front image 20F and the rear image 20B. Is preferable.

The front / rear / left / right determination unit 142 preferably determines the positions of the right side image area and the left side image area of the captured image in the circumferential direction in addition to the front image area and the rear image area. It is preferable that the image rotating unit 131 adjusts the positions of the right side image area and the left side image area in the circumferential direction in addition to the front image area and the rear image area. In addition to the front image area and the rear image area, the image extraction unit 132 extracts each area image of the right side image area and the left side image area from the captured images, and obtains the right side image 20R and the left side image 20L. It is preferable to generate.

<Second Embodiment>
A second embodiment provides a vehicle image pickup device 100 capable of examining how the blur of an image of a subject imaged by the vehicle image pickup device 100 should be corrected and appropriately correcting the blur of the image. In the second embodiment, the description of the common parts with the first embodiment will be omitted. The specific configuration of the vehicle imaging device 100 of the second embodiment is the same as that shown in FIG.

FIG. 11 conceptually shows a state in which a subject such as a sign is in front of the vehicle 200, approaches the subject as the vehicle 200 moves, passes by the side of the subject, and the subject moves relatively backward. Shown. A subject such as a sign does not appear in the front image 20F, the right side image 20R, and the rear image 20B at the same time, but a subject whose position changes relatively is shown in the front image 20F, the right side image 20R, and the rear image 20B. It is shown at the same time.

In the front image 20F and the rear image 20B, since the subject does not change much with time, the positions of the long-exposure image 21L and the short-exposure image 21S of the object hardly deviate from each other, and the image of the subject does not blur. However, in the right-side image 20R and the left-side image 20L, since the captured image of the subject located near the vehicle 200 changes significantly with time, the long-exposure image 21L and the short-time exposure are caused by the deviation of the imaging timing. The position is shifted from the image 21S, and the image of the subject is blurred.

As described above, the image of the subject in the right side image 20R and the left side image 20L is more likely to be blurred than the image of the subject in the front image 20F and the rear image 20B. Therefore, in the second embodiment, blurring is corrected only for the image of the subject in the right side image 20R and the left side image 20L as follows.

In the example shown in FIG. 11, the HDR image synthesizing unit 133 shifts the long-exposure image 21L previously obtained based on the motion vector in the right-side image 20R so as to match the position of the short-exposure image 21S. Above, the long-exposure image 21L and the short-exposure image 21S are combined. FIG. 11 shows an example in which the short exposure is given first and the short exposure is given after the long exposure, but the reverse may be true.

As shown in FIG. 12, by the process of correcting the image blurring by the HDR image synthesizing unit 133, the image 21 of the subject in which the image blurring is corrected in the right side image 20R is obtained. In the front image 20F and the rear image 20B, since the subject does not change much with time, the image 21 without blurring can be obtained without executing the process of correcting the blurring of the image by the HDR image compositing unit 133.

The process executed by the image processing unit 13 or the control unit 16 in the second embodiment will be described with reference to the flowchart shown in FIG. When the process is started, the image extraction unit 132 extracts the right side image area and the left side image area in step S21. Step S21 may be a process of extracting the front image area, the rear image area, the right side image area, and the left side image area as in the first embodiment.

In step S22, the HDR image synthesizing unit 133 synthesizes the long-exposure image and the short-exposure image in the right-side image region and the left-side image region with reference to the motion vector. As a result, the predetermined subject included in the right-side image or the left-side image based on the long-exposure image 21L and the predetermined subject included in the right-side image and the left-side image based on the short-exposure image 21S. The blurring of the image due to the deviation of the imaging timing of the image is corrected.

In step S23, the control unit 16 determines whether or not the power of the vehicle imaging device 100 has been turned off. Unless the power of the vehicle image pickup apparatus 100 is turned off (NO), the control unit 16 and the image processing unit 13 repeat the processes of steps S21 to S23. When the power of the vehicle imaging device 100 is turned off (YES), the control unit 16 ends the process.

The configuration and operation of the vehicle imaging device 100 of the second embodiment are as follows. The imaging unit 12 in the vehicle imaging device 100 of the second embodiment generates a long-exposure image in which the subject is imaged in the first exposure time and a short-time exposure image in which the subject is imaged in the second exposure time. .. The vehicle imaging device 100 of the second embodiment includes an image extraction unit 132, an HDR image composition unit 133, a motion vector detection unit 141, and a front-rear / left-right determination unit 142.

The motion vector detection unit 141 detects the motion vector of the captured image. The front / rear / left / right determination unit 142 determines the positions of at least the right side image region and the left side image region of the captured image in the circumferential direction. The image extraction unit 132 extracts each region image of the right side image region and the left side image region from the captured images to generate the right side image 20R and the left side image 20L.

The HDR image compositing unit 133 includes a right-side image 20R and a left-side image 20L generated by the image extraction unit 132 based on the long-exposure image, and a right-side image 20R and a left-side image 20L generated based on the short-time exposure image. Are synthesized respectively. At this time, the HDR image synthesis unit 133 refers to the motion vector detected by the motion vector detection unit 141 in the right side image area or the left side image area. As a result, the dynamic range of the right side image 20R or the left side image 20L is expanded, and the blurring of the subject image included in the right side image 20R or the left side image 20L is corrected.

As described above, according to the vehicle imaging device 100 of the second embodiment, the blurring of the image can be appropriately corrected.

The front / rear / left / right determination unit 142 preferably determines the positions of the front image region and the rear image region of the captured image in the circumferential direction in addition to the right image region and the left image region. In addition to the right side image area and the left side image area, the image extraction unit 132 extracts each area image of the front image area and the rear image area from the captured images to generate the front image 20F and the rear image 20B. Is preferable.

The HDR image synthesizing unit 133 includes the front image 20F and the rear image 20B generated by the image extraction unit 132 based on the long exposure image without referring to the motion vector detected by the motion vector detection unit 141, and the short exposure image. It is preferable to synthesize the front image 20F and the rear image 20B generated based on the above. As a result, the dynamic range of the front image 20F and the rear image 20B is expanded. In the second embodiment, the image rotation unit 131 may be omitted, but it is preferable to include the image rotation unit 131.

<Third Embodiment>
The third embodiment is an embodiment developed from the second embodiment. In the third embodiment, it is examined how to select the target for correcting the blur of the image among the subjects included in the image generated by the vehicle imaging device 100, and the blur of the image of the selected subject is determined. Provided is an image pickup device 100 for a vehicle that can be appropriately corrected. In the third embodiment, the description of the common parts with the second embodiment will be omitted. The specific configuration of the vehicle imaging device 100 of the third embodiment is the same as that of FIG.

14A and 14B show an example of the right side image 20R and the left side image 20L, respectively. As shown in FIG. 14A, the right-side image 20R includes, as subjects, a vehicle 230 traveling in the lane to the right of the vehicle 200, which is the own vehicle (not shown here), and a sign 31. The vehicle 230 is traveling at a higher speed than the vehicle 200, and relatively moves to the left of the image 20R on the right side (in front of the vehicle 200). The sign 31 moves relatively to the right of the right side image 20R (behind the vehicle 200). The left side image 20L includes a sign 32 as a subject. The sign 32 moves relatively to the left of the left image 20L (behind the vehicle 200).

The control unit 16 has one of the following options as a correction mode when the HDR image composition unit 133 preferentially corrects image blur when a plurality of motion vectors having different movement directions are detected. It is set. The plurality of motion vectors having different directions of motion may be two motion vectors in opposite directions. A user of the vehicle imaging device 100, such as the driver 300, selects one of the correction modes by an operation unit (not shown), and the selected correction mode is set in the control unit 16.

The correction mode 1 is a correction mode for correcting the blurring of a subject moving behind the vehicle 200. The correction mode 2 is a correction mode for correcting the blurring of a subject moving in front of the vehicle 200. The correction mode 3 is a correction mode for correcting the blurring of the subject closer to the vehicle 200. The correction mode 4 is a correction mode for correcting the blurring of a subject moving forward when the area of the subject moving forward of the vehicle 200 is equal to or larger than a predetermined ratio in the right side image 20R or the left side image 20L. be. The predetermined ratio is, for example, 20%.

Regarding the correction mode 3, if the subject is a stationary object, the subject farther from the vehicle 200 moves relatively at a low speed, and the subject closer to the vehicle 200 moves relatively at a high speed. The control unit 16 can identify the subject closer to the vehicle 200 based on the motion vector. The subject closer to the vehicle 200 may be extracted by the image analysis unit 14 performing image analysis. In this case, the control unit 16 may control the HDR image composition unit 133 so as to correct the blurring of the subject closer to the vehicle 200 based on the image analysis result by the image analysis unit 14.

The control unit 16 may individually set any of the correction modes 1 to 4 for the right side image 20R and the left side image 20L.

In FIG. 14A, if the correction mode 1 is set, the blur of the image of the marker 31 is corrected by the HDR image compositing unit 133 in the same manner as in FIGS. 11 and 12. If the correction mode 2 is set, the blur of the image of the vehicle 230 is corrected by the HDR image compositing unit 133. If the correction mode 3 is set, since the vehicle 230 is closer to the vehicle 200 than the sign 31, the blurring of the image of the vehicle 230 is corrected by the HDR image compositing unit 133.

In FIG. 14A, if the correction mode 4 is set, the area of the image of the vehicle 230 moving relatively forward of the vehicle 200 is 20% or more in the right side image 20R, so that the image of the vehicle 230 is blurred. Is corrected by the HDR image compositing unit 133.

In FIG. 14B, there is only a motion vector in which the sign 32 moves to the left. Therefore, the control unit 16 controls the HDR image synthesis unit 133 so as to correct the blurring of the image of the sign 32 by referring to the motion vector in which the sign 32 moves, regardless of the setting of the correction mode.

The process executed by the image processing unit 13 or the control unit 16 in the third embodiment will be described with reference to the flowchart shown in FIG. In FIG. 15, an example is taken when it is determined that two motion vectors in opposite directions exist after the processing is started.

In FIG. 15, the image extraction unit 132 extracts the right side image area and the left side image area in step S31. Step S31 may be a process of extracting the front image area, the rear image area, the right side image area, and the left side image area as in the first embodiment. In step S32, the control unit 16 determines whether or not the correction mode 1 is set to preferentially correct the blurring of the subject moving behind the vehicle 200. If the correction mode 1 is set (YES), the HDR image compositing unit 133 corrects the blurring of the image of the subject moving backward by referring to the backward movement vector in step S33, and performs the process. The process proceeds to step S39.

If the correction mode 1 is not set in step S32 (NO), the control unit 16 is set to the correction mode 2 in step S34 to preferentially correct the blurring of the subject moving in front of the vehicle 200. Judge whether or not. If the correction mode 2 is set (YES), the HDR image compositing unit 133 corrects the blurring of the image of the subject moving forward by referring to the forward motion vector in step S35, and performs the process. The process proceeds to step S39.

If it is not set to the correction mode 2 in step S34 (NO), the control unit 16 is set to the correction mode 3 in step S36 to preferentially correct the blurring of the subject located closer to the vehicle 200. Determine if it is. If the correction mode 3 is set (YES), the HDR image compositing unit 133 refers to the motion vector of the subject located closer to the image of the subject located closer in step S37. The blurring is corrected, and the process shifts to step S39.

If the correction mode 3 is not set in step S36 (NO), it means that the correction mode 4 is set. In step S38, the control unit 16 determines whether or not the area of the subject moving in front of the vehicle 200 is equal to or larger than a predetermined ratio. If the area of the subject moving in front of the vehicle 200 is not equal to or larger than a predetermined ratio (NO), the HDR image compositing unit 133 refers to the backward motion vector in step S33 to image the subject moving backward. The blurring is corrected, and the process shifts to step S39.

If the area of the subject moving forward of the vehicle 200 in step S38 is equal to or larger than a predetermined ratio (YES), the HDR image compositing unit 133 moves forward with reference to the forward motion vector in step S35. The blurring of the image of the subject to be performed is corrected, and the process shifts to step S39.

In step S39, the control unit 16 determines whether or not the power of the vehicle imaging device 100 has been turned off. Unless the power of the vehicle image pickup apparatus 100 is turned off (NO), the control unit 16 and the image processing unit 13 repeat the processes of steps S31 to S39. When the power of the vehicle imaging device 100 is turned off (YES), the control unit 16 ends the process.

The configuration and operation of the vehicle imaging device 100 of the third embodiment are as follows. The imaging unit 12 in the vehicle imaging device 100 of the third embodiment generates a long-exposure image in which the subject is imaged in the first exposure time and a short-time exposure image in which the subject is imaged in the second exposure time. .. The vehicle imaging device 100 of the third embodiment includes an image extraction unit 132, an HDR image composition unit 133, a motion vector detection unit 141, and a front-rear / left-right determination unit 142.

The motion vector detection unit 141 detects the motion vector of the captured image. The front / rear / left / right determination unit 142 determines the positions of at least the right side image region and the left side image region of the captured image in the circumferential direction. The image extraction unit 132 extracts each region image of the right side image region and the left side image region from the captured images to generate the right side image 20R and the left side image 20L.

The HDR image compositing unit 133 includes a right-side image 20R and a left-side image 20L generated by the image extraction unit 132 based on the long-exposure image, and a right-side image 20R and a left-side image 20L generated based on the short-time exposure image. Are synthesized respectively. At this time, the HDR image synthesis unit 133 refers to the motion vector detected by the motion vector detection unit 141 in the right side image area or the left side image area.

The motion vector detection unit 141 may detect a plurality of motion vectors different from each other in the right side image area or the left side image area. The HDR image compositing unit 133 corrects the blurring of the image of the subject determined to be the subject to be corrected according to the preset correction mode included in the right side image 20R or the left side image 20L. Refer to the motion vector of the target subject.

Therefore, according to the vehicle imaging device 100 of the third embodiment, it is possible to appropriately correct the blurring of the image of the selected subject.

The plurality of motion vectors that are different from each other may be a first motion vector to the rear of the vehicle 2000 and a second motion vector to the front of the vehicle 200. As the correction mode, a subject that moves relatively behind the vehicle 200 can be a subject to be corrected. In this case, the HDR image compositing unit 133 refers to the first motion vector and refers to the right side image 20R or the left side image 20L generated based on the long exposure image and the right side generated based on the short exposure image. Combine with image 20R or left side image 20L.

As the correction mode, a subject relatively moving in front of the vehicle 200 can be set as a subject to be corrected. In this case, the HDR image compositing unit 133 refers to the second motion vector and refers to the right side image 20R generated based on the long exposure image or the right side image generated based on the left side image and the short exposure image. Combine with 20R or left side image 20L.

When there are a plurality of subjects in the right side image 20R or the left side image 20L, in addition to the above two correction modes, as a correction mode, a subject located closer to the vehicle 200 may be a subject to be corrected. .. The HDR image compositing unit 133 refers to the first or second motion vector in which the subject located closer to the subject moves, and refers to the right side image 20R or the left side based on the long-exposure image and the short-exposure image. Image 20L is combined.

In the right-side image 20R or the left-side image 20L, there may be a first subject that moves relatively behind the vehicle 200 and a second subject that moves relatively in front of the vehicle 200. At this time, in addition to the above two or three correction modes, as the correction mode, the subject to be corrected is determined according to the ratio of the area of the second subject in the right side image 20R or the left side image 20L. You may.

If the ratio of the area of the second subject is equal to or more than a predetermined ratio of the right side image 20R or the left side image 20L, the HDR image compositing unit 133 refers to the first motion vector and refers to the long exposure image. And the right side image 20R or the left side image 20L based on the short exposure image is combined. If the area of the second subject is equal to or greater than a predetermined ratio of the right side image 20R or the left side image 20L, the HDR image synthesizing unit 133 refers to the second motion vector and refers to the long exposure image and the short exposure image. The right side image 20R or the left side image 20L based on the time-exposure image is combined.

<Fourth Embodiment>
A fourth embodiment provides a vehicle imaging device 100 capable of obtaining an appropriate front image 20F, rear image 20B, right side image 20R, and left side image 20L according to the incident direction of sunlight. In the fourth embodiment, the description of the common parts with the first embodiment will be omitted. The specific configuration of the vehicle imaging device 100 of the fourth embodiment is the same as that of FIG.

In FIG. 16, the vehicle 241 is traveling in front of the lane in which the vehicle 200, which is the own vehicle, is traveling, and the vehicle 242 is traveling behind. The vehicle 243 is running in the right lane of the lane in which the vehicle 200 is running, and the vehicle 244 is running in the left lane. Assuming that sunlight illuminates the vehicles 200 and 241 to 244 from the direction of the white arrow, the vehicles 200 and 241 to 244 are formed with a bright portion without hatching and a dark portion with hatching. In this way, a bright portion and a dark portion are formed depending on the incident direction of the sunlight when the sunlight illuminates the subject.

In the fourth embodiment, the vehicle imaging device 100 makes the method of synthesizing the long-exposure image and the short-exposure image in the HDR image synthesizing unit 133 different depending on the incident direction of sunlight.

As shown in FIG. 17, as an example, the control unit 16 divides the 360-degree direction imaged by the vehicle imaging device 100 into eight directions of 45 degrees each. The control unit 16 has the HDR image synthesizing unit 133 so as to make the method of synthesizing the long-exposure image and the short-exposure image different depending on which of the eight directions the sunlight is incident on. To control. The eight directions are front, front right, right, rear right, rear, rear left, left, and front left. The image analysis unit 14 can determine the incident direction of sunlight by analyzing the captured image. The image analysis unit 14 functions as a sunlight incident direction determining unit that determines the incident direction of sunlight.

The process executed by the image processing unit 13, the image analysis unit 14, or the control unit 16 in the fourth embodiment will be described with reference to the flowcharts shown in FIGS. 18A and 18B. In FIG. 18A, when the process is started, the image analysis unit 14 determines the incident direction of sunlight in step S401. In step S402, the control unit 16 determines whether or not the incident direction of sunlight is determined to be the forward direction by the image analysis unit 14.

If the incident direction of sunlight is determined to be the front direction in step S402 (YES), the control unit 16 determines the proportion of the long-exposure image in the front image 20F in step S403, the rear image 20B, and the right side image. The HDR image compositing unit 133 is controlled so as to increase the ratio of the long-exposure image in the 20R and the left side image 20L. This is because the rear of the vehicle 241 located in front of the vehicle 200 becomes a shadow, and the vehicle imaging device 100 images the rear of the vehicle 241. The control unit 16 shifts the process to step S418 following step S403.

As an example, when the ratio of the long-exposure image to the short-exposure image in the rear image 20B, the right side image 20R, and the left side image 20L is 4: 6, the control unit 16 controls the front image in step S403. The HDR image compositing unit 133 is controlled so that the ratio of the long-exposure image to the short-exposure image at 20F is 9: 1.

If the incident direction of sunlight is not determined to be the forward direction in step S402 (NO), the control unit 16 determines in step S404 whether or not the incident direction of sunlight is determined to be the right front direction. If the incident direction of sunlight is determined to be the front right direction (YES), the control unit 16 determines the ratio of the long-exposure image in the front image 20F and the right side image 20R in step S405 to the rear image 20B and the left side. The HDR image compositing unit 133 is controlled so as to increase the ratio of the long-exposure image in the image 20L. The control unit 16 shifts the process to step S418 following step S405.

As an example, when the ratio of the long-exposure image to the short-exposure image in the rear image 20B and the left-side image 20L is 4: 6, the control unit 16 controls the long-exposure image in the front image 20F and the right-side image 20R. The HDR image compositing unit 133 is controlled so that the ratio between the image and the short-exposure image is 9: 1.

If the incident direction of sunlight is not determined to be the front right direction in step S404 (NO), the control unit 16 determines in step S406 whether or not the incident direction of sunlight is determined to be the right direction. If it is determined that the incident direction of sunlight is to the right (YES), the control unit 16 determines the ratio of the long-exposure image in the front image 20F in step S407 to the rear image 20B, the right image 20R, and the left side. The HDR image compositing unit 133 is controlled so as to increase the ratio of the long-exposure image in the square image 20L. The method of increasing the proportion of the long-exposure image is the same as in step S405. The control unit 16 shifts the process to step S418 following step S407.

If the incident direction of sunlight is not determined to be the right direction in step S406 (NO), the control unit 16 determines in step S408 whether or not the incident direction of sunlight is determined to be the rear right direction. If it is determined that the incident direction of sunlight is the right rear direction (YES), the control unit 16 determines the ratio of the long-exposure image in the rear image 20B and the right side image 20R in step S409 in the front image 20F and the left side. The HDR image compositing unit 133 is controlled so as to increase the ratio of the long-exposure image in the square image 20L. The method of increasing the proportion of the long-exposure image is the same as in step S405. The control unit 16 shifts the process to step S418 following step S409.

If the incident direction of sunlight is not determined to be the rear right direction in step S408 (NO), the control unit 16 determines whether or not the incident direction of sunlight is determined to be the rear direction in step S410 of FIG. 18B. judge. If it is determined that the incident direction of sunlight is the rear direction (YES), the control unit 16 determines the ratio of the long-exposure image in the rear image 20B in step S411 to the front image 20F, the right side image 20R, and the left side. The HDR image compositing unit 133 is controlled so as to increase the ratio of the long-exposure image in the square image 20L. The method of increasing the proportion of the long-exposure image is the same as in step S405. The control unit 16 shifts the process to step S418 following step S411.

If the incident direction of sunlight is not determined to be the rear direction in step S410 (NO), the control unit 16 determines in step S412 whether or not the incident direction of sunlight is determined to be the rear left direction. If it is determined that the incident direction of sunlight is the left rear direction (YES), the control unit 16 determines the ratio of the long-exposure image in the rear image 20B and the left image 20L in step S413 to the front image 20F and the right side. The HDR image compositing unit 133 is controlled so as to increase the ratio of the long-exposure image in the square image 20R. The method of increasing the proportion of the long-exposure image is the same as in step S405. The control unit 16 shifts the process to step S418 following step S413.

If the incident direction of sunlight is not determined to be the left rear direction in step S412 (NO), the control unit 16 determines in step S414 whether or not the incident direction of sunlight is determined to be the left direction. If it is determined that the incident direction of sunlight is the left direction (YES), the control unit 16 determines the ratio of the long-exposure image in the left side image 20L in step S415 to the front image 20F, the rear image 20B, and the right side. The HDR image compositing unit 133 is controlled so as to increase the ratio of the long-exposure image in the square image 20R. The method of increasing the proportion of the long-exposure image is the same as in step S405. The control unit 16 shifts the process to step S418 following step S415.

If the incident direction of sunlight is not determined to be the left direction in step S414 (NO), the control unit 16 determines whether or not the incident direction of sunlight is determined to be the front left direction in step S416. If it is determined that the incident direction of sunlight is the left front direction (YES), the control unit 16 determines the ratio of the long-exposure image in the front image 20F and the left side image 20L in step S417 to the rear image 20B and the right side. The HDR image compositing unit 133 is controlled so as to increase the ratio of the long-exposure image in the image 20R. The method of increasing the proportion of the long-exposure image is the same as in step S405. The control unit 16 shifts the process to step S418 following step S417.

If the incident direction of sunlight is not determined to be the left front direction in step S416 (NO), the control unit 16 shifts the process to step S418.

In step S418, the control unit 16 determines whether or not the power of the vehicle imaging device 100 has been turned off. Unless the power of the vehicle imaging device 100 is turned off (NO), the image processing unit 13, the image analysis unit 14, or the control unit 16 repeats the processes of steps S401 to S418. When the power of the vehicle imaging device 100 is turned off (YES), the control unit 16 ends the process.

The configuration and operation of the vehicle imaging device 100 of the fourth embodiment are as follows. The imaging unit 12 in the vehicle imaging device 100 of the fourth embodiment generates a long-exposure image in which the subject is imaged in the first exposure time and a short-time exposure image in which the subject is imaged in the second exposure time. .. The vehicle imaging device 100 of the fourth embodiment includes a sunlight incident direction determination unit (image analysis unit 14), an image extraction unit 132, an HDR image composition unit 133, and a front / rear / left / right determination unit 142.

The front / rear / left / right determination unit 142 determines the positions of the front image region, the rear image region, the right side image region, and the left side image region of the captured image in the circumferential direction. The image extraction unit 132 extracts each region image of the front image region, the rear image region, the right side image region, and the left side image region from the captured images, and extracts the front image 20F, the rear image 20B, and the right side image 20R. , And each direction image of the left side image 20L is generated.

The HDR image compositing unit 133 includes a front image 20F, a rear image 20B, a right side image 20R, and a left side image 20L generated based on a long-exposure image, and a front image 20F and a rear image 20F generated based on a short-time exposure image. The image 20B, the right side image 20R, and the left side image 20L are combined.

The HDR image synthesizing unit 133 has a length selected from the front image 20F, the rear image 20B, the right side image 20R, and the left side image 20L according to the incident direction of sunlight determined by the sunlight incident direction determining unit. The proportion of directional images based on time-exposed images is increased over the proportion of directional images based on unselected long-exposure images.

According to the vehicle imaging device 100 of the fourth embodiment, it is possible to obtain an appropriate front image 20F, rear image 20B, right side image 20R, and left side image 20L according to the incident direction of sunlight.

The sunlight incident direction determination unit divides the 360-degree direction imaged by the vehicle imaging device 100 into a plurality of directions including at least the front direction, the rear direction, the right direction, and the left direction, and the direction of the sunlight incident is any one. It suffices to judge whether it is the direction of.

The HDR image compositing unit 133 increases the proportion of the front image based on the long-exposure image when the incident direction of sunlight is the front direction, and makes the long-exposure image when the incident direction of sunlight is the rear direction. It is better to increase the proportion of the underlying posterior image. The HDR image compositing unit 133 increases the proportion of the right side image based on the long-exposure image when the incident direction of sunlight is to the right, and the long-exposure image when the incident direction of sunlight is to the left. It is better to increase the proportion of the left side image based on.

The sunlight incident direction determination unit adds the 360-degree direction imaged by the vehicle imaging device 100 to the front direction, the rear direction, the right direction, and the left direction, as well as the right front direction, the right rear direction, the left rear direction, and the left front direction. It is preferable to divide into a plurality of directions including the above to determine which direction the sunlight is incident on.

The HDR image compositing unit 133 increases the ratio of the front image and the right side image based on the long-exposure image when the incident direction of sunlight is the right front direction, and when the incident direction of sunlight is the right rear direction, It is preferable to increase the proportion of the rear image and the right side image based on the long exposure image. The HDR image compositing unit 133 increases the ratio of the rear image and the left side image based on the long-exposure image when the incident direction of sunlight is the left rear direction, and when the incident direction of sunlight is the left front direction. It is preferable to increase the proportion of the front image and the left image based on the long exposure image.

<Fifth Embodiment>
In the fifth embodiment, when the vehicle 200 is traveling, the front image 20F and the rear image 20B are appropriate according to the position of the vehicle 200 from before entering the tunnel to after traveling in the tunnel and exiting the tunnel. , A vehicle imaging device 100 capable of obtaining a right side image 20R and a left side image 20L. In the fifth embodiment, the description of the common parts with the first embodiment will be omitted. The specific configuration of the vehicle imaging device 100 of the fifth embodiment is the same as that of FIG.

In FIG. 19, the vehicle 200 traveling on the road approaches the tunnel 50 and travels in the area R51. At this time, the image of the inside of the tunnel 50 included in the front image 20F becomes a predetermined ratio or more. The control unit 16 controls the ratio of the long-exposure image when the HDR image composition unit 133 combines the long-exposure image and the short-exposure image in the front image 20F so that the image inside the tunnel 50 can be clearly seen. It is preferable to control the HDR image compositing unit 133 so as to increase the size.

As an example, when the ratio of the long-exposure image to the short-exposure image is 4: 6 in a state where the captured image does not include the image inside the tunnel 50 before the vehicle 200 reaches the region R51. The control unit 16 sets the ratio of the long-exposure image to the short-exposure image in the front image 20F to 8: 2.

The vehicle 200 further approaches the entrance 50in of the tunnel 50, and reaches the area R52 where a part of the vehicle 200 enters the tunnel 50. At this time, the control unit 16 controls the HDR image synthesis unit 133 so as to increase the ratio of the long-exposure image when combining the long-exposure image and the short-exposure image in the right-side image 20R and the left-side image 20L. It is better to do it.

As an example, when the ratio of the long-exposure image to the short-exposure image in the right-side image 20R and the left-side image 20L is 4: 6 when the vehicle 200 is traveling in the region R51, the control unit 16 determines. The ratio of the long-exposure image to the short-exposure image in the right-side image 20R and the left-side image 20L is 8: 2.

The vehicle 200 further travels in the tunnel 50 and reaches the area R53. At this time, the images other than the tunnel 50 on the entrance 50in side of the tunnel 50 included in the rear image 20B are at a predetermined ratio or less. The control unit 16 preferably controls the HDR image compositing unit 133 so as to increase the ratio of the long-exposure image when composing the long-exposure image and the short-exposure image in the rear image 20B.

As an example, when the vehicle 200 is traveling in the regions R51 and R52 and the ratio of the long-exposure image to the short-exposure image in the rear image 20B is 4: 6, the control unit 16 controls the rear image 20B. The ratio of the long-exposure image to the short-exposure image is 8: 2.

The vehicle 200 further travels in the tunnel 50 and reaches the region R54 approaching the exit 50out of the tunnel 50. At this time, the images other than the tunnel 50 on the exit 50out side of the tunnel 50 included in the front image 20F have a predetermined ratio or more. The control unit 16 reduces the ratio of the long-exposure image when combining the long-exposure image and the short-exposure image in the front image 20F so that the image outside the tunnel 50 can be clearly seen. It is preferable to control the synthesis unit 133.

As an example, the ratio of the long-exposure image to the short-exposure image in the front image 20F is 8: 2 until just before the vehicle 200 reaches the region R54. Therefore, when the vehicle 200 reaches the region R54, the long-exposure image and the short-time exposure are exposed. The ratio to the image is 4: 6.

The vehicle 200 further travels in the tunnel 50 and reaches the area R55 where a part of the vehicle 200 exits the exit 50out of the tunnel 50. At this time, the control unit 16 controls the HDR image synthesis unit 133 so as to reduce the ratio of the long-exposure image when combining the long-exposure image and the short-exposure image in the right-side image 20R and the left-side image 20L. It is better to do it.

As an example, since the ratio of the long-exposure image to the short-exposure image in the right-side image 20R and the left-side image 20L is 8: 2 until just before the vehicle 200 reaches the region R55, it takes a long time after reaching the region R55. The ratio of the exposed image to the short-exposure image is 4: 6.

When the vehicle 200 completely goes out of the tunnel 50 and reaches the area R56, the image of the inside of the tunnel 50 included in the rear image 20B becomes a predetermined ratio or less. The control unit 16 controls the HDR image compositing unit 133 so that the HDR image compositing unit 133 reduces the ratio of the long-exposure image when the long-exposure image and the short-exposure image are combined in the rear image 20B. good.

As an example, since the ratio of the long-exposure image to the short-exposure image in the rear image 20B is 8: 2 until just before the vehicle 200 reaches the region R56, the long-exposure image and the short-time exposure when the vehicle 200 reaches the region R56. The ratio to the image is 4: 6.

The image analysis unit 14 can determine where the vehicle 200 is traveling in the regions R51 to R56 shown in FIG. 19 based on the captured image. The image analysis unit 14 functions as a tunnel travel determination unit.

When the vehicle imaging device 100 includes a GNSS receiving unit that receives radio waves from satellites for a global navigation satellite system (GNSS) such as GPS (Global Positioning System) and map information, a control unit. 16 may detect the position inside and outside the tunnel 50 of the vehicle 200 based on the GNSS signal received by the GNSS receiving unit and the map information. In this case, the control unit 16 functions as a tunnel travel determination unit.

The process executed by the image processing unit 13, the tunnel traveling determination unit (image analysis unit 14), or the control unit 16 will be described with reference to the flowcharts shown in FIGS. 20A and 20B in the fifth embodiment. In FIG. 20A, when the process is started, the tunnel travel determination unit recognizes the tunnel in front of the vehicle 200 in step S501, and determines whether or not the image inside the tunnel 50 has a predetermined ratio or more. judge.

If it is determined in step S501 that the image inside the tunnel 50 exceeds a predetermined ratio (YES), the control unit 16 increases the ratio of the long-exposure image in the front image 20F in step S502. The HDR image compositing unit 133 is controlled to shift the process to step S513. If it is not determined in step S501 that the image of the inside of the tunnel 50 is equal to or greater than a predetermined ratio (NO), the tunnel traveling determination unit determines in step S503 whether or not the vehicle 200 has entered the tunnel 50. judge.

If it is determined in step S503 that the vehicle 200 has entered the tunnel 50 (YES), the control unit 16 increases the proportion of the long-exposure image in the right-side image 20R and the left-side image 20L in step S504. The HDR image compositing unit 133 is controlled so that the process shifts to step S513. If it is not determined in step S503 that the vehicle 200 has entered the tunnel 50 (NO), the tunnel traveling determination unit is other than the tunnel 50 on the entrance 50in side of the tunnel 50 included in the rear image 20B in step S505. It is determined whether or not the image of is equal to or less than a predetermined ratio.

If it is determined in step S505 that the image other than the tunnel 50 on the entrance 50in side of the tunnel 50 is equal to or less than a predetermined ratio (YES), the control unit 16 determines in step S506 the long-exposure image in the rear image 20B. The HDR image compositing unit 133 is controlled so as to increase the ratio of the above, and the process shifts to step S513. If it is not determined in step S505 that the image other than the tunnel 50 on the entrance 50in side of the tunnel 50 is less than or equal to the predetermined ratio (NO), the tunnel traveling determination unit is displayed on the front image 20F in step S507 of FIG. 20B. It is determined whether or not the images other than the tunnel 50 on the exit 50 out side of the included tunnel 50 are at least a predetermined ratio.

If it is determined in step S507 that the image other than the tunnel 50 on the exit 50 out side of the tunnel 50 is equal to or greater than a predetermined ratio (YES), the control unit 16 determines in step S508 the long-exposure image in the front image 20F. The HDR image compositing unit 133 is controlled so as to reduce the ratio of the above, and the process shifts to step S513. If it is not determined in step S507 that the images other than the tunnel 50 on the exit 50 out side of the tunnel 50 are equal to or greater than the predetermined ratio (NO), the tunnel traveling determination unit determines in step S509 that a part of the vehicle 200 is tunneled. Determine if it has moved out of 50.

If it is determined in step S509 that a part of the vehicle 200 has moved out of the tunnel 50 (YES), the control unit 16 exposes the right side image 20R and the left side image 20L for a long time in step S510. The HDR image compositing unit 133 is controlled so as to reduce the proportion of the image, and the process shifts to step S513. If it is not determined in step S509 that a part of the vehicle 200 has moved out of the tunnel 50 (NO), the tunnel travel determination unit determines that the vehicle 200 has moved out of the tunnel 50 in step S511. , It is determined whether or not the image of the inside of the tunnel 50 included in the rear image 20B is equal to or less than a predetermined ratio.

If it is determined in step S511 that the image inside the tunnel 50 is less than or equal to a predetermined ratio (YES), the control unit 16 reduces the ratio of the long-exposure image in the rear image 20B in step S512. The HDR image compositing unit 133 is controlled to shift the process to step S513. If it is not determined in step S511 that the image inside the tunnel 50 is equal to or less than a predetermined ratio (NO), the control unit 16 shifts the process to step S513.

In step S513, the control unit 16 determines whether or not the power of the vehicle imaging device 100 has been turned off. Unless the power of the vehicle imaging device 100 is turned off (NO), the image processing unit 13, the tunnel travel determination unit, or the control unit 16 repeats the processes of steps S501 to S513. When the power of the vehicle imaging device 100 is turned off (YES), the control unit 16 ends the process.

The configuration and operation of the vehicle imaging device 100 of the fifth embodiment are as follows. The imaging unit 12 in the vehicle imaging device 100 of the fifth embodiment generates a long-exposure image in which the subject is imaged in the first exposure time and a short-time exposure image in which the subject is imaged in the second exposure time. .. The vehicle imaging device 100 of the fifth embodiment includes a tunnel traveling determination unit (image analysis unit 14), an image extraction unit 132, an HDR image composition unit 133, and a front / rear / left / right determination unit 142.

The front / rear / left / right determination unit 142 determines the positions of the front image region, the rear image region, the right side image region, and the left side image region of the captured image in the circumferential direction. The image extraction unit 132 extracts each region image of the front image region, the rear image region, the right side image region, and the left side image region from the captured images, and extracts the front image 20F, the rear image 20B, and the right side image 20R. , And each direction image of the left side image 20L is generated.

The HDR image compositing unit 133 includes a front image 20F, a rear image 20B, a right side image 20R, and a left side image 20L generated based on a long-exposure image, and a front image 20F and a rear image 20F generated based on a short-time exposure image. The image 20B, the right side image 20R, and the left side image 20L are combined.

The tunnel travel determination unit determines a state in which the traveling vehicle 200 approaches the tunnel 50, enters the tunnel 50, travels in the tunnel 50, and exits the tunnel 50.

The HDR image compositing unit 133 sets the ratio of the directional image based on the long-exposure image of the selected directional image among the front image 20F, the rear image 20B, the right side image 20R, and the left side image 20L, or the short-time exposure image. Adjust the proportion of directional images based on. The HDR image synthesizing unit 133 determines the ratio according to the position of the vehicle 200 from before the vehicle 200 enters the tunnel 50 to after traveling in the tunnel 50 and exiting the tunnel 50, based on the determination by the tunnel traveling determination unit. adjust.

Specifically, the HDR image compositing unit 133 may adjust the ratio as follows according to the position of the vehicle 200.

When the tunnel traveling determination unit determines that the vehicle 200 approaches the tunnel 50 and the image inside the tunnel 50 is equal to or more than a predetermined ratio of the front image 20F, the HDR image composition unit 133 is exposed for a long time. It is preferable to increase the proportion of the front image 20F based on the image.

When the tunnel traveling determination unit determines that the vehicle 200 has entered the tunnel 50, the HDR image composition unit 133 preferably increases the ratio of the right side image 20R and the left side image 20L based on the long exposure image.

When the tunnel running determination unit determines that the images other than the tunnel 50 on the entrance 50in side of the tunnel 50 are less than or equal to a predetermined ratio of the rear image 20B after the vehicle 200 has entered the tunnel 50, the HDR image composition unit 133 It is preferable to increase the proportion of the rear image 20B based on the long-exposure image.

When the tunnel traveling determination unit determines that the vehicle 200 approaches the exit 50out of the tunnel 50 and the image other than the tunnel 50 on the exit 50out side becomes a predetermined ratio or more of the front image 20F, the HDR image composition unit For 133, it is preferable to reduce the proportion of the front image 20F based on the long-exposure image. That is, the ratio of the front image 20F based on the long-exposure image increased when the vehicle 200 approaches the tunnel 50 as described above is returned to the original ratio.

When the tunnel traveling determination unit determines that the vehicle 200 has gone out of the tunnel 50, the HDR image composition unit 133 preferably reduces the ratio of the right side image 20R and the left side image 20L based on the long exposure image. .. That is, the ratio of the right side image 20R and the left side image 20L based on the long-exposure image increased when the vehicle 200 enters the tunnel 50 as described above is returned to the original ratio.

When the tunnel traveling determination unit determines that the vehicle 200 has gone out of the tunnel 50 and the image inside the tunnel 50 is less than or equal to a predetermined ratio of the rear image 20B, the HDR image composition unit 133 has a long time. It is preferable to reduce the proportion of the rear image 20B based on the exposed image. That is, the ratio of the rear image 20B based on the long-exposure image increased when the image other than the tunnel 50 on the entrance 50in side of the tunnel 50 becomes equal to or less than a predetermined ratio of the rear image 20B is returned to the original ratio. ..

As described above, according to the vehicle imaging device 100 of the fifth embodiment, an appropriate front image 20F, depending on the position of the vehicle 200 from before the vehicle 200 enters the tunnel 50 to after the vehicle exits the tunnel 50, A rear image 20B, a right side image 20R, and a left side image 20L can be obtained.

<Sixth Embodiment>
A sixth embodiment provides a vehicle image pickup device 100 capable of storing a captured image with less blurring when an event such as an accident occurs in the vehicle 200. In the sixth embodiment, the description of the common parts with the first embodiment will be omitted.

As shown in FIG. 21, the vehicle imaging device 100 of the sixth embodiment includes an acceleration sensor 60 connected to the bus 19. The acceleration sensor 60 is an example of an event detection sensor that determines whether or not an event has occurred in the vehicle 200. The recording / playback unit 17 has a memory card 170 and a ring buffer 171. The ring buffer 171 is a non-detachable main body memory included in the vehicle imaging device 100. The memory card 170 is provided with an event recording area 172 and a normal recording area 173. Event recording area 172 This is a recording area in which data is not automatically overwritten.

The event recording area 172 and the normal recording area 173 are not limited to being provided in the memory card 170. The event recording area 172 may be provided in the main body memory. The normal recording area 173 may be provided in the main body memory. The event recording area 172 and the normal recording area 173 may be provided in the main body memory.

The ring buffer 171 has a capacity for recording captured image data of a 360-degree captured image of a long-exposure image and a short-exposure image for a predetermined time, and the captured image data is cyclically stored in the ring buffer 171. .. That is, when the captured image data is stored in all the capacities of the ring buffer 171, the oldest captured image data is rewritten to the latest captured image data, and the operation of updating the captured image data is repeated.

The front image 20F, the rear image 20B, the right side image 20R, and the left side image 20L image-combined by the HDR image compositing unit 133 are recorded in the normal recording area 173 without being temporarily recorded in the ring buffer 171. ..

The process executed by the control unit 16 in the sixth embodiment will be described with reference to the flowchart shown in FIG. When the process is started, the control unit 16 stores the captured image data in the ring buffer 171 for a predetermined time in step S61. As described above, the captured image data is the captured image data of the 360-degree captured image of the long-exposure image and the short-exposure image.

In step S62, the control unit 16 determines whether or not an event has occurred based on the change in acceleration detected by the acceleration sensor 60. The control unit 16 determines that an event has occurred when the acceleration suddenly increases or decreases.

If it is not determined in step S62 that an event has occurred (NO), the control unit 16 repeats the processes of steps S61 and S62. If it is determined in step S62 that an event has occurred (YES), the control unit 16 shifts the process to step S63. In step S63, the control unit 16 copies the captured image data stored in the ring buffer 171 to the event recording area 172, and ends the process.

When an event occurs, the subject that is desired to be recorded in the normal recording area 173 without blurring of the image may be recorded in a blurred state. Specifically, as an example, in the following cases, the image is recorded in the normal recording area 173 with a large amount of blurring.

In FIG. 14A, it is assumed that the area of the vehicle 230 in the right side image 20R is less than 20%. In this case, the HDR image compositing unit 133 corrects the blurring of the image of the sign 31 that moves relatively backward in the right side image 20R. If an accident occurs between the vehicle 200 and the vehicle 230 in this state, the image blur of the vehicle 230 is not corrected by the HDR image compositing unit 133, so that the image is recorded in the normal recording area 173 in a state where the image blur is large. ..

Therefore, when an event occurs, it is not sufficient to simply store the composite image of the long-exposure image and the short-exposure image by the HDR image compositing unit 133 in the normal recording area 173. When an event occurs, it is preferable to copy and save the captured image data of the 360-degree captured image of the long-exposure image and the short-exposure image before the image is synthesized by the HDR image compositing unit 133 in the event recording area 172. According to the sixth embodiment, when an event such as an accident occurs in the vehicle 200, a captured image with less blurring can be stored.

In the vehicle image pickup apparatus 100 of the sixth embodiment, the image data of each direction image synthesized by the HDR image synthesis unit 133 is also stored in the ring buffer 171 and copied and saved in the event recording area 172 when an event occurs. You may do so. Further, in the vehicle image pickup apparatus 100 of the sixth embodiment, each direction image synthesized by the HDR image synthesis unit 133 may be configured not to be recorded in the ring buffer 171 or the normal recording area 173.

As described above, in the vehicle imaging device 100 of the sixth embodiment, the imaging unit 12 generates a long-exposure image and a short-exposure image. The vehicle imaging device 100 of the sixth embodiment includes an image extraction unit 132 and an HDR image composition unit 133. The image extraction unit 132 generates a front image, a rear image, a right side image, and a left side image based on the long exposure image and the short exposure image. The HDR image compositing unit 133 synthesizes the front image 20F, the rear image 20B, the right side image 20R, and the left side image 20L based on the long exposure image and the short exposure image.

Further, the vehicle imaging device 100 of the sixth embodiment includes a ring buffer 171 and an event detection sensor (accelerometer 60). The ring buffer 171 periodically stores the long-exposure image and the short-exposure image. The event detection sensor determines whether or not an event has occurred in the vehicle 200. When the event detection sensor determines that an event has occurred in the vehicle 200, the control unit 16 copies and saves the long-exposure image and the short-time exposure image stored in the ring buffer 171 in the event recording area 172. To control.

According to the vehicle imaging device 100 of the sixth embodiment, it is possible to store an captured image with less blurring when an event such as an accident occurs in the vehicle 200.

<7th Embodiment>
A seventh embodiment provides a vehicle imaging device 100 capable of generating an appropriate in-vehicle image. FIG. 23 shows a 360-degree captured image generated by the imaging unit 12 in a state where the direction of the front image 20F is correctly adjusted to the front of the vehicle 200 by the processes shown in FIGS. 8 and 9. The in-vehicle image area includes an instrument panel with instruments. The image near the instrument panel is important in the in-vehicle image area. Therefore, the vehicle imaging device 100 of the seventh embodiment executes the process shown in FIG. 24.

In step S71, the control unit 16 determines whether or not the front-rear direction of the vehicle image pickup device 100 (vehicle 200) has been determined and the directions of the front image 20F and the rear image 20B have been adjusted. If it is not determined that the front-rear direction has been determined (NO), the control unit 16 repeats the process of step S71 until the processes shown in FIGS. 8 and 9 of the first embodiment are completed.

If it is determined in step S71 that the front-rear direction has been determined (YES), the control unit 16 sets the instrument panel area 71 in front of the vehicle interior image area as shown in FIG. 23 in step S72. The control unit 16 can set a predetermined area in front of the vehicle interior image area as the instrument panel area 71. The image analysis unit 14 may detect the instrument panel by image analysis, and the control unit 16 may set the instrument panel area 71 based on the detection result by the image analysis unit 14.

The control unit 16 sets an area for setting the instrument panel area 71 and the non-instrument panel area 72, which is an area other than the instrument panel area 71, in the vehicle interior image area in the long-exposure image and the short-time exposure image. Functions as a department.

In step S73, the control unit 16 optimizes the vehicle interior image so that the instrument panel region 71 is clearly captured. Specifically, when the control unit 16 generates a long-exposure image and a short-exposure image so that the high-brightness image displayed by the LED or the like in the instrument panel area 71 is not captured in an overexposed state. The imaging unit 12 is controlled so as to optimize the exposure time of the image.

The first exposure time when generating a long-exposure image and the second exposure time when generating a short-exposure image are set as follows. The first and second exposure times have a maximum time and a minimum time, and the first and second exposure times are set to a time between the maximum time and the shortest time determined according to the brightness of the image. ..

As shown in FIG. 25 (a), it is assumed that the ratio of the first exposure time of the long exposure to the second exposure time of the short exposure is, for example, 10: 1. It is assumed that the interior of the vehicle 200 is dark and the ratio of the optimum exposure time in the non-instrument panel region 72 is 100% of the maximum time as shown in FIG. 25 (b). It is assumed that the instrument panel region 71 is brighter than the non-instrument panel region 72, and the ratio of the optimum exposure time in the instrument panel region 71 is 50% of the maximum time as shown in FIG. 25 (c).

Usually, when a bright region and a dark region exist in one image region, the optimum exposure time ratio is determined in consideration of the area of the bright region and the area of the dark region. It is assumed that the area of the instrument panel area 71 and the non-instrument panel area 72 is 1: 9. In this case, the control unit 16 uses a calculation formula of 9/10 × 100% + 1/10 × 50% = 95% to shorten the first exposure time of the long exposure as shown in FIG. 25 (d). The ratio of the optimum exposure time to the second exposure time of the time exposure is determined to be 95%. However, if the optimum exposure time ratio is determined to be 95%, overexposure will occur in the instrument panel region 71.

Therefore, the control unit 16 optimizes both of them based on 50%, which is the ratio of the optimum exposure time in the instrument panel region 71, and 100%, which is the ratio of the optimum exposure time in the non-instrument panel region 72. It is better to determine the ratio of exposure time. The ratio of the optimum exposure time in the instrument panel region 71 is defined as the first optimum exposure time ratio, and the ratio of the optimum exposure time in the non-instrument panel region 72 is defined as the second optimum exposure time ratio. The control unit 16 sets a third optimum exposure time ratio, which is a ratio of exposure time common to the instrument panel region 71 and the non-instrument panel region 72.

At this time, the control unit 16 does not consider the area ratio of the instrument panel area 71 and the non-instrument panel area 72. As shown in FIG. 25 (e), the control unit 16 sets, as an example, 75%, which is an average value of 50% and 100%, as the third optimum exposure time ratio. Since the exposure time in the instrument panel region 71 is shortened from 95% to 75%, the possibility of overexposure can be reduced.

Here, the case where the non-instrument panel area 72 is dark and the instrument panel area 71 is bright is taken as an example, but depending on the relationship between the brightness of the instrument panel area 71 and the non-instrument panel area 72, it is as follows. It is preferable to set the upper limit of the ratio of the exposure time in the instrument panel area 71.

The shortest exposure time ratio under the condition that the image in the instrument panel area 71 is not overexposed is T71S, the longest exposure time ratio under the condition that the image in the instrument panel area 71 is not overexposed is T71L, and the instrument panel area is The optimum exposure time ratio in 71 is T71O. The optimum exposure time ratio in the non-instrument panel region 72 is T72O. The control unit 16 can determine the condition of whether or not the image is underexposed and the condition of whether or not the image is overexposed. Therefore, the control unit 16 can calculate the shortest exposure time that does not cause blackout, the longest exposure time that does not cause overexposure, and the optimum exposure time according to the brightness of the image. Can be decided.

When the interior of the vehicle is dark and the instrument panel area 71 is brighter than the non-instrument panel area 72, the exposure time ratios T71S, T71L, T71O, and T72O are, for example, 10%, 50%, 30%, and 80%. When the maximum exposure time ratio T71L and the optimum exposure time ratio T72O satisfy T71L <T72O, the control unit 16 sets the imaging unit 12 so that the maximum value of the exposure time ratio in the instrument panel region 71 is the maximum exposure time ratio T71L. It is good to control.

When the interior of the vehicle is illuminated by sunlight and the non-instrument panel area 72 is brighter than the instrument panel area 71, the exposure time ratios T71S, T71L, T71O, and T72O are, for example, 60%, 80%, 70%, and 20%. become that way. When the shortest exposure time ratio T71S and the optimum exposure time ratio T72O satisfy T72O <T71S, the control unit 16 sets the imaging unit 12 so that the minimum value of the exposure time ratio in the instrument panel region 71 is the shortest exposure time ratio T71S. It is good to control.

If the control unit 16 controls the imaging unit 12 in this way, the instrument panel area 71 is not imaged in an overexposed state, and the non-instrument panel area 72 is imaged in a bright state. Therefore, according to the vehicle image pickup device 100 of the seventh embodiment, an appropriate vehicle interior image can be generated.

Of the configurations shown in FIGS. 2 and 21, at least the image processing unit 13, the image analysis unit 14, and the control unit 16 include a computer mounted on the vehicle imaging device 100 or a central processing unit (CPU) of the computer. It can be configured by a computer program (image processing program) to be executed by the CPU.

The first embodiment may be an image processing program that causes the CPU to execute the processes shown in FIGS. 8 and 9. The second embodiment may be an image processing program that causes the CPU to execute the process shown in FIG. The third embodiment may be an image processing program that causes the CPU to execute the process shown in FIG. The fourth embodiment may be an image processing program that causes the CPU to execute the processes shown in FIGS. 18A and 18B. The fifth embodiment may be an image processing program that causes the CPU to execute the processes shown in FIGS. 20A and 20B. The sixth embodiment may be an image processing program that causes the CPU to execute the processing shown in FIG. The seventh embodiment may be an image processing program that causes the CPU to execute the processing shown in FIG. 24.

The present invention is not limited to the first to seventh embodiments described above, and various modifications can be made without departing from the gist of the present invention. The first to seventh embodiments can be arbitrarily combined as long as they do not contradict each other. The proper use of hardware and software (computer program) is optional.

11 Fisheye lens 12 Imaging unit 13 Image processing unit 14 Image analysis unit (sunlight incident direction determination unit, tunnel travel determination unit)
15 Communication unit 16 Control unit (area setting unit)
17 Recording / playback unit 18 Monitor 19 Bus 20B Rear image 20F Front image 20L Left side image 20R Right side image 50 Tunnel 50in Entrance 50out Exit 60 Accelerometer (event detection sensor)
131 Image rotation unit 132 Image extraction unit 133 High dynamic range image composition unit 141 Motion vector detection unit 142 Front / rear / left / right judgment unit 161 Measurement unit 170 Memory card 171 Ring buffer 172 Event recording area 173 Normal recording area 200, 241-244 Vehicle 202 Steering Wheel 300 driver

Claims (5)

It is an image pickup device for vehicles that can be installed inside the vehicle.
A long-exposure image in which light is incident through a fisheye lens and a 360-degree subject is imaged in the first exposure time, and a short exposure image in which a 360-degree subject is imaged in a second exposure time shorter than the first exposure time. An imaging unit that generates a time-exposure image and
The front / rear and left / right sides of the vehicle are determined based on the long-exposure image and the short-exposure image, and the front image region, the rear image region, and the right side of the long-exposure image and the short-time exposure image are determined. The front / rear / left / right determination unit that determines the position of the image area and the left side image area in the circumferential direction,
From the long-exposure image and the short-exposure image, each region image of the front image region, the rear image region, the right side image region, and the left side image region is extracted, and the front image and the rear image are extracted. , An image extraction unit that generates images in each direction of the right side image and the left side image,
The front image, the rear image, the right side image, and the left side image generated by the image extraction unit based on the long exposure image, and the front image generated by the image extraction unit based on the short exposure image. , A high dynamic range image compositing unit that synthesizes the rear image, the right side image, and the left side image, respectively.
A sunlight incident direction determining unit that determines the incident direction of sunlight incident on the vehicle imaging device,
With
The high dynamic range image synthesizing unit is selected from the front image, the rear image, the right side image, and the left side image according to the incident direction of sunlight determined by the sunlight incident direction determining unit. A vehicle imaging device that increases the proportion of directional images based on the long-exposure image that has been created from the proportion of directional images based on the unselected long-exposure image. The sunlight incident direction determination unit divides the 360-degree direction imaged by the vehicle imaging device into a plurality of directions including at least the front direction, the rear direction, the right direction, and the left direction, and the sunlight incident direction is determined. Determine which direction you are in
The high dynamic range image compositing unit
When the incident direction of sunlight is the forward direction, the proportion of the front image based on the long-exposure image is increased.
When the incident direction of sunlight is the rear direction, the proportion of the rear image based on the long-exposure image is increased.
When the incident direction of sunlight is to the right, the proportion of the right-hand image based on the long-exposure image is increased.
The vehicle imaging device according to claim 1, which increases the proportion of the left side image based on the long exposure image when the incident direction of sunlight is the left direction. The sunlight incident direction determination unit adds the 360-degree direction imaged by the vehicle imaging device to the front direction, the rear direction, the right direction, and the left direction, as well as the right front direction, the right rear direction, the left rear direction, and the left front. Divide into multiple directions including the direction to determine which direction the sunlight is incident on,
The high dynamic range image compositing unit
When the incident direction of sunlight is the front right direction, the ratio of the front image and the right side image based on the long exposure image is increased.
When the incident direction of sunlight is the rear right direction, the ratio of the rear image and the right side image based on the long-exposure image is increased.
When the incident direction of sunlight is the rear left direction, the ratio of the rear image and the left image based on the long-exposure image is increased.
The vehicle imaging device according to claim 2, wherein when the incident direction of sunlight is the left front direction, the ratio of the front image and the left side image based on the long-exposure image is increased. This is an image processing method for an image pickup device for vehicles installed in a vehicle.
A long-exposure image in which the image pickup unit in which light is incident through a fisheye lens captures a 360-degree subject in the first exposure time and a 360-degree subject in the second exposure time shorter than the first exposure time. Generates a short-exposure image of the image
The front / rear and left / right sides of the vehicle are determined based on the long-exposure image and the short-time exposure image, and the front image region, the rear image region, and the right side of the long-exposure image and the short-time exposure image are determined. Determine the position of the image area and the left image area in the circumferential direction,
From the long-exposure image and the short-exposure image, each region image of the front image region, the rear image region, the right side image region, and the left side image region is extracted, and the front image and the rear image are extracted. , Right side image, and left side image in each direction,
Front image, rear image, right side image, and left side image generated based on the long exposure image, and front image, rear image, right side image, and left side image generated based on the short time exposure image. And each are combined to generate a high dynamic range composite image,
The direction of incidence of sunlight incident on the vehicle imaging device is determined, and the direction of incidence is determined.
When generating the high dynamic range composite image, the long exposure selected from the front image, the rear image, the right side image, and the left side image according to the incident direction of the determined sunlight. An image processing method that increases the proportion of directional images based on an image more than the proportion of directional images based on the unselected long-exposure image. For computers installed in vehicle imaging devices installed in vehicles,
A long-exposure image in which a 360-degree subject is imaged in the first exposure time by an imaging unit in which light is incident through a fisheye lens, and a 360-degree subject in a second exposure time shorter than the first exposure time. And the step of generating a short-exposure image of the image
The front / rear and left / right sides of the vehicle are determined based on the long-exposure image and the short-time exposure image, and the front image region, the rear image region, and the right side of the long-exposure image and the short-time exposure image are determined. Steps to determine the circumferential position of the image area and the left image area,
From the long-exposure image and the short-exposure image, each region image of the front image region, the rear image region, the right side image region, and the left side image region is extracted, and the front image and the rear image are extracted. , A step to generate a right-side image, and a step to generate a left-side image in each direction,
Front image, rear image, right side image, and left side image generated based on the long exposure image, and front image, rear image, right side image, and left side image generated based on the short time exposure image. And the steps to generate a high dynamic range composite image by synthesizing each
A step of determining the incident direction of sunlight incident on the vehicle imaging device, and
When generating the high dynamic range composite image, the long exposure selected from the front image, the rear image, the right side image, and the left side image according to the incident direction of the determined sunlight. A step of increasing the proportion of directional images based on the image from the proportion of directional images based on the unselected long-exposure image.
An image processing program that executes.

Images