JP2017169048A - Imaging apparatus, image processing apparatus, object detection system, object detection method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus, mounted on a vehicle, including photoelectric conversion elements arranged in a lattice shape, for imaging a direction parallel to a plane of vehicle traveling using an image sensor that performs processing of replacing light received simultaneously at photoelectric conversion elements aligned in a line into a gradation signal sequentially from the row of one end of the image sensor to the row of the other end, capable of imaging objects outside the vehicle without any loss even when vibration in a direction perpendicular to the traveling plane of the vehicle is applied to the imaging apparatus.SOLUTION: A CMOS camera 2 is mounted on a vehicle and includes photoelectric conversion elements aligned in a lattice shape, and images the direction parallel to the traveling plane of the vehicle by using a CMOS sensor that performs processing of replacing light received simultaneously at photoelectric conversion elements aligned in a line in the direction vertical to the traveling plane of the vehicle into a gradation signal sequentially from a row at one end to a row at the other end of the image sensor to output the gradation signal representing a photographed image as an imaging signal to an image processing apparatus 1.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging device, an image processing device, an object detection system, an object detection method, and a program that are mounted on a vehicle and photograph a direction parallel to a traveling surface of the vehicle.

  In recent years, in order to realize a vehicle driving assistance device or an automatic driving device, there is a method of monitoring and sensing the situation outside the vehicle with a camera sensor using a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera. It has been put into practical use. The driving assistance device described here is a device that performs control intervention of acceleration, braking, and steering as necessary when there is a risk that the vehicle will collide with an obstacle for a vehicle controlled by the driver, for example. That is. The driving assistance device includes an actuator that operates a prime mover, a braking device, and a steering device, an electronic control device that controls the actuator, and sensors that are input to the electronic control device, and is also referred to as a driving support system. In addition, as a simpler driving assistance device, if there is a risk of the vehicle colliding with an obstacle, a warning sound is emitted by a buzzer or a speaker, or a lamp is emitted or a warning is given to the display screen as necessary. There are devices that give warnings and instructions to the driver by means of visual stimulation such as display.

  An automatic driving device is a control device that automatically drives a vehicle that is not controlled by the driver by controlling acceleration, braking, and steering. Similar to the driving assistance device, the automatic driving device includes an actuator that operates a prime mover, a braking device, and a steering device, an electronic control device that controls the actuator, and sensors that are input to the electronic control device. Hereinafter, the driving assistance device and the automatic driving device are collectively referred to as a vehicle control device. Applications of using a camera sensor in a vehicle control device include detecting other vehicles, pedestrians, and obstacles such as falling objects in the vicinity of the vehicle or in a traveling road or side road. For example, when the camera sensor detects an obstacle, the vehicle control device controls acceleration, braking, and steering in order to avoid a collision.

  As an imaging device for a camera sensor, there are several advantages of using a CMOS camera compared to a CCD camera. Compared with a CCD camera, a CMOS camera has a lower power supply voltage for operation, so that it consumes less power and requires fewer peripheral circuitry, thereby reducing product costs. Further, since the operation speed of the CMOS camera is faster than that of the CCD camera, there is an advantage that the time taken to sense an obstacle can be shortened and the number of times of imaging per unit time can be increased. Moreover, the shorter the time for sensing an obstacle, the longer the braking and steering time for avoiding the obstacle, and the safer and more reliable avoidance action can be taken.

  On the other hand, when a rolling shutter type image sensor such as a CMOS image sensor included in a CMOS camera is used, a specific image distortion called “rolling shutter phenomenon”, “rolling shutter distortion”, or “focal plane distortion” is used. Happens. Imaging pixels of a general rolling shutter type image sensor are sequentially read from the upper column for each column in the horizontal direction, and the imaging timing is different for each column. When the image sensor is stationary with respect to the imaging target, there is no problem, but the positional relationship between the image sensor and the imaging target has changed due to vibration applied to the image sensor or the imaging target moving. The imaging timing is shifted in each column. As a result, a shift or difference occurs in the shooting position of each row, and the shot image is distorted. There is a method for correcting distortion of a captured image due to such a rolling shutter phenomenon.

  Patent Document 1 uses an image sensor that can switch the reading order of pixel values between a horizontal line direction and a vertical line direction, and detects and switches the vibration direction with an acceleration sensor or the like to switch the pixel values in each main scanning direction. An image sensor and a camera system that reduce line deviation and correct image distortion caused by focal plane shutter operation are disclosed.

JP 2006-54788 A

  In particular, in a situation where a rolling shutter type image sensor such as a CMOS image sensor is used to photograph a direction parallel to the traveling surface of the vehicle, the image sensor is connected to the image sensor because the vehicle passes through unevenness on the road surface. When vibration in a direction perpendicular to the traveling surface is applied, the photographing position is shifted in a direction perpendicular to the traveling surface of the vehicle for each row. Depending on the vibration speed, there may be a case where the shooting positions of adjacent rows are shifted by one row or more. Due to the shift of the shooting position in each row, pixels of an object outside the vehicle that should have been shot may be missing from the shot image.

  The technique of Patent Document 1 corrects image distortion, and has a detrimental effect that the image becomes narrower by the correction. In the narrowed part, a missing part of the photographing target occurs. In some cases, a part of an object (such as a road or an obstacle) outside the vehicle cannot be detected due to a pixel loss caused by the rolling shutter phenomenon. In particular, there may be a case where an obstacle on a road far away in the traveling direction, which appears only on one image to several pixels on the image sensor, cannot be detected.

  The present invention has been made in view of the circumstances as described above, and is provided with a photoelectric conversion element mounted in a vehicle and arranged in a vertical and horizontal lattice pattern, and a light signal simultaneously received by the photoelectric conversion elements arranged in a row is a gradation signal. In an imaging device that captures a direction parallel to the traveling surface of the vehicle using an image sensor that sequentially performs processing to replace the image sensor from one end row to the other end row, the imaging device is perpendicular to the traveling surface of the vehicle. It is an object to make it possible to photograph an object outside the vehicle without any loss even when vibrations in various directions are applied.

  In order to achieve the above object, an imaging apparatus according to the present invention includes photoelectric conversion elements mounted in a vehicle and arranged in a vertical and horizontal grid pattern, and replaces light received simultaneously by the photoelectric conversion elements arranged in a row with a gradation signal. Processing is performed sequentially from one end row of the image sensor to the other end row, and the image simultaneously formed by the photoelectric conversion elements arranged in a row is an image in a direction perpendicular to the traveling surface of the vehicle in the object field. A certain image sensor is used to photograph a direction parallel to the traveling surface of the vehicle.

  According to the present invention, the process of replacing the light received at the same time by the photoelectric conversion elements mounted on the vehicle and arranged in a vertical and horizontal grid pattern with the grayscale signals at the one end of the image sensor is performed. In an imaging device that captures a direction parallel to a traveling surface of a vehicle using an image sensor that sequentially performs from the row to the other end row, an image formed simultaneously by the photoelectric conversion elements arranged in a row is a vehicle in the field of view. By using an image sensor that is an image in a direction perpendicular to the traveling surface of the vehicle, it is possible to photograph an object outside the vehicle without loss even when the imaging device is subjected to vibration in a direction perpendicular to the traveling surface of the vehicle. .

It is a figure which shows the structural example of the object detection system which concerns on embodiment of this invention. It is a figure which shows the example of the picked-up image in the state where the vibration is not added to the CMOS camera. It is a figure which shows the example of the picked-up image of the state where the vibration of the direction perpendicular | vertical to the vehicle running surface is added to the CMOS camera in which a line scan direction corresponds to the direction parallel to the vehicle running surface. It is a figure which shows the example of the picked-up image of the state where the vibration of the direction perpendicular | vertical to the vehicle running surface is added to the CMOS camera in which a line scan direction corresponds to the direction perpendicular | vertical to the vehicle running surface. It is a figure which shows the example of a picked-up image when there exists an obstruction on a distant road in the state where the vibration is not added to a CMOS camera. An example of a captured image in the case where there is an obstacle on a distant road in a state where vibration in a direction perpendicular to the vehicle traveling surface is applied to the CMOS camera whose line scan direction coincides with a direction parallel to the vehicle traveling surface is shown. FIG. An example of a captured image in the case where there is an obstacle on a distant road in a state where vibration in a direction perpendicular to the vehicle traveling surface is applied to the CMOS camera in which the line scan direction coincides with the direction perpendicular to the vehicle traveling surface is shown. FIG. It is a flowchart which shows an example of operation | movement of the object detection system which concerns on embodiment. It is a block diagram which shows an example of the hardware constitutions of the image processing apparatus which concerns on embodiment of this invention.

  EMBODIMENT OF THE INVENTION Below, the form for implementing this invention is demonstrated in detail with reference to drawings. In addition, the same code | symbol is attached | subjected to the part which is the same or it corresponds in a figure. In this embodiment, an example in which the imaging device is a CMOS camera will be described.

  FIG. 1 is a diagram illustrating a configuration example of an object detection system according to an embodiment of the present invention. The object detection system 100 includes a CMOS camera 2, a vibration detection sensor 3, an image processing device 1, and an object detection device 4. The CMOS camera 2, the vibration detection sensor 3, the image processing device 1, and the object detection device 4 are mounted on a vehicle. The CMOS camera 2 is a rolling shutter type electronically controlled camera module, and photographs a direction parallel to the traveling surface of the vehicle. The CMOS camera 2 includes an optical lens and a CMOS image sensor, and replaces light emitted or reflected by an object outside the vehicle with an electrical signal. The CMOS image sensor included in the CMOS camera 2 includes photoelectric conversion elements arranged in a vertical and horizontal grid pattern, and each photoelectric conversion element corresponds to a pixel of a captured image. The intensity of light received by the photoelectric conversion element is converted into a gradation signal which is an electrical signal indicating the gradation of the pixel. The gradation of the pixel is an amount representing the brightness (luminance) of the pixel. For example, if the gradation signal is an 8-bit digital signal, 256 gradations can be expressed, and the darkest luminance is 0, The brightest luminance is 255.

  The CMOS image sensor of the CMOS camera 2 performs a process of replacing light simultaneously received by the photoelectric conversion elements arranged in a line with a gradation signal (hereinafter referred to as gradation signal conversion process) from one column of the CMOS image sensor to the other. Repeat until the end of the row. Hereinafter, the column direction of photoelectric conversion elements that simultaneously receive light is referred to as a line scan direction, and a direction perpendicular to the line scan direction on the surface of the CMOS image sensor is referred to as a frame scan direction. The number of pixels in the line scan direction × the number of pixels in the frame scan direction is the number of pixels (or resolution) of the CMOS image sensor, and a captured image is expressed by the gradation of these pixels. There are various standards for the gradation signal, but any digital signal indicating the gradation of each pixel may be used. For example, digital signals transmitted by a total of 12 lines including a clock line, 8 data lines indicating gradation, 1 line of effective signal line, 1 frame of effective signal line, and 1 frame of start signal line are used.

  The CMOS camera 2 transmits a gradation signal representing a captured image to the image processing apparatus 1 as an imaging signal. The CMOS camera 2 is installed in such a posture that the line scan direction is a direction perpendicular to the traveling surface of the vehicle. The CMOS image sensor of the CMOS camera 2 replaces the light simultaneously received by the photoelectric conversion elements arranged in a line in a direction perpendicular to the traveling surface of the vehicle with a gradation signal. That is, the image simultaneously formed by the photoelectric conversion elements arranged in a line in the line scan direction of the CMOS image sensor is an image in a direction perpendicular to the traveling surface of the vehicle in the object scene. The effect obtained when the line scan direction of the CMOS image sensor of the CMOS camera 2 coincides with the direction perpendicular to the traveling surface of the vehicle will be described with reference to FIGS. 2 to 7, for the sake of convenience, it is assumed that an object outside the vehicle is not moving.

  FIG. 2 is a diagram illustrating an example of a captured image in a state where no vibration is applied to the CMOS camera. In the photographed image of FIG. 2, the road on which the vehicle is traveling, the tree on the shoulder, and clouds are reflected in the sky. Even when the line scan direction of the CMOS image sensor of the CMOS camera 2 coincides with the direction parallel to the traveling surface of the vehicle, even when the line scan direction coincides with the direction perpendicular to the traveling surface of the vehicle, When the CMOS camera 2 is not vibrated, the captured image is not distorted.

  FIG. 3 is a diagram illustrating an example of a captured image in a state where vibration in a direction perpendicular to the vehicle traveling surface is applied to the CMOS camera in which the line scan direction coincides with a direction parallel to the vehicle traveling surface. FIG. 3A shows an example of a captured image when the CMOS camera 2 is moving in a direction away from the ground. Compared with a captured image in the case where no vibration is applied to the CMOS camera 2 in FIG. 2, in FIG. 3A, a road and a shoulder tree are photographed as if they are stretched in the vertical direction. This occurs because the shooting position of the next row changes during the time required for the one row of gradation signal conversion processing. Due to the shift of the shooting position in each row, sky clouds are missing from the shot image. FIG. 3B is an example of a captured image when the CMOS camera 2 is moving in a direction approaching the ground. In FIG. 3 (b), the image is taken as if the road, roadside trees, and clouds in the sky are crushed, as compared to the image taken when the CMOS camera in FIG. 2 is not vibrated. This also occurs because the shooting position of the next row changes during the time required for the one row of gradation signal conversion processing.

  FIG. 4 is a diagram illustrating an example of a captured image in a state where vibration in a direction perpendicular to the vehicle traveling surface is applied to the CMOS camera in which the line scan direction coincides with a direction perpendicular to the vehicle traveling surface. Compared to the photographed image in the case where no vibration is applied to the CMOS camera 2 in FIG. 2, it can be seen in FIG. 4 that the horizon, roadside trees, and clouds are distorted upward. Since the CMOS image sensor of the CMOS camera 2 according to the present embodiment is installed in a posture in which the line scan direction is a direction perpendicular to the traveling surface of the vehicle, the following is brought about by the time required for one row of gradation signal conversion processing. The change in the imaging position of the column is parallel to the line scan direction. For this reason, when the CMOS camera 2 is subjected to vibration in a direction perpendicular to the traveling surface of the vehicle, the captured image rises to the right when the CMOS camera 2 moves in a direction approaching the ground, and the captured image is When the CMOS camera 2 is moving away from the ground, it falls downward. Next, the case where an obstacle exists in the far direction of the traveling direction of the road on which the vehicle on which the CMOS camera 2 is mounted travels will be described. When the obstacle is far from the vehicle by a certain distance or more, an image is formed on the CMOS image sensor with a size of about one pixel to several pixels.

  FIG. 5 is a diagram showing an example of a photographed image when there is an obstacle on a distant road with no vibration applied to the CMOS camera. FIG. 5 is an enlarged view of the periphery of the road where there is an obstacle near the horizon of the photographed image in a state where vibration is not applied to the CMOS camera 2 as shown in FIG. 2, and a square surrounded by a grid line is shown. One pixel. The color intensity in the square represents the gradation of each pixel. The darkest area in the figure is the road. There is a light-colored portion of one pixel in the road area, which is an obstacle pixel P representing a distant obstacle. Here, the obstacle pixel P has a size of one pixel because the obstacle is far away and an image is formed with a size of one pixel or less on the CMOS image sensor. In this example, since the obstacle is brighter than the road, the obstacle pixel P has a higher gradation than the pixel capturing the road. When the obstacle is darker than the road, the obstacle pixel P has a lower gradation than the pixel capturing the road.

  FIG. 6 is a photographed image in the case where there is an obstacle on a distant road in a state where vibration in a direction perpendicular to the vehicle running surface is applied to the CMOS camera whose line scan direction coincides with a direction parallel to the vehicle running surface. It is a figure which shows the example of. FIG. 6A is an enlarged view of the periphery of a road with an obstacle near the horizon of a captured image when the CMOS camera 2 as shown in FIG. 3A is moving away from the ground. The obstacle pixel P in FIG. 6A has a size of two pixels in the vertical direction. This is the same reason that the entire captured image is stretched in the vertical direction as shown in FIG. FIG. 6B is an enlarged view of the vicinity of the road where there is an obstacle near the horizon of the captured image when the CMOS camera 2 as shown in FIG. 3B is moving in the direction approaching the ground. In FIG. 6B, there is no pixel capturing an obstacle. This is because the vibration in the direction perpendicular to the traveling surface of the vehicle is applied to the CMOS camera 2, so that the next column is the obstacle in the time required for one row of gradation signal conversion processing capturing the portion above the obstacle. This is because the photographing apparatus moves to a position where the lower part is captured.

  When the line scan direction of the CMOS image sensor of the CMOS camera 2 is parallel to the traveling surface of the vehicle, a captured image that does not capture an obstacle may be generated as shown in FIG. If there is no pixel that captures the obstacle, the obstacle cannot be detected therefrom. Even when a method for detecting an obstacle from a plurality of captured images is used, if an image without an obstacle is included, the detection accuracy decreases.

  FIG. 7 is a photographed image in the case where there is an obstacle on a distant road in a state where vibration in the direction perpendicular to the vehicle traveling surface is applied to the CMOS camera in which the line scan direction coincides with the direction perpendicular to the vehicle traveling surface. It is a figure which shows the example of. FIG. 7 is an enlarged view of the periphery of the road where there is an obstacle near the horizon of the photographed image in a state where the vibration in the direction perpendicular to the traveling surface of the vehicle is applied to the CMOS camera 2 as shown in FIG. The obstacle pixel P captures an obstacle with a size of one pixel. Since the CMOS image sensor of the CMOS camera 2 according to the present embodiment is installed in a posture in which the line scan direction is perpendicular to the traveling surface of the vehicle, the CMOS camera 2 moves in the direction perpendicular to the traveling surface of the vehicle. However, since the pixels in the line scan direction (direction perpendicular to the traveling surface of the vehicle) are simultaneously converted into gradation signals, the obstacle is sufficiently far away, and the size is one pixel or less on the CMOS image sensor. Obstacles can be captured even when the image is formed with the.

  Thus, in the object detection system 100, the CMOS image sensor of the CMOS camera 2 replaces the light simultaneously received by the photoelectric conversion elements arranged in a line in a direction perpendicular to the traveling surface of the vehicle with a gradation signal, so that the vehicle Even when there is an obstacle far away in the traveling direction of the traveling road, and the CMOS camera 2 is subjected to vibration in a direction perpendicular to the traveling surface of the vehicle, the obstacle can be captured.

  The vibration detection sensor 3 in FIG. 1 is an acceleration sensor, and converts the detected acceleration into an electrical signal and transmits it to the image processing apparatus 1 as a vibration signal. The vibration signal may be one in which acceleration is expressed by voltage with a single wire, or may be a digital signal such as serial communication. The vibration detection sensor 3 is installed so as to be able to detect vibration in a direction perpendicular to the traveling surface of the vehicle applied to the CMOS camera 2 mounted on the vehicle. That is, at least one axis in the acceleration detection direction of the vibration detection sensor 3 is preferably installed in a direction perpendicular to the vehicle running surface so as not to be perpendicular to the direction perpendicular to the vehicle running surface. The vibration detection sensor 3 is installed at a position where the same acceleration as that of the CMOS camera 2 is applied. With the configuration as described above, the vibration detection sensor 3 can detect vibration in a direction perpendicular to the traveling surface of the vehicle applied to the CMOS camera 2.

  The image processing apparatus 1 includes a digital signal processing circuit such as an LSI (Large-Scale Integration) or an FPGA (field-programmable gate array). The functional configuration includes an imaging signal acquisition unit 11, a vibration signal acquisition unit 12, and a deviation amount. A calculation unit 13, an image correction unit 14, and a corrected image signal output unit 15 are provided. The imaging signal acquisition unit 11 receives the imaging signal from the CMOS camera 2 and sends it to the image correction unit 14. The vibration signal acquisition unit 12 receives the vibration signal from the vibration detection sensor 3 and sends it to the deviation amount calculation unit 13. When the vibration signal is an analog signal, the vibration signal acquisition unit 12 includes an appropriate A / D conversion (analog / digital conversion) circuit, converts the vibration signal into a digital signal, and sends the digital signal to the deviation amount calculation unit 13. Alternatively, the vibration signal may be converted into a digital signal by using an A / D conversion circuit mounted as a function of the LSI or FPGA.

  The shift amount calculation unit 13 calculates the shift amount of the shooting position of each column of the CMOS image sensor of the CMOS camera 2 based on the vibration signal received from the vibration signal acquisition unit 12. The shift amount calculation unit 13 generates a correction amount signal indicating a correction amount for correcting the shift of each column of the captured image from the calculated shift amount of the shooting position of each column, and sends the correction amount signal to the image correction unit 14. The correction amount signal is used to correct a photographed image having a distortion of rising right shoulder or descending shoulder as shown in FIGS. 4 and 7 generated by vibration in a direction perpendicular to the traveling surface of the vehicle applied to the CMOS camera 2. The amount by which each column is returned in the reverse direction by the amount shifted in the line scan direction is shown. Since the distortion of the right shoulder rises or lowers in the photographed image is caused by the shift of the shooting position in the line scan direction of each column, if the column in which the shift has occurred is returned in the reverse direction by the amount shifted in the line scan direction. It is possible to correct the distortion of the right shoulder rise or the right shoulder fall.

  As mentioned above, the vibration signal indicates the acceleration in the direction perpendicular to the running surface of the vehicle. Therefore, if the gravitational acceleration in the direction perpendicular to the running surface of the vehicle is subtracted, the vibration signal is generated when the vehicle gets over a road step. Due to the vibration, acceleration in a direction perpendicular to the traveling surface of the vehicle applied to the CMOS camera 2 can be detected. Since the speed in the direction perpendicular to the traveling surface of the vehicle at any moment can be calculated, the amount of shift in the direction perpendicular to the traveling surface of the vehicle in each row at the timing of the gradation signal conversion process can be calculated. If the amount of deviation in the direction perpendicular to the running surface of the vehicle is calculated, the amount of deviation in the line scan direction of each row with respect to the first row can be found, so only the same amount as the amount of deviation in that line scan direction By shifting in the reverse direction, it is possible to correct an image that has been distorted by the rolling shutter phenomenon. Therefore, the shift amount calculation unit 13 outputs the shift amount in the line scan direction of each column with respect to the first column as a correction amount signal.

  The image correction unit 14 corrects the captured image of the CMOS camera 2 based on the imaging signal received from the imaging signal acquisition unit 11 and the correction amount signal received from the shift amount calculation unit 13. The image correction unit 14 restores the shift in the line scan direction with respect to the column in which the shift of the image represented by the imaging signal is generated according to the correction amount indicated by the correction amount signal, so that the image that has been distorted by the rolling shutter phenomenon is displayed. to correct. At this time, although there are portions where pixels are not present above and below the image, the image processing apparatus 1 according to the present invention is to generate an image for detecting an object outside the vehicle, so that a corrected captured image is captured. It only needs to include the target object. For example, a gradation value 0 may be inserted into a portion where no image exists, or a value (for example, −1) indicating that no pixel exists may be set and inserted. The image correction unit 14 generates a corrected image signal representing the corrected captured image and sends it to the corrected image signal output unit 15. The corrected image signal output unit 15 transmits the corrected image signal received from the image correction unit 14 to the object detection device 4.

  The object detection device 4 detects the obstacle pixel P from the corrected image signal received from the image processing device 1, and outputs the detection result as an object detection signal. The output method of the object detection signal may be screen display, audio output, or output to the vehicle control device. The object detection device 4 extracts, for example, a road area from a corrected captured image indicated by the corrected image signal, and a fault whose gradation is higher or lower than a threshold value than a pixel presenting the road in the road area. The object pixel P is detected. The method for detecting an obstacle from a captured image is not limited to this, and an existing technique may be used. When the CMOS camera 2 is installed in a posture in which the line scan direction is parallel to the traveling surface of the vehicle as in the past, the traveling direction of the road on which the vehicle travels as described with reference to FIG. In some cases, the obstacle pixel P does not exist in the photographed image even though the obstacle exists far away. If there is no obstacle pixel P, it is impossible to detect the obstacle by any method for detecting the obstacle from the captured image. Here, processing performed by each device of the object detection system 100 will be described with reference to FIG.

  FIG. 8 is a flowchart illustrating an example of the operation of the object detection system according to the embodiment. The following processing is repeatedly executed while each device of the object detection system 100 is activated. The CMOS camera 2 captures a direction parallel to the traveling surface of the vehicle (step S11). The CMOS camera 2 transmits a gradation signal representing a captured image in which the line scan direction and the direction perpendicular to the traveling surface of the vehicle coincide with each other as an imaging signal to the image processing apparatus 1 (step S12), and ends the process. A photographed image in which the line scan direction indicated by the photographing signal transmitted from the CMOS camera 2 coincides with the direction perpendicular to the vehicle running surface is a direction perpendicular to the vehicle running surface of the CMOS camera 2 as shown in FIG. As shown in FIG. 7, the obstacle pixel P is caused by the vibration in the direction perpendicular to the traveling surface of the vehicle applied to the CMOS camera 2. It will not disappear. The vibration detection sensor 3 detects acceleration in a direction perpendicular to the traveling surface of the vehicle (step S21). The electrical signal obtained by converting the acceleration in the direction perpendicular to the traveling surface of the vehicle is transmitted as a vibration signal to the image processing apparatus 1 (step S22), and the process is terminated.

  The imaging signal acquisition unit 11 of the image processing apparatus 1 receives the imaging signal from the CMOS camera 2 (step S31) and sends it to the image correction unit 14. The vibration signal acquisition unit 12 receives the vibration signal from the vibration detection sensor 3 (step S <b> 32) and sends it to the deviation amount calculation unit 13. Based on the vibration signal received from the vibration signal acquisition unit 12, the shift amount calculation unit 13 calculates the shift amount of the shooting position of each column of the CMOS image sensor of the CMOS camera 2 (step S33). The shift amount calculation unit 13 generates a correction amount signal indicating a correction amount for correcting the shift of each column of the captured image from the calculated shift amount of the shooting position of each column, and sends the correction amount signal to the image correction unit 14. The image correction unit 14 corrects the captured image of the CMOS camera 2 based on the imaging signal received from the imaging signal acquisition unit 11 and the correction amount signal received from the shift amount calculation unit 13 (step S34). The image correction unit 14 is a column in which a shift of the image represented by the imaging signal is generated in accordance with the correction amount signal with respect to a captured image that is rising or falling right as illustrated in FIGS. 4 and 7. By correcting the shift in the line scan direction, an image distorted by the rolling shutter phenomenon is corrected.

  The image correction unit 14 generates a corrected image signal representing the corrected captured image and sends it to the corrected image signal output unit 15. The corrected image signal output unit 15 transmits the corrected image signal received from the image correction unit 14 to the object detection device 4 (step S35), and ends the process. When the object detection device 4 receives the corrected image signal from the image processing device 1 (step S41), the object detection device 4 detects the obstacle pixel P from the corrected captured image indicated by the correction image signal (step S42). The object detection device 4 outputs the detection result as an object detection signal (step S43), and ends the process.

  As described above, according to the CMOS camera 2 that is mounted on the vehicle of the present embodiment and photographs the direction parallel to the traveling surface of the vehicle using the rolling shutter type CMOS image sensor, the photoelectric conversion elements arranged in a line When a CMOS image sensor is used in which the image formed at the same time is an image in a direction perpendicular to the traveling surface of the vehicle in the object field, vibration in a direction perpendicular to the traveling surface of the vehicle is applied to the CMOS camera 2 Even so, it is possible to photograph an object outside the vehicle without any loss. Further, according to the object detection system 100, the corrected captured image represented by the corrected image signal input from the image processing apparatus 1 to the object detection apparatus 4 is a vibration in a direction perpendicular to the traveling surface of the vehicle applied to the CMOS camera 2. As a result, the obstacle pixel P does not disappear and there is no shift in the column direction of the captured image, so that the object detection device 4 can easily detect the object. Note that when an object outside the vehicle moves in a direction parallel to the traveling surface of the vehicle, pixel loss may occur due to the rolling shutter phenomenon. However, in the case of a moving object, the image appears only in one pixel to several pixels. There is no need to detect at a distant point.

  In the above embodiment, the image processing device 1 receives the imaging signal transmitted by the CMOS camera 2 and corrects the captured image and then transmits it to the object detection device 4. However, the CMOS camera 2 detects the imaging signal as an object. The object detection device 4 may transmit to the device 4, detect the obstacle pixel P from the imaging signal received from the CMOS camera 2, and output the detection result as an object detection signal.

  In the above embodiment, it is assumed that the photographing direction of the CMOS camera 2 and the optical axis coincide with each other, but the present invention is not limited to this. If the image simultaneously formed by the photoelectric conversion elements arranged in a row in the CMOS image sensor is an image in a direction perpendicular to the traveling surface of the vehicle in the object field, the optical path may be changed by a mirror or the like. . When the optical path is changed, the line scan direction of the CMOS image sensor may not match the direction perpendicular to the traveling surface of the vehicle.

  FIG. 9 is a block diagram showing an example of a hardware configuration of the image processing apparatus according to the embodiment of the present invention. As shown in FIG. 9, the image processing apparatus 1 includes a control unit 31, a main storage unit 32, an external storage unit 33, an operation unit 34, a display unit 35, and an input / output unit 36. The main storage unit 32, the external storage unit 33, the operation unit 34, the display unit 35, and the input / output unit 36 are all connected to the control unit 31 via the internal bus 30.

  The control unit 31 includes a CPU (Central Processing Unit) and the like, and executes each process of the deviation amount calculation unit 13 and the image correction unit 14 of the image processing device 1 according to a control program 39 stored in the external storage unit 33. .

  The main storage unit 32 is composed of a RAM (Random-Access Memory) or the like, loads a control program 39 stored in the external storage unit 33, and is used as a work area of the control unit 31.

  The external storage unit 33 includes a nonvolatile memory such as a flash memory, a hard disk, a DVD-RAM, and a DVD-RW, and stores in advance a program for causing the control unit 31 to perform processing of the image processing apparatus 1. In accordance with an instruction from the control unit 31, the data stored by this program is supplied to the control unit 31, and the data supplied from the control unit 31 is stored.

  The operation unit 34 includes a pointing device such as a keyboard and a mouse, and an interface device that connects the keyboard and the pointing device to the internal bus 30. When the user inputs information to the image processing apparatus 1, the input information is supplied to the control unit 31 via the operation unit 34.

  The display unit 35 is composed of a CRT or LCD. When the user inputs information to the image processing apparatus 1, an operation screen is displayed.

  The input / output unit 36 includes a serial interface or a parallel interface. The input / output unit 36 is connected to the CMOS camera 2, the vibration detection sensor 3, and the object detection device 4. The input / output unit 36 functions as the imaging signal acquisition unit 11, the vibration signal acquisition unit 12, and the corrected image signal output unit 15.

  The processing of the imaging signal acquisition unit 11, the vibration signal acquisition unit 12, the deviation amount calculation unit 13, the image correction unit 14, and the corrected image signal output unit 15 of the image processing apparatus 1 illustrated in FIG. The main storage unit 32, the external storage unit 33, the operation unit 34, the display unit 35, the input / output unit 36, and the like are used as resources for processing.

  In addition, the hardware configuration and the flowchart described above are merely examples, and can be arbitrarily changed and modified.

  The central part that performs processing of the image processing apparatus 1 including the control unit 31, the main storage unit 32, the external storage unit 33, the operation unit 34, the display unit 35, the input / output unit 36, the internal bus 30, and the like is dedicated. It can be realized using a normal computer system regardless of the system. For example, a computer program for executing the above operation is stored and distributed on a computer-readable recording medium (flexible disk, CD-ROM, DVD-ROM, etc.), and the computer program is installed in the computer. Thus, the image processing apparatus 1 that executes the above-described processing may be configured. Further, the computer program may be stored in a storage device included in a server device on a communication network such as the Internet, and the image processing apparatus 1 may be configured by being downloaded by a normal computer system.

  When the functions of the image processing apparatus 1 are realized by sharing an OS (operating system) and an application program, or by cooperation between the OS and the application program, only the application program part is stored in a recording medium or a storage device. May be.

  It is also possible to superimpose a computer program on a carrier wave and provide it via a communication network. For example, a computer program may be posted on a bulletin board (BBS, Bulletin Board System) on a communication network, and the computer program may be provided via the network. The computer program may be activated and executed in the same manner as other application programs under the control of the OS, so that the processing of the image processing apparatus 1 can be executed.

  DESCRIPTION OF SYMBOLS 1 Image processing apparatus, 2 CMOS camera, 3 Vibration detection sensor, 4 Object detection apparatus, 11 Imaging signal acquisition part, 12 Vibration signal acquisition part, 13 Deviation amount calculation part, 14 Image correction part, 15 Correction image signal output part, 30 Internal bus, 31 control unit, 32 main storage unit, 33 external storage unit, 34 operation unit, 35 display unit, 36 input / output unit, 39 control program, 100 object detection system, P obstacle pixel.

Claims (5)

  A process for replacing photoelectric conversion elements mounted on a vehicle and arranged in a vertical and horizontal grid pattern, and simultaneously receiving light received by the photoelectric conversion elements arranged in a row with a gradation signal from one end row of the image sensor to the other end The image sensor is sequentially formed by the photoelectric conversion elements arranged in the row, and the image sensor is an image in a direction perpendicular to the traveling surface of the vehicle in the object field, and An imaging device that captures a direction parallel to a running surface. A process for replacing photoelectric conversion elements mounted on a vehicle and arranged in a vertical and horizontal grid pattern, and simultaneously receiving light received by the photoelectric conversion elements arranged in a row with a gradation signal from one end row of the image sensor to the other end The image sensor is sequentially formed by the photoelectric conversion elements arranged in the row, and the image sensor is an image in a direction perpendicular to the traveling surface of the vehicle in the object field, and An imaging device that captures a direction parallel to the traveling surface and outputs the gradation signal representing the captured image as an imaging signal, and an acceleration that is mounted on the vehicle and is perpendicular to the traveling surface of the vehicle, An image processing apparatus connected to a vibration detection sensor that outputs an electric signal obtained by converting an acceleration in a direction perpendicular to the traveling surface of the vehicle as a vibration signal,
An imaging signal acquisition unit that acquires the imaging signal from the imaging device;
A vibration signal acquisition unit for acquiring the vibration signal from the vibration detection sensor;
Based on the vibration signal acquired by the vibration signal acquisition unit, the shift amount of the shooting position of each column of the image sensor is calculated, and from the calculated shift amount of the shooting position of each column, each column of the captured image is calculated. A deviation amount calculation unit for generating a correction amount signal indicating a correction amount for correcting the deviation;
Based on the imaging signal acquired by the imaging signal acquisition unit and the correction amount signal generated by the deviation amount calculation unit, the shift of each column of the captured image is corrected, and the correction representing the corrected captured image An image correction unit for generating an image signal;
A corrected image signal output unit that outputs the corrected image signal generated by the image correction unit;
An image processing apparatus comprising: A process for replacing photoelectric conversion elements mounted on a vehicle and arranged in a vertical and horizontal grid pattern, and simultaneously receiving light received by the photoelectric conversion elements arranged in a row with a gradation signal from one end row of the image sensor to the other end The image sensor is sequentially formed by the photoelectric conversion elements arranged in the row, and the image sensor is an image in a direction perpendicular to the traveling surface of the vehicle in the object field, and An imaging device that captures a direction parallel to the traveling surface and outputs the gradation signal representing the captured image as an imaging signal;
A vibration detection sensor for detecting an acceleration in a direction perpendicular to the traveling surface of the vehicle and outputting an electric signal obtained by converting the acceleration in a direction perpendicular to the traveling surface of the vehicle as a vibration signal;
An imaging signal acquisition unit for acquiring the imaging signal from the imaging device;
A vibration signal acquisition unit for acquiring the vibration signal from the vibration detection sensor;
Based on the vibration signal acquired by the vibration signal acquisition unit, the shift amount of the shooting position of each column of the image sensor is calculated, and from the calculated shift amount of the shooting position of each column, each column of the captured image is calculated. A deviation amount calculation unit for generating a correction amount signal indicating a correction amount for correcting the deviation;
Based on the imaging signal acquired by the imaging signal acquisition unit and the correction amount signal generated by the deviation amount calculation unit, the shift of each column of the captured image is corrected, and the correction representing the corrected captured image An image correction unit for generating an image signal, and
A corrected image signal output unit that outputs the corrected image signal generated by the image correction unit;
An image processing apparatus comprising:
An object detection device for detecting an object existing outside the vehicle based on the corrected image signal output by the image processing device;
An object detection system comprising: A process for replacing photoelectric conversion elements mounted on a vehicle and arranged in a vertical and horizontal grid pattern, and simultaneously receiving light received by the photoelectric conversion elements arranged in a row with a gradation signal from one end row of the image sensor to the other end An image pickup device including the image sensor is executed in which the images simultaneously formed by the photoelectric conversion elements arranged in a row are images in the direction perpendicular to the traveling surface of the vehicle in the object scene. ,
Photographing a direction parallel to the traveling surface of the vehicle using the image sensor;
Outputting the gradation signal representing a captured image as an imaging signal;
The vibration detection sensor mounted on the vehicle executes,
Detecting acceleration in a direction perpendicular to the traveling surface of the vehicle;
Outputting an electrical signal obtained by converting acceleration in a direction perpendicular to the traveling surface of the vehicle as a vibration signal;
Executed by the image processing device,
Based on the vibration signal output from the vibration detection sensor, a shift amount of the shooting position of each column of the image sensor is calculated, and a shift of each column of the captured image is calculated from the calculated shift amount of the shooting position of each column. A deviation amount calculating step for generating a correction amount signal indicating a correction amount for correcting
Based on the imaging signal output from the imaging device and the correction amount signal generated in the shift amount calculation step, the corrected image signal representing the corrected captured image is corrected by correcting the shift of each column of the captured image. An image correction step for generating
A corrected image signal output step for outputting the corrected image signal generated in the image correction step;
Executed by the object detection device,
Detecting an object existing outside the vehicle based on the corrected image signal output by the image processing device;
An object detection method comprising: A process for replacing photoelectric conversion elements mounted on a vehicle and arranged in a vertical and horizontal grid pattern, and simultaneously receiving light received by the photoelectric conversion elements arranged in a row with a gradation signal from one end row of the image sensor to the other end The image sensor is sequentially formed by the photoelectric conversion elements arranged in the row, and the image sensor is an image in a direction perpendicular to the traveling surface of the vehicle in the object field, and An imaging device that captures a direction parallel to the traveling surface and outputs the gradation signal representing the captured image as an imaging signal, and an acceleration that is mounted on the vehicle and is perpendicular to the traveling surface of the vehicle, A computer connected to a vibration detection sensor that outputs an electrical signal obtained by converting an acceleration in a direction perpendicular to the traveling surface of the vehicle as a vibration signal;
Based on the vibration signal output from the vibration detection sensor, a shift amount of the shooting position of each column of the image sensor is calculated, and a shift of each column of the captured image is calculated from the calculated shift amount of the shooting position of each column. A deviation amount calculation unit that generates a correction amount signal indicating a correction amount for correcting the image, and the captured image output based on the imaging signal output from the imaging device and the correction amount signal generated by the deviation amount calculation unit. An image correction unit that corrects a shift of each column and generates a corrected image signal representing the corrected captured image;
Program to function as.

Images