JPWO2020170723A1 - Imaging device - Google Patents

Abstract

The present invention reduces the effects of focal plane distortion without causing missing frames or pixels. FIG. 2A shows an image in which focal plane distortion does not exist, and FIG. 2B shows an image in which focal plane distortion exists. FIG. 2C shows an image in which a blurring process is performed on an image in which focal plane distortion is present. When the three images in FIG. 2C are continuously displayed, the influence of the focal plane distortion felt by the user as the whole image is reduced.

Description

The present invention relates to an image pickup device using an image pickup device that operates in a rolling shutter system.

In a solid-state image sensor, particularly a CMOS image sensor, pixels are arranged two-dimensionally, and the operation at the time of image pickup is often performed by a rolling shutter method in which exposure is sequentially performed for each horizontal line. Can be cheaper. In this case, for example, the exposure timing of the pixels of the upper line and the exposure timing of the pixels of the lower line are different from each other. In general, when an image is taken of a subject with a small movement, such an deviation in exposure timing does not become a big problem.

On the other hand, if the subject moves at high speed, or if the size or direction of the angle of view fluctuates during imaging, or if there is such a deviation in the exposure timing, the image in which distortion occurs with respect to the actual image may appear. It is output from the CMOS image sensor. Such image distortion is known as focal plane distortion.

As a method of reducing the influence of such focal plane distortion, for example, it is known to replace an image of a frame in which focal plane distortion occurs with an image created based on an image of a frame immediately before and immediately after the image. In this case, for example, in the simplest case, the image of the frame in which the focal plane distortion occurs can be replaced with the image of the frames before and after the frame in which the focal plane distortion does not occur.

Further, in the technique described in Patent Document 1, in order to perform a higher correction than this, in a frame in which focal plane distortion occurs, a reference time point in this frame is set, and the frame images immediately before and after are analyzed. Then, the positional relationship between the pixel at the reference time point and the pixel in the frame image immediately before and immediately after corresponding to this pixel is calculated. As a result, the image of the frame in which the focal plane distortion occurs is corrected by linear interpolation using the images of the frames before and after, assuming that the pixel is at the position at the reference time point.

Further, in the technique described in Patent Document 2, the in-vehicle image pickup apparatus recognizes the moving speed information of the vehicle on which the image pickup device is mounted, and performs focal plane distortion by performing coordinate conversion of pixels based on the moving speed information. The image in the frame where is generated is corrected.

Japanese Unexamined Patent Publication No. 2006-148496 Japanese Unexamined Patent Publication No. 2012-222374

As in the technique described in Patent Document 1, an image of a frame in which focal plane distortion occurs is created by using an image of a frame in which focal plane distortion does not occur immediately before and after the image, or these images are used. In the case of correcting (complementing) with the existing information, if there is only one frame in which the focal plane distortion occurs, an image in which the focal plane distortion is corrected can be obtained in this frame. However, when there are a plurality of frames in which focal plane distortion occurs, the above processing becomes difficult because there is no frame in which focal plane distortion does not occur immediately before and after. In this case, it is necessary to use the image of the frame in which the frame is missing or the focal plane distortion is present as it is. Further, since this process is complicated, it is difficult to perform this process in real time, or the power consumption during this process is large.

Further, in the technique described in Patent Document 2, if highly accurate movement speed information can be obtained, correction can be performed with high accuracy. However, as described above, since the position of each pixel after correction changes from the actual position of each pixel, the range in the real space shown in the corrected image and the real space shown in the image before correction are shown. It is different from the range in. This difference is especially noticeable when the moving speed is high. In this case, the pixel corresponding to the corrected image may not be included in the range of the image before the correction, and in this case, the pixel is missing from the corrected image. Therefore, in the corrected video (moving image), only an unnatural video in which the frame in which the pixel is missing is inserted between the frames before and after the focal plane distortion does not occur can be obtained. There wasn't.

Therefore, a technique for reducing the influence of focal plane distortion without causing the loss of frames and pixels has been desired.

The present invention has been made in view of such a situation, and an object of the present invention is to solve the above-mentioned problems.

The present invention is an image pickup apparatus using a camera module in which an image pickup element operating in a rolling shutter system and an optical system are combined to output a video signal constituting a moving image, and the temporal change of the image pickup state in the camera module. A fluctuation detection unit that detects the rate, a blurring processing unit that performs blurring processing on the video signal, and a frame image corresponding to the time when the temporal change rate becomes equal to or higher than the threshold value in the video signal. It is provided with a control unit that instructs the blur processing unit to perform the processing.
Further, the present invention is an image pickup apparatus using a camera module in which an image pickup element operating in a rolling shutter system and an optical system are combined to output a video signal constituting a moving image, and the time of the image pickup state in the camera module. A fluctuation detection unit that detects a target change rate, an external processing information creation unit that creates external processing information that indicates the time when the temporal change rate becomes equal to or higher than a threshold value in correspondence with a frame in the video signal, and the video signal. It is provided with a control unit that outputs the external processing information in association with each other.

According to the present invention, the influence of focal plane distortion can be reduced without causing the loss of frames and pixels.

It is a figure which shows the structure of the image pickup apparatus which concerns on embodiment. It is a figure which shows the example of the image (when there is a zoom operation) output by the image pickup apparatus which concerns on embodiment. It is a figure which shows the example of the image (when there is a pan angle change) output by the image pickup apparatus which concerns on embodiment. It is a figure which shows the example of the image (when there is a zoom operation) output by a conventional image pickup apparatus. It is a figure which shows the example of the image (when there is a pan angle change) output by a conventional image pickup apparatus. It is a figure which shows the example of the image (when there is a pan angle change) output by another conventional image pickup apparatus. It is a flowchart which shows the operation of the image pickup apparatus which concerns on embodiment. It is a figure which shows the structure of the modification of the image pickup apparatus which concerns on embodiment.

Next, a mode for carrying out the present invention will be specifically described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an image pickup apparatus 1 according to an embodiment of the present invention. In this image pickup apparatus 1, a camera module 10 to which an optical system 11 and an image pickup element 12 are fixed is used. The optical system 11 is configured by combining a plurality of lenses, and the enlargement / reduction (zoom-out, zoom-in) of the angle of view is mechanically controlled by the zoom adjustment unit 13. The image sensor 12 is composed of a two-dimensional CMOS sensor.

Further, the camera module 10 is configured to be rotatable along the horizontal direction and the vertical direction with respect to the main body fixed to the facility or the like. Therefore, the object to be imaged (viewing direction) can be adjusted by adjusting the set angle (pan angle) in the horizontal direction and the set angle (tilt angle) in the vertical direction of the camera module 10. Therefore, a pan / tilt angle control unit 21 including a pan axis motor for controlling the pan angle and tilt angle of the camera module 10, a tilt axis motor, and a drive mechanism for driving the camera module by these is provided. Has been done. The control unit 20 configured by using a CPU or the like controls the zoom adjustment unit 13 and the pan / tilt angle control unit 21 by a user operation or a remote signal from the outside, and the size and direction of the angle of view of the camera module 10 Can be controlled.

The gamma correction unit 22 performs gamma correction on the moving image signal output from the image pickup device 12. Finally, this video signal is output to the outside by the video signal output unit 23. With this video signal, the video captured by the camera module 10 can be displayed on an external display.

Further, the image sensor 12 sequentially outputs the outputs of the pixels arranged in two dimensions in a constant format. At this time, the exposure timing of each pixel is sequentially set from the upper line to the lower side for each column (horizontal line) in the two-dimensional array. That is, the image sensor 12 is driven by the rolling shutter method. Therefore, when the size and direction of the angle of view in the camera module 10 suddenly fluctuate, focal plane distortion may occur in the video signal output from the image sensor 12.

Here, in the image pickup apparatus 1, it is possible to perform a blurring process on the image for each frame in the output video signal. Here, the blurring process is a filtering process for removing spatial high-frequency components in a two-dimensional image signal, and is performed by a method using a well-known algorithm. The video signal passes through the blurring processing unit 24 that performs such a blurring process using a filter in front of the video signal output unit 23.

This blurring process is performed only on the image of the frame in which the time variation of the imaging state is large. The temporal fluctuation of the imaging state includes the fluctuation of the size of the angle of view and the fluctuation of the direction of the angle of view. In order to detect the former fluctuation, the camera module 10 is provided with a zoom motion detection unit 14. Further, in order to detect the latter fluctuation, a pan / tilt operation detecting unit 25 for detecting the fluctuation of the pan angle and the tilt angle of the camera module 10 is provided. The zoom motion detection unit 14 and the pan / tilt motion detection unit 25 are examples of fluctuation detection units for recognizing the temporal change rate of the imaging state.

Here, the zoom motion detection unit 14 recognizes the rate of change in the zoom motion (rate of change over time). It consists of, for example, a sensor that recognizes the position of the lens. However, it is not necessary to be able to recognize the time change rate of the zoom operation with high accuracy, and it is sufficient if the amount of change in the position (focal length) of this lens can be recognized. On the other hand, the pan / tilt motion detection unit 25 is composed of a sensor that recognizes the pan angle and the rate of change of the tilt angle as described above by, for example, the rotation speed of the pan axis motor and the tilt axis motor, and determines the pan angle and the tilt angle. There is no need to recognize with high accuracy. Alternatively, an acceleration sensor or the like that can detect the rate of change of the pan angle and the tilt angle may be used.

The zoom motion detection unit 14 and the pan / tilt motion detection unit 25 are set so that these rate of change can be recognized in real time in frame units of video signals. If the control unit 20 can also recognize these rate of change in real time by the signal for controlling the zoom adjustment unit 13 and the pan / tilt angle control unit 21, it is for measuring these rate of change. The zoom operation detection unit 14 and the pan / tilt operation detection unit 25 are unnecessary, and the zoom adjustment unit 13 and the pan / tilt angle control unit 21 themselves can be used as the fluctuation detection unit.

An example of a moving image obtained by this processing will be specifically described when an image including the character “F” is captured together with a grid at regular intervals. First, a case where the zoom operation is performed without changing the pan angle and the tilt angle will be described. In FIG. 2, it is assumed that the process of reducing the angle of view (zooming in) is performed only in the second frame (frame 2), and in the frame 3 immediately after that, compared with the case in the frame 1 immediately before that. "F" is expanded.

FIG. 2A shows an ideal image in which focal plane distortion does not exist in this case, and FIG. 2B shows an image in the case where focal plane distortion exists as an image for each frame. As described above, since the image sensor 12 operates by the rolling shutter method, in the image of the frame 2 of FIG. 2 (b), the character "F" is ideal (frame 2 of FIG. 2 (a)). It is displayed with large distortion due to plane distortion. When the images of frames 1 to 3 are sequentially output in the moving image of this video signal, the greatly distorted image of frame 2 is conspicuous.

On the other hand, FIG. 2C is an example when the above-mentioned blurring process is performed. Here, the images of the frames 1 and 3 are the same as those of FIGS. 2 (a) and 2 (b), and only the image of the frame 2 in which the zoom operation is performed is subjected to the blurring process. When the three images in FIG. 2C are continuously displayed, it is difficult for the user to visually recognize only the short-time image in which the focal plane distortion occurs, so that the focal in the impression that the user receives as a whole moving image. The effect of plane distortion is reduced.

In the example of FIG. 2, the subject is the character "F", but for example, when the object to be photographed by the image pickup device 1 is a person, especially a famous person, the personality right due to the image having focal plane distortion is present. Infringement may occur. Such a situation can be avoided by applying a blurring process to the image of the frame in which the focal plane distortion is presumed to occur as described above.

FIG. 3 shows an image of each frame when the same zoom operation as described above is not performed at all and only the pan angle (horizontal angle) is changed while the tilt angle is constant. It is shown in the same manner. Here, as in the above case, it is assumed that the fluctuation of the pan angle occurs only in the frame 2. Here, in an ideal image (FIG. 3A) in which focal plane distortion does not exist, the character "F" moves from the left side to the right side from frame 1 to frame 3. In the case of FIG. 3 (b) where there is focal plane distortion and the above processing is not performed at all, the character "F" is distorted in the frame 2 in a manner different from that of FIG. 2 (b) due to the focal plane distortion. It is displayed in. Even in this case, in the case of FIG. 3C in which the above-mentioned blurring process is performed, only the short-time image in which the focal plane distortion occurs due to the blurring process performed only on the image of the frame 2 is displayed. It becomes difficult to see, and the influence of focal plane distortion as a whole moving image is reduced.

On the other hand, as in the prior art, the image of the frame 2 is compared with the case where the image of the frame 2 is corrected based on the images of the frames 1 and 2. FIG. 4 shows an example of an image (when a zoom operation is performed in the frame 2) corresponding to FIG. 2 (FIG. 2 (c)) in this case. Here, the ideal image and the image in the case where the focal plane distortion is present and the blurring process is not performed are shown in FIGS. 2 (a) and 2 (b), respectively. Here, as described above, this correction is corrected so that the image is at a certain reference time point, and this reference time point is assumed to be an initial time point when the image of frame 2 is captured. Therefore, in the frame 2 of FIG. 4, it is the lower region where the exposure timing is particularly late that is greatly affected by the correction.

This correction is performed on the frame 2 (in FIG. 2B) as described above, but since the lower region of the frame 2 is a zoomed-in image, the lower side to be corrected in the frame 2 The area (the area below the letter "F") is outside the range of the frame image used for correction. Therefore, in the image of the frame 2 of FIG. 4, the same image as the frame 2 of FIG. 2 (a) that should be originally obtained can be obtained on the upper side, but the corrected image cannot be obtained on the lower side (black area). ) Occurs. That is, when the correction is performed using the images of the frames before and after in this way, the image is substantially defective. In this example, the object to be imaged is the character "F", but as described above, when the object is a person or the like, such a partially missing image may be particularly problematic.

FIG. 5 also shows an image (when the pan angle is changed in the frame 2) corresponding to FIG. 3 (FIG. 3 (c)) when the same correction is performed. Here, the ideal image, the image in the case where the focal plane distortion exists and the blurring process is not performed at all, is shown in FIGS. 3 (a) and 3 (b). Also in this case, since a region outside the range of the image to be used for the correction is required for the correction, the image of the frame 2 after the correction also has a defect in a region different from that of FIG.

Further, FIG. 6 shows a case where the image of the frame 2 is corrected to be a composite image of the frames 1 and 3 in the image corresponding to FIG. 3 (FIG. 3 (c)). In this case, the image of the frame 2 is significantly different from the actual image (frame 2 of FIG. 3A). Depending on the subject, the difference between the image obtained by such a compositing process and the original image becomes extremely large.

In the image pickup apparatus 1 of FIG. 1, as shown in FIGS. 2 and 3, the image defect as described above does not occur in the frame 2, and the corrected image is between the original image. Differences can be made inconspicuous. Further, since the process of performing the blurring process on the image for each frame as described above is simpler than the correction process described in Patent Document 1 and the like, the time required for the process is short and the power consumption due to this is small. Therefore, this process can be easily performed in real time at the time of imaging.

FIG. 7 is a flowchart of a process for determining whether to perform a blurring process on the image of each frame. This process is performed frame by frame. Here, the control unit 20 determines the change in the image pickup state from the image sensor 12 during the exposure period (the period from the first exposure timing to the last exposure timing when this image is obtained) to the zoom operation detection unit 14, pan / tilt. It is detected by the motion detection unit 25 (variation detection unit) (S1). Here, the zoom motion detection unit 14 directly reflects the amount of change in the zoom, for example, the position of the lens (focal length) at the start of exposure and the position of the lens (focal length) at the end of exposure. The difference (mm) of the lens is recognized as the amount of change in the zoom, and the pan / tilt motion detection unit 25 can recognize, for example, the pan angle and the change speed (° / sec) of the pan angle and the tilt angle.

Next, the control unit 20 determines whether or not the recognized zoom change amount is equal to or greater than a predetermined threshold value (S2). When the value is equal to or higher than the threshold value (S2: Yes), the control unit 20 instructs the blurring processing unit 24 to perform the blurring process on the obtained image (S3), and the processed image is this frame. Is output as an image of (S4).

When the recognized amount of change in zoom is less than a predetermined threshold value (S2: No), the control unit 20 determines whether or not the change in pan angle and tilt angle is large (S5). This determination is made by comprehensively looking at changes in the pan angle and tilt angle. For example, when the change speed of the pan angle is at least a preset threshold value (pan angle change speed threshold value) or more, the change speed of the tilt angle is equal to or higher than a preset threshold value (tilt angle change speed threshold value). In any of the cases, the changes in the pan angle and the tilt angle can be set to be large (S5: Yes). Alternatively, for example, in order to comprehensively evaluate these angle change rates, this determination is made based on the sum of squares of the pan angle change speed and the tilt angle change speed and the magnitude relationship of the corresponding threshold values. You may let me do it. When it is determined that the changes in the pan angle and the tilt angle are large (S5: Yes), the image is subjected to the blurring process in the same manner as described above, and then output (S4).

When it is not determined that the changes in the pan angle and the tilt angle are large (S5: No), the blurring process is not performed (S6), and the image as it is is output as the image of this frame (S4). By performing the above operation for each frame, a video signal that has been subjected to blurring processing is output only to the image of the frame in which the focal plane distortion has occurred as described above.

As for the threshold values used in the above determinations (S2, S5), the zoom change amount, the pan angle, and the tilt angle change speed are not zero, but since these are small, the blurring process is not performed in the range where the focal plane distortion is not conspicuous. Is set as appropriate. Alternatively, in the image pickup apparatus 1, these threshold values may be set by the user. Further, when the halo treatment is applied, the degree of the halo treatment may be changed according to these changes. For example, when the degree of focal plane distortion is large due to the high change amount, the halo treatment may be performed. The degree of halo can be increased.

Next, a modified example of the above-mentioned image pickup apparatus 1 will be described. The above-mentioned image pickup apparatus 1 can output the video signal subjected to the above-mentioned processing to the outside, and can display this video signal as it is on a display or the like outside. However, it is also possible to perform the same processing outside the image pickup apparatus immediately before displaying the video signal. The image pickup apparatus 2 as a modified example is configured assuming such a case. FIG. 8 is a block diagram corresponding to FIG. 1 showing the configuration.

In this image pickup apparatus 2, unlike the image pickup apparatus 1, the blurring processing unit 24 is not used, so that the blurring processing is not performed at all and the focal plane distortion is generated in some frames. Is emitted to the outside via the video signal output unit 23. On the other hand, instead of controlling the halo processing unit 24 in the above case, the control unit 20 determines for each frame whether or not the change in the imaging conditions as described above is large (whether or not the halo processing is required). An external processing information creating unit 26 for creating external processing information showing the results (S2, S5 in FIG. 6) is provided. The external processing information can be created in any format as long as the frame that requires blurring processing is specified in the output video signal, and can be in a format that becomes metadata attached to the image signal. ..

According to this image pickup apparatus 2, an unprocessed video signal is obtained by an external device, and the device side performs the blurring process using the same blurring processing unit as described above, and images an image based on the blurring process. It can be displayed in the same manner as when the device 1 is used. In this case, if necessary, it is possible to see an image on the device side in which focal plane distortion exists but the blurring process is not performed. In addition, the degree of blurring can be adjusted by an external device.

In the above example, it is set to perform the blurring process when the size of the angle of view of the camera module 10 fluctuates (zoom operation) and the direction of the angle of view fluctuates (changes in pan and tilt angles). However, when a temporal change in the imaging state that causes focal plane distortion occurs, the same effect can be obtained by performing the same blurring process. For example, the same can be applied even when there is a sudden movement of the image pickup apparatus main body in the horizontal and vertical directions instead of the camera module. The operation to be recognized as such a change in the imaging state to be targeted is appropriately set according to the configuration of the camera module (imaging apparatus). Accordingly, the fluctuation detection unit, the threshold value for the above determination, and the like are appropriately set.

In addition, the degree of blurring can be set as appropriate. For example, when the frames in which focal plane distortion occurs are continuous, it is not necessary to perform the same processing on all the continuous frames. For example, when focal plane distortion occurs continuously in three or more frames, the degree of blurring may be reduced in the first and last frames in time and increased in the center frame. ..

Further, if the exposure time of the image sensor becomes long, focal plane distortion is likely to occur. Therefore, if the exposure time is longer than the threshold value, it may be determined that the blurring process is performed. Alternatively, the longer the exposure time, the lower the threshold value (threshold value of zoom change amount, threshold value of pan angle / tilt angle change speed) for determining that the blurring process is performed.

The present invention has been described above based on the embodiments. It is understood by those skilled in the art that this embodiment is an example, and that various modifications are possible in the combination of each of these components, and that such modifications are also within the scope of the present invention.

1, 2 Image sensor 10 Camera module 11 Optical system 12 Image sensor 13 Zoom adjustment unit 14 Zoom motion detection unit (variation detection unit) 20 Control unit 21 Pan / tilt angle control unit 22 Gamma correction unit 23 Video signal output unit 24 Halo Processing unit 25 Pan / tilt motion detection unit (variation detection unit) 26 External processing information creation unit

The present invention is an image pickup apparatus using a camera module in which an image pickup element operating in a rolling shutter system and an optical system are combined to output a video signal constituting a moving image, and the temporal change of the image pickup state in the camera module. A fluctuation detection unit that detects the rate, a blurring processing unit that performs blurring processing on the video signal, and a frame image corresponding to the time when the temporal change rate becomes equal to or higher than the threshold value in the video signal. It is provided with a control unit that instructs the blur processing unit to perform the processing.
Further, the present invention is an image pickup apparatus using a camera module in which an image pickup element operating in a rolling shutter system and an optical system are combined to output a video signal constituting a moving image, and the time of the image pickup state in the camera module. rate of change and the variation detection unit for detecting the out of frame of the video signal, and the external processing information creation unit for creating an external processing information indicating a frame of the video signal in which the temporal change rate is equal to or larger than the threshold value, It includes a control unit that outputs the video signal and the external processing information.

Claims (4)

An image pickup device that uses a camera module that outputs a video signal that constitutes a moving image by combining an image pickup element that operates in a rolling shutter system and an optical system.
A fluctuation detection unit that detects the temporal change rate of the imaging state in the camera module, and
In the video signal, the halo processing unit that performs the halo processing and
In the video signal, a control unit that instructs the halo processing unit to perform halo processing on the image of the frame corresponding to the time point when the temporal change rate becomes equal to or higher than the threshold value.
An imaging device characterized by comprising. In the halo processing unit,
The image pickup apparatus according to claim 1, wherein the degree of the blurring process is set to be large when the rate of change with time is large. The first aspect of claim 1, wherein the fluctuation detection unit recognizes, as the temporal change rate, the amount of change in the zoom operation of the optical system or the speed of the operation that changes the viewing direction of the camera module. Imaging device. An image pickup device that uses a camera module that outputs a video signal that constitutes a moving image by combining an image pickup element that operates in a rolling shutter system and an optical system.
A fluctuation detection unit that detects the temporal change rate of the imaging state in the camera module, and
An external processing information creation unit that creates external processing information that indicates the time when the temporal change rate becomes equal to or higher than the threshold value in correspondence with the frame in the video signal.
A control unit that outputs the video signal in association with the external processing information,
An imaging device characterized by comprising.

Images