WO2020170723A1 - Dispositif d'imagerie - Google Patents

Dispositif d'imagerie Download PDF

Info

Publication number
WO2020170723A1
WO2020170723A1 PCT/JP2020/002771 JP2020002771W WO2020170723A1 WO 2020170723 A1 WO2020170723 A1 WO 2020170723A1 JP 2020002771 W JP2020002771 W JP 2020002771W WO 2020170723 A1 WO2020170723 A1 WO 2020170723A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
frame
video signal
imaging device
focal plane
Prior art date
Application number
PCT/JP2020/002771
Other languages
English (en)
Japanese (ja)
Inventor
拓洋 澁谷
Original Assignee
株式会社日立国際電気
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社日立国際電気 filed Critical 株式会社日立国際電気
Priority to JP2021501759A priority Critical patent/JP7121852B2/ja
Publication of WO2020170723A1 publication Critical patent/WO2020170723A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules

Definitions

  • the present invention relates to an image pickup apparatus using an image pickup element that operates in a rolling shutter system.
  • pixels are two-dimensionally arranged, and an operation at the time of image pickup is often performed by a rolling shutter system that sequentially performs exposure for each horizontal line.
  • a rolling shutter system that sequentially performs exposure for each horizontal line.
  • the exposure timing of the pixels on the upper line and the exposure timing of the pixels on the lower line are displaced.
  • such a shift in the exposure timing does not cause a big problem.
  • the image of the frame in which the focal plane distortion has occurred can be replaced with an image in which the images of the frames before and after the frame in which the focal plane distortion has not occurred are combined.
  • an in-vehicle image pickup device recognizes movement speed information of a vehicle equipped with the same, and performs coordinate conversion of pixels based on the movement speed information to obtain focal plane distortion. The image in the frame in which is generated is corrected.
  • the present invention has been made in view of such a situation, and an object thereof is to solve the above problems.
  • the present invention relates to an image pickup apparatus that uses a camera module that outputs a video signal that forms a moving image by combining an image pickup device that operates in a rolling shutter system and an optical system, and a temporal change of an image pickup state in the camera module.
  • a fluctuation detection unit that detects a rate, a blur processing unit that performs blur processing in the video signal, and a frame image that corresponds to a time point when the temporal change rate is equal to or greater than a threshold in the video signal.
  • a control unit for instructing the blur processing unit to perform the processing.
  • the present invention is also an imaging device using a camera module that outputs a video signal that forms a moving image by combining an imaging device that operates in a rolling shutter system and an optical system, and the time of the imaging state in the camera module is used.
  • Change detection unit that detects a dynamic change rate
  • an external process information creation unit that creates external process information that indicates the time when the temporal change rate is equal to or greater than a threshold value in association with the frame in the video signal
  • the video signal And a control unit that outputs the external processing information in association with each other.
  • FIG. 6 is a diagram showing an example of a video (when there is a zoom operation) output by the image pickup apparatus according to the embodiment.
  • FIG. 6 is a diagram showing an example of an image (when there is a pan angle change) output by the image pickup apparatus according to the embodiment.
  • It is a figure which shows the example of the image
  • It is a figure which shows the example of the video output (when there is a pan angle change) output by the conventional imaging device.
  • 6 is a flowchart showing an operation of the image pickup apparatus according to the embodiment. It is a figure which shows the structure of the modification of the imaging device which concerns on embodiment.
  • FIG. 1 is a block diagram showing a configuration of an image pickup apparatus 1 according to the embodiment of the present invention.
  • a camera module 10 in which an optical system 11 and an image pickup element 12 are fixed is used.
  • the optical system 11 is configured by combining a plurality of lenses, and the enlargement/reduction of the angle of view (zoom out, zoom in) is mechanically controlled by the zoom adjustment unit 13.
  • the image sensor 12 is composed of a two-dimensional CMOS sensor.
  • the camera module 10 is configured to be rotatable in the horizontal direction and the vertical direction with respect to the main body fixed to a facility or the like. Therefore, by adjusting the set angle (pan angle) in the horizontal direction and the set angle (tilt angle) in the vertical direction of the camera module 10, the object to be imaged (viewing direction) can be adjusted. Therefore, a pan/tilt angle control unit 21 including a pan axis motor and a tilt axis motor for controlling the pan angle and the tilt angle of the camera module 10 and a drive mechanism for driving the camera module by these is provided. Has been.
  • the control unit 20 configured by using a CPU or the like controls the zoom adjustment unit 13 and the pan/tilt angle control unit 21 according to a user's operation or a remote signal from the outside, and controls the size and direction of the angle of view of the camera module 10. Can be controlled.
  • the gamma correction unit 22 performs gamma correction on the video signal of the moving image output from the image sensor 12. Finally, the video signal is output to the outside by the video signal output unit 23. With this video signal, the video imaged by the camera module 10 can be displayed on an external display.
  • the image pickup device 12 sequentially outputs the outputs of the two-dimensionally arranged pixels in a fixed format. At this time, the exposure timing of each pixel is sequentially set from the upper line toward the lower side for each column (horizontal line) in the two-dimensional array. That is, the image pickup device 12 is driven by the rolling shutter method. Therefore, when the size and direction of the angle of view in the camera module 10 change abruptly, focal plane distortion may occur in the video signal output from the image sensor 12.
  • the blurring process is a filtering process for removing spatial high frequency components in the two-dimensional image signal, and is performed by a method using a well-known algorithm.
  • the video signal passes through a blur processing unit 24 that performs such blur processing using a filter before the video signal output unit 23.
  • the temporal variation of the imaging state includes the variation of the size of the angle of view and the variation of the direction of the angle of view.
  • the zoom operation detection unit 14 is provided in the camera module 10. Further, in order to detect the latter variation, a pan/tilt operation detection unit 25 that detects the variation of the pan angle and the tilt angle of the camera module 10 is provided.
  • the zoom operation detecting unit 14 and the pan/tilt operation detecting unit 25 are an example of a variation detecting unit for recognizing the temporal change rate of the imaging state.
  • the zoom operation detection unit 14 recognizes the change rate (temporal change rate) of the zoom operation.
  • This is composed of, for example, a sensor that recognizes the position of the lens.
  • the pan/tilt operation detection unit 25 is configured by a sensor that recognizes the change rate of the pan angle and the tilt angle as described above, for example, by the rotation speed of the pan axis motor and the tilt axis motor. It is not necessary to recognize with high precision.
  • an acceleration sensor or the like that can detect the change rate of the pan angle and the tilt angle may be used.
  • the zoom operation detection unit 14 and the pan/tilt operation detection unit 25 are set so that the change rates of these can be recognized in real time for each frame of the video signal. If the control unit 20 can similarly recognize the change rates of these in real time by the signals for controlling the zoom adjustment unit 13 and the pan/tilt angle control unit 21, it is necessary to actually measure these change rates.
  • the zoom operation detection unit 14 and the pan/tilt operation detection unit 25 are unnecessary, and the zoom adjustment unit 13 and the pan/tilt angle control unit 21 themselves can be used as the fluctuation detection unit.
  • FIG. 2A shows an ideal image in which focal plane distortion does not exist in this case
  • FIG. 2B shows an image in which focal plane distortion exists as an image for each frame.
  • the character “F” is ideal when the focal point is from the frame 2 of FIG. It is distorted greatly due to plane distortion.
  • the images of the frames 1 to 3 are sequentially output in the moving image by this video signal, the greatly distorted image of the frame 2 is conspicuous.
  • FIG. 2(c) is an example of the case where the above-described blur processing is performed.
  • the images of the frames 1 and 3 are the same as those in FIGS. 2A and 2B, and the blur processing is applied only to the image of the frame 2 in which the zoom operation is performed.
  • the three images in FIG. 2C are continuously displayed, it is difficult for the user to visually recognize only the image for a short time in which the focal plane distortion has occurred, and thus the focal point in the impression that the user receives as a whole of the moving image. The effect of plane distortion is reduced.
  • the subject is the letter “F”.
  • the target imaged by the imaging device 1 is a person, particularly a well-known person or the like
  • the person's right of a person with a focal plane distortion is not recognized. Infringement may occur. As described above, such a situation is avoided by performing the blurring process on the image of the frame estimated to have the focal plane distortion.
  • FIG. 3 shows an image of each frame when the same zooming operation as described above is performed and the panning angle (horizontal angle) is changed while the tilt angle is kept constant. The same is shown.
  • the fluctuation of the pan angle occurs only in the frame 2.
  • FIG. 3A In an ideal image (FIG. 3A) where there is no focal plane distortion, the character “F” moves from the left side to the right side from frame 1 to frame 3.
  • FIG. 3B In the case of FIG. 3B in which the above processing is not performed at all due to the focal plane distortion, the character “F” is distorted in frame 2 in a manner different from the case of FIG. 2B due to the focal plane distortion. Is displayed. Also in this case, in the case of FIG.
  • the blur processing is performed only on the image of the frame 2, so that only a short-time image in which focal plane distortion has occurred is generated. It becomes difficult to visually recognize, and the influence of focal plane distortion in the entire moving image is reduced.
  • FIG. 4 shows an example of an image corresponding to FIG. 2 (FIG. 2C) in this case (when the zoom operation is performed in the frame 2).
  • FIGS. 2A and 2B an ideal image and an image when focal plane distortion is present and no blur processing is performed are shown in FIGS. 2A and 2B, respectively.
  • this correction is performed so that this image is at a certain reference time point, and it is assumed that this reference time point is the initial time point when the image of the frame 2 is captured. For this reason, in the frame 2 of FIG. 4, it is the lower area where the exposure timing is particularly late that is greatly affected by the correction.
  • This correction is performed on the frame 2 (as shown in FIG. 2B) as described above, but since the lower area of the frame 2 is a zoomed-in image, the lower area of the frame 2 to be corrected is corrected.
  • the area (area below the letter “F”) is outside the range of the image of the frame used for correction. Therefore, in the image of the frame 2 in FIG. 4, an image similar to that of the frame 2 in FIG. 2A, which should be originally obtained, is obtained in the upper side, but in the lower side, the corrected image cannot be obtained (black area). ) Occurs. That is, when the correction is performed using the images of the preceding and succeeding frames in this manner, the image is substantially defective.
  • the object to be imaged is the letter “F”, but as described above, when the object is a person or the like, such a partially missing image may be a particular problem.
  • FIG. 5 similarly shows an image (when the pan angle changes in frame 2) corresponding to FIG. 3 (FIG. 3C) when the same correction is performed.
  • FIGS. 3A and 3B an ideal image and an image in the case where focal plane distortion exists and no blur processing is performed are shown in FIGS. 3A and 3B.
  • a region outside the range of the image used for the correction is necessary for the correction, and thus the corrected image of the frame 2 also has a defect in a region different from that in FIG.
  • FIG. 6 shows a case where the image corresponding to FIG. 3 (FIG. 3(c)) is corrected such that the image of the frame 2 is a combined image of the frames 1 and 3.
  • the image of frame 2 is significantly different from the actual image (frame 2 of FIG. 3A).
  • the difference between the image obtained by such synthesis processing and the original image becomes extremely large.
  • the image loss as described above does not occur in the frame 2, and the image between the corrected image and the original image is not generated. You can make the difference inconspicuous. Further, as described above, the process of performing the blurring process on the image for each frame is simpler than the correction process described in Patent Document 1 and the like, so that the time required for the process is short and the power consumption by the process is small. Therefore, this process can be easily performed in real time at the time of image capturing.
  • FIG. 7 is a flowchart of a process of determining whether to perform a blur process on each frame image. This process is performed for each frame.
  • the control unit 20 changes the imaging state during the exposure period (the period from the first exposure timing to the last exposure timing when obtaining this image) from the image sensor 12, the zoom operation detection unit 14, the pan/tilt. It is detected by the motion detector 25 (variation detector) (S1).
  • the zoom operation detection unit 14 uses, for example, the position of the lens (focal length) when the exposure is started and the position of the lens (focal length) when the exposure is ended as the amount that directly reflects the zoom change amount. (Mm) is recognized as the amount of zoom change, and the pan/tilt operation detection unit 25 can recognize the pan/tilt angle change speed (°/sec), for example.
  • control unit 20 determines whether the recognized zoom change amount is equal to or more than a predetermined threshold value (S2). When it is equal to or more than the threshold value (S2: Yes), the control unit 20 instructs the blur processing unit 24 to perform blur processing on the obtained image (S3), and the processed image is the frame. Is output as an image (S4).
  • the control unit 20 determines whether or not the change in the pan angle and the tilt angle is large (S5). This determination is made by comprehensively looking at the changes in the pan angle and the tilt angle. For example, if at least the pan angle change speed is equal to or more than a preset threshold (pan angle change speed threshold), the tilt angle change speed is equal to or greater than a preset threshold (tilt angle change speed threshold). In any of the cases, the change in the pan angle and the tilt angle can be large (S5: Yes).
  • this determination is made based on the magnitude sum of the square sum of the pan angle change speed and the tilt angle change speed, and the threshold value set in advance corresponding thereto. May be done.
  • S5 the image is subjected to the blurring process in the same manner as described above (S3) and then output (S4).
  • the blur processing is not performed (S6), and the image as it is is output as the image of this frame (S4).
  • the video signal subjected to the blur processing is output only to the image of the frame in which the focal plane distortion occurs as described above.
  • the threshold value used for the above determinations (S2, S5) is not zero in the zoom change amount and the change speed of the pan angle and tilt angle, but the blur processing is not performed in a range where the focal plane distortion is not conspicuous because these are small Is set as appropriate.
  • these thresholds may be settable by the user.
  • the degree of the blurring process may be changed according to the variation amount. For example, when the focal plane distortion is large due to the high variation amount, the blurring amount is large. It is possible to increase the degree of bending.
  • the image pickup apparatus 1 can output the video signal that has been subjected to the above processing to the outside, and can display the video signal as it is on the display outside. However, it is also possible to perform the same processing outside the imaging device immediately before displaying the video signal.
  • the imaging device 2 as a modified example is configured assuming such a case.
  • FIG. 8 is a block diagram showing the configuration corresponding to FIG.
  • the image pickup apparatus 2 does not use the blurring processing unit 24, so that the blurring processing is not performed at all, and a video signal in which focal plane distortion has occurred in some frames. Is transmitted to the outside via the video signal output unit 23.
  • the control unit 20 determines for each frame whether or not the change in the above-described imaging condition is large (whether or not the blur processing is required).
  • An external process information creating unit 26 is provided which creates external process information indicating the result (S2, S5 in FIG. 6).
  • the external processing information may be created in any format as long as a frame that requires gradation processing is specified in the output video signal, and may be in a format that becomes metadata accompanying the image signal. ..
  • this image pickup device 2 an image signal that has not been processed is obtained by an external device, and this device performs the blur processing using the same blur processing section as described above, and images based on this are captured. It can be displayed in the same manner as when the device 1 is used. In this case, if necessary, it is possible to view an image on which the focal plane distortion is present but the blur processing is not performed on the device side. In addition, adjustment of the degree of blurring processing can be performed by an external device.
  • the blurring process is set to be performed when the size of the angle of view in the camera module 10 changes (zoom operation) and the direction of the angle of view changes (changes in pan and tilt angles).
  • the same effect can be obtained by performing the blurring process in the same manner.
  • the same can be applied when there is a sudden movement in the horizontal and vertical directions of the image pickup apparatus main body instead of the camera module.
  • the operation to be recognized as the change of the image pickup state to be the target is appropriately set according to the configuration of the camera module (image pickup apparatus).
  • the fluctuation detection unit, the threshold value for the above determination, and the like are also appropriately set.
  • the degree of shading processing can be set appropriately. For example, when the frames in which the focal plane distortion occurs are continuous, it is not necessary to perform the same processing on all the continuous frames. For example, when the focal plane distortion continuously occurs in three or more frames, the degree of blurring processing may be reduced in the first and last frames in time and increased in the central frame. ..
  • the threshold value for determining that the blurring process is performed may be set lower as the exposure time is longer.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)

Abstract

La présente invention réduit les effets de distorsion de plan focal sans provoquer l'apparition de trames ou de pixels manquants.　La fig. 2 (a) représente une vidéo dans laquelle une distorsion de plan focal n'est pas présente, et la fig. 2 (b) représente une vidéo dans laquelle une distorsion de plan focal est présente. La fig. 2 (c) représente une vidéo obtenue par réalisation d'un traitement de gradation sur une vidéo dans laquelle une distorsion de plan focal est présente. Lorsque les trois images de la fig. 2 (c) sont affichées successivement, les effets de la distorsion du plan focal, pour la vidéo dans son ensemble, comme perçu par un utilisateur, sont réduits.
PCT/JP2020/002771 2019-02-18 2020-01-27 Dispositif d'imagerie WO2020170723A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2021501759A JP7121852B2 (ja) 2019-02-18 2020-01-27 撮像装置

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019-026249 2019-02-18
JP2019026249 2019-02-18

Publications (1)

Publication Number Publication Date
WO2020170723A1 true WO2020170723A1 (fr) 2020-08-27

Family

ID=72145048

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/002771 WO2020170723A1 (fr) 2019-02-18 2020-01-27 Dispositif d'imagerie

Country Status (2)

Country Link
JP (1) JP7121852B2 (fr)
WO (1) WO2020170723A1 (fr)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010035133A (ja) * 2008-02-26 2010-02-12 Canon Inc 動画像符号化装置及び動画像符号化方法
JP2016058876A (ja) * 2014-09-09 2016-04-21 キヤノン株式会社 撮像装置及びその制御方法、及び撮像システム

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010035133A (ja) * 2008-02-26 2010-02-12 Canon Inc 動画像符号化装置及び動画像符号化方法
JP2016058876A (ja) * 2014-09-09 2016-04-21 キヤノン株式会社 撮像装置及びその制御方法、及び撮像システム

Also Published As

Publication number Publication date
JPWO2020170723A1 (ja) 2021-12-02
JP7121852B2 (ja) 2022-08-18

Similar Documents

Publication Publication Date Title
CN109842753B (zh) 摄像头防抖系统、方法、电子设备和存储介质
CN110012224B (zh) 摄像头防抖系统、方法、电子设备和计算机可读存储介质
US8149280B2 (en) Face detection image processing device, camera device, image processing method, and program
TWI446098B (zh) 成像裝置及其快門驅動模式選擇方法
US7782362B2 (en) Image pickup device for changing a resolution of frames and generating a static image based on information indicating the frames
CN109951638B (zh) 摄像头防抖系统、方法、电子设备和计算机可读存储介质
US20120057028A1 (en) Imaging system and pixel signal readout method
JP2011130383A (ja) 撮像装置及びその制御方法
JP2009010616A (ja) 撮像装置および画像出力制御方法
US8970711B2 (en) Imaging apparatus for correcting distortion in image captured using rolling shutter method and distortion correction method
JP4245045B2 (ja) 撮像装置、撮像信号処理方法及びプログラム
US7936385B2 (en) Image pickup apparatus and imaging method for automatic monitoring of an image
US20150288949A1 (en) Image generating apparatus, imaging apparatus, and image generating method
JP4235510B2 (ja) デジタルカメラ及び固体撮像装置
JP5387341B2 (ja) 撮像装置
WO2017104102A1 (fr) Dispositif d'imagerie
JP2009135713A (ja) 撮像装置のブレで生じる画像歪みの補正方法及び装置
JP5393877B2 (ja) 撮像装置および集積回路
WO2020170723A1 (fr) Dispositif d'imagerie
JP2019080261A (ja) 撮像装置
JP4969349B2 (ja) 撮像装置及び撮像方法
JP2012244243A (ja) 撮像装置および撮像装置の制御方法
JP6843651B2 (ja) 画像処理装置、画像処理装置の制御方法、及び、プログラム
JP2014216692A (ja) 高解像度化処理付き撮影装置
JP2014216694A (ja) 高解像度化処理付き追尾雲台装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20758930

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021501759

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20758930

Country of ref document: EP

Kind code of ref document: A1