JP4977569B2 - Imaging control apparatus, imaging control method, imaging control program, and imaging apparatus - Google Patents

Description

  The present invention relates to a photographing control device that controls photographing in a photographing device, a photographing control method, a photographing control program that causes an information processing device to operate as a photographing control device, and a photographing device that includes the photographing control device.

  Recently, CMOS type image sensors have been used instead of CCD solid-state image sensors. This CMOS type image sensor requires a plurality of power supplies, whereas the CCD solid-state image sensor requires a single power supply, so the configuration around the power supply is very simple and can be downsized. And the advantage of reduced power consumption due to the CMOS type structure.

  However, while a CCD solid-state image sensor can perform all-pixel simultaneous exposure with an electronic shutter, in the case of shooting with a rolling shutter of a CMOS type image sensor, exposure is non-simultaneous when the subject is a moving object. Therefore, there is a drawback that rolling distortion occurs.

  Therefore, Patent Documents 1 to 3 propose techniques for solving the drawbacks.

  In Patent Document 1, a plurality of images are taken continuously, a moving object in the image is detected using a plurality of images, and signal processing such as linear interpolation is performed according to the speed of the moving object to correct rolling distortion. Techniques to do this are disclosed.

  Patent Document 2 discloses a technique for measuring the speed of a moving object using a speed sensor and correcting rolling distortion in accordance with the speed of the moving object.

In Patent Document 3, a product on a line is photographed in advance at a predetermined location and a reference image is recorded, and a photographed image photographed at that location when the product is flowing on the line and the reference image described above. A technique for correcting rolling distortion at the time of photographing by comparing the two is disclosed.
Japanese Patent Laid-Open No. 2006-148496 JP 2004-96498 A JP 2006-58945 A

  By the way, when continuous shooting is performed by rolling shutter shooting, rolling distortion occurs every shooting and the position of the moving object moves even if the background is the same every shooting.

  If the technique disclosed in Patent Document 1 and Patent Document 2 is used to correct the rolling distortion at each shooting time during continuous shooting, the rolling distortion is caused based on the movement of the moving object at each shooting time during continuous shooting. As a result of the correction, the background may be corrected and the background may be distorted.

  In order to prevent this, it is conceivable to mount a global shutter function on a CMOS type image sensor. However, since it is difficult to miniaturize pixels, the performance is equivalent to that of a CMOS type image sensor without a global shutter. It is very difficult to get.

  The technique disclosed in Patent Document 3 is an effective technique for photographing the same subject, and cannot be applied to a normal photographing apparatus.

  SUMMARY OF THE INVENTION In view of the above circumstances, the present invention provides a photographing control device, a photographing method, and a photographing method thereof that do not cause distortion in the background even if rolling distortion is corrected when continuous photographing is performed by rolling shutter photographing. It is an object of the present invention to provide a photographing program to be realized and a photographing device provided with the photographing control device.

An imaging control apparatus of the present invention that achieves the above object is composed of a plurality of pixel lines, and each pixel line is sequentially different after a predetermined exposure time has elapsed after the electronic shutter is opened sequentially at each pixel line. An imaging control device that controls imaging in an imaging device including an imaging element capable of rolling shutter imaging that reads an image signal at a timing,
The photographing apparatus has a continuous photographing mode for continuously photographing a plurality of images,
When the photographing device is in the continuous photographing mode, the photographing device performs continuous rolling shutter photographing, and at least one of the plural photographings is taken from the imaging element. Shooting control means for performing reference shooting for generating a reference image with relatively low rolling distortion by relatively high-speed reading of
Distortion correction means for correcting a rolling distortion of a distortion correction target image obtained by photographing excluding the reference photographing among the plurality of rolling shutter photographing based on the reference image.

  According to the photographing control device of the present invention, when the photographing device is in the continuous photographing mode, the photographing device is configured to perform reference photographing on the photographing device by high-speed reading from the image sensor for at least one of the plural photographings. Thus, based on the reference image with relatively little rolling distortion, the rolling distortion of the distortion correction target image obtained by shooting other than the reference shooting is corrected.

  In other words, a reference image with a relatively small rolling distortion is generated by the reference shooting, and the rolling distortion of the distortion correction target image is corrected so as to match the reference image.

  As a result, the rolling distortion is suitably corrected in continuous shooting, and the background is corrected so as to match the background in the reference image, thereby suppressing the background from being distorted.

Here, when the photographing control unit causes the photographing apparatus to perform reference photographing that generates a reference image with relatively low rolling distortion by relatively high-speed reading from the imaging element,
The imaging control unit may cause the imaging apparatus to generate the reference image by reference imaging that causes the imaging device to perform relative high-speed reading by thinning out to a relatively small number of pixels. ,
Further, the photographing control means causes the photographing apparatus to perform relative high-speed reading with the number of bits obtained by relatively reducing the number of bits representing a pixel value per pixel from the image pickup device. It may be one that generates a reference image,
Further, the imaging control means causes the imaging apparatus to generate the reference image by reference imaging that causes the imaging device to perform high-speed reading by reading only a relatively narrow partial area of the imaging element. It may be a thing.

  In any case, the reference image of the subject to be photographed is generated in a short time.

Further, the photographing apparatus includes a mechanical shutter that opens and closes the subject light that irradiates the imaging element,
The photographing control unit may cause the photographing apparatus to perform reference photographing that generates a reference image that avoids rolling distortion by photographing using the mechanical shutter instead of the reference photographing.
The photographing control unit may cause the photographing apparatus to perform reference photographing using the mechanical shutter and high-speed reading.

  When the mechanical shutter is used, it is possible to obtain a reference image having no rolling distortion by performing the reference photographing by the global shutter photographing.

Here, a first range in which the level of rolling distortion of the distortion correction target image with respect to the reference image is less than a first reference value, the first reference value, and a second reference greater than the first reference value. A distortion level determination means for determining which range is a second range that is intermediate to the value and a third range that is equal to or greater than the second reference value ;
It is preferable that the distortion correction unit corrects the rolling distortion of the distortion correction target image only when the distortion level determination unit determines that the image is in the second range.

  When the level of the rolling distortion is in the first range, an image having a certain level of image quality can be obtained without any particular correction. When the level of the rolling distortion is in the third range, the rolling distortion is corrected even if it is corrected. A distorted image is obtained without being cut. Therefore, it is preferable to use a configuration in which rolling distortion is corrected only when it is in the second range.

Further, the distortion correction means includes a distortion correction availability determination means for determining whether or not the image is an image capable of correcting rolling distortion.
It said distortion compensation means only if it the distortion corrected image is an image that can correct the rolling strain is determined by the distortion correction Allowed determination means, rolling distortion of the distortion correction image It is also possible to correct this.

Here, the photographing control means may perform reference photographing for at least the first one of a plurality of consecutive photographs.
Further, the photographing control means may cause the reference photographing to be performed every time a predetermined number of photographs are taken.

  As described above, when the subject is a moving object, the moving object moves each time continuous shooting is performed even if the subject is the same background. Therefore, it is possible to perform reference shooting as necessary, or to perform reference shooting for every predetermined number of frames, and to determine whether the moving object that is the main subject is out of the shooting angle of view. good.

Furthermore, a motion detection means for detecting the motion of the subject between two images obtained by photographing at adjacent times is provided,
The photographing control unit may cause the photographing apparatus to perform the reference photographing for each of a plurality of photographings according to the speed of movement of the subject detected by the motion detecting unit.

  Then, when the subject is fast, reference shooting is performed at relatively short intervals, and the position of a fast-moving moving object is confirmed and corrected based on the reference image obtained by the reference shooting. It can be determined whether it is possible. In this way, when the subject is slow, the moving object, which is the main subject, is unlikely to deviate from the shooting angle of view, so the base image is taken at relatively long intervals, and the distortion correction processing is performed based on the reference image obtained by the standard shooting. When the subject is fast, the moving object, which is the main subject, moves out of the shooting angle of view within a short time, so the reference image obtained by performing the reference shooting at relatively short intervals is used. The distortion correction process can be performed based on the above.

  Here, when the motion of the subject detected by the motion detection means is relatively fast, the moving object that is the main subject immediately deviates from the angle of view, and when the motion of the subject is relatively slow, Even if a moving object does not easily deviate from the angle of view, if continuous shooting is performed with the same number of images, if the movement of the subject is relatively fast, an image in which the main subject deviates from the angle of view at the end of continuous shooting will be displayed. It will be filmed. In view of this, it is preferable to limit the number of continuous shots in accordance with the movement of the subject detected by the motion detection means so that unnecessary shooting is not performed.

Here, the imaging control method of the present invention that achieves the above object is configured by a plurality of pixel lines, and each pixel line is opened after a predetermined exposure time has elapsed after opening the electronic shutter at different timings for each pixel line. An imaging control method for controlling imaging in an imaging apparatus having an imaging element capable of rolling shutter imaging that sequentially reads out image signals at different timings,
The photographing apparatus has a continuous photographing mode for continuously photographing a plurality of images,
When the photographing apparatus is in the continuous photographing mode, the photographing apparatus performs continuous rolling shutter photographing, and at least one of the plural photographings is relative to the imaging element. A shooting control step for performing reference shooting for generating a reference image with relatively low rolling distortion by means of typical high-speed readout;
And a distortion correction step of correcting a rolling distortion of a distortion correction target image obtained by shooting excluding the reference shooting among the plurality of rolling shutter shootings based on the reference image.

The shooting control program of the present invention for achieving the above object
The information processing apparatus is executed by an information processing apparatus that executes a program. The information processing apparatus is configured by a plurality of pixel lines, and the electronic shutter is opened at different timings for each of the pixel lines, and a predetermined exposure time is set. A shooting control program that operates as a shooting control device that controls shooting in a shooting device that includes a rolling shutter shooting that reads out image signals at different timings sequentially for each pixel line after a lapse,
The photographing apparatus has a continuous photographing mode for continuously photographing a plurality of images,
The information processing apparatus is
When the photographing apparatus is in the continuous photographing mode, the photographing apparatus performs continuous rolling shutter photographing, and at least one of the plural photographings is relative to the imaging element. Photographing control means for performing reference photographing for generating a reference image with relatively low rolling distortion by means of typical high-speed readout;
Based on the reference image, the image forming apparatus is operated as a shooting control device including distortion correction means for correcting rolling distortion of a distortion correction target image obtained by shooting excluding the reference shooting among the plurality of rolling shutter shootings. It is characterized by that.

Furthermore, the imaging device of the present invention that achieves the above object provides:
In a photographing device that forms an image of a subject on an image sensor with a photographing optical system and generates an image signal representing the subject with the image sensor,
The image sensor is composed of a plurality of pixel lines, and the plurality of pixel lines are sequentially opened at different timings for each pixel line after the electronic shutter is opened at different timings and a predetermined exposure time has elapsed. Rolling shutter photography that reads out image signals is possible,
Furthermore, the imaging device
It has a continuous shooting mode for continuously shooting multiple images,
Upon receiving a shooting instruction when in the continuous shooting mode, the image pickup device performs a plurality of continuous rolling shutter shootings, and at least one of the plurality of shots is taken from the image pickup device. Shooting control means for performing reference shooting for generating a reference image with relatively low rolling distortion by relatively high-speed reading of
Distortion correction means for correcting a rolling distortion of a distortion correction target image obtained by photographing excluding the reference photographing among the plurality of rolling shutter photographing based on the reference image.

Here, it is provided with a mechanical shutter that opens and closes the subject light that irradiates the image sensor,
It is preferable that the photographing control means performs reference photographing that generates a reference image that avoids rolling distortion by photographing using the mechanical shutter, instead of the reference photographing.

  According to the present invention, it is possible to realize a photographing control apparatus and a photographing control method that do not cause distortion in the background even if the rolling distortion is corrected when continuous photographing by rolling shutter photographing is performed, and the photographing control method thereof. An imaging control program and an imaging apparatus including the imaging control apparatus are realized.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is an external perspective view of a digital camera, which is a first embodiment of a photographing apparatus of the present invention, seen obliquely from the front top.

  A digital camera 100 shown in FIG. 1 captures subject light and generates image data. On the front surface of the digital camera 100, a zoom lens barrel 111 including an optical zoom lens is flashed in synchronization with photographing. A flash light emission window 112 for emitting light, an AF auxiliary light lamp 113 for improving autofocus (AF) accuracy at low contrast such as in a dark place, and the like are provided.

  On the top surface of the digital camera 100, a power switch 114, a release button 115, and a rotary mode dial 116 are provided around the release button 115. The mode dial 116 can be switched alternately between the still image mode and the moving image mode by turning the dial. Further, the digital camera 100 of the present embodiment is provided with a continuous shooting mode in the still image mode, and the continuous shooting mode is designated by a menu / OK button (described later) on the back side. ing.

  FIG. 2 is an external view showing the back of the digital camera shown in FIG.

  2 includes an LCD panel 117A, a finder 124, a MENU / OK button 118, a DISP / BACK button 119, a photo mode button 120, a playback button 121, a cross button 122, a zoom button 123, and the like. It has been.

  The LCD panel 117A displays an image composed of image data generated by capturing subject light with a photographing lens and information at various settings.

  The MENU / OK button 118 is a button for displaying various menus at the time of shooting or reproduction on the LCD panel 117A, or selecting and determining a desired menu from the displayed menus. As described above, the digital camera of the present embodiment is provided with a continuous shooting mode in the shooting mode, and the continuous shooting mode is designated by operating the MENU / OK button 118 and a cross button 122 described later. Then, continuous shooting is performed.

  The DISP / BACK button 119 is a button for switching the state of the screen displayed on the LCD panel 117A, returning the operation by the MENU / OK button 118 or the like to the previous state, or canceling the operation.

  The photo mode button 120 is a button for setting the number of pixels, sensitivity, color, number of prints, and the like.

  The reproduction button 121 is a button for reproducing and displaying a captured image recorded on a recording medium 1059 described later on the LCD 117.

  The role of the cross button 122 changes in various modes. For example, the cross button 122 is a macro button or a flash button in the still image mode, and a thumbnail selection button displayed on the LCD panel 117A in the playback mode. The macro button is a button for switching whether or not to perform macro shooting. When the macro button is pressed once, the macro shooting mode is set, and when pressed again, the macro shooting mode is canceled. The flash button is a button for sequentially switching the flash state, such as auto flash, red-eye reduction flash, forced flash, flash prohibition, slow shutter flash, and auto flash each time the button is pressed.

  The zoom button 123 is a button for zooming in on the telephoto side (tele side) by pressing down to the right and zooming out on the wide angle side (wide side) by pressing down on the left side.

  Next, the circuit configuration of the digital camera 100 shown in FIGS. 1 and 2 will be described.

  FIG. 3 is a block diagram illustrating a circuit configuration of the digital camera 100 illustrated in FIGS. 1 and 2.

  The operation of the digital camera 100 is controlled by cooperation between the CPU 130 and the image processing unit 140. The image processing unit 140 is configured by a processor dedicated to image processing, and executes an image processing process according to a program in the main memory MEM in response to an instruction from the CPU 130. The image processing unit 140 includes a memory control circuit 141, a digital signal processing circuit 142, a compression / decompression circuit 143, and an integration circuit 144.

  On the other hand, the input unit of the CPU 130 is provided with a MENU / OK button 118, a DISP / BACK button 119, a photo mode button 120, a playback button 121, a cross button 122, a zoom button 123, and a release button 114 shown in FIG. When the operation unit OP is connected and an operation signal is supplied from the operation unit OP, the CPU 130 causes the image sensor 131, the analog signal processing circuit 132, and the ADC 133 to execute processing in accordance with the operation signal, and performs imaging. The image processing of the image signal after the image signal generated by the element 131 is converted into a digital signal by the ADC 133 is executed by the image processing processor constituting the image processing unit 140. In the present embodiment, a CMOS type image sensor that performs rolling shutter imaging is used for the image sensor.

  Also, FIG. 3 shows two buses, an address bus 130B and a data bus 130C. The CPU 130 accesses the image processing unit 140 using the address bus 130 </ b> B of the two buses, or the image processing unit 140 accesses the CPU 130. In addition, the instruction content from the CPU 130 is notified to each circuit 141 to 144 in the image processing unit 140 via the data bus 130C of the two buses 130B and 130C, or conversely from each circuit 141 to 145. The progress status of the process is notified to the CPU 130.

  Further, digital image data output from the ADC 133 is guided onto the data bus 130C by the memory control circuit 141 in the image processing unit 140 in order to synchronize the processing timing between the ADC 133 and the bus 130C.

  The configuration and operation of the digital camera 100 whose operation is controlled by cooperation between the CPU 130 and the image processing unit 140 will be described with reference to FIG.

  In the photographing optical system 1110 shown in the upper left of FIG. 3, a photographing lens 1111 and a diaphragm 1112 are provided. FIG. 3 also shows that an infrared filter 1113 and an LPD (Low Pass Filter) 1114 are provided in the photographing optical system 1110.

  The taking lens 1111 includes a zoom lens and a focus lens. These lenses are driven by a lens driving unit 1301 to move in the optical axis direction, and the operation of the lens driving unit 1301 is controlled by the CPU 130. The diaphragm 1112 is driven by the diaphragm driving unit 1302, and the operation of the diaphragm driving unit 1302 is also controlled by the CPU 130. Further, the image sensor 137 is driven by the image sensor drive unit 1303, and the operation of the image sensor drive unit 1303 is also controlled by the CPU 130. Image data is generated by the image sensor 131 by rolling shutter shooting in accordance with the driving by the image sensor driving unit 1303 and is output to the analog signal processing circuit 133 at the subsequent stage.

  In the analog signal processing circuit 132, noise reduction processing or the like of the image data output from the image sensor 131 is performed, and the image data subjected to the noise reduction processing is supplied to an ADC (Analog Digital Converter) 133 at the subsequent stage. The ADC 133 converts analog image data into digital image data.

  After the analog image data is converted into digital image data by the ADC 133, the memory control circuit 140 in the image processing unit 140 that has received the process start signal from the CPU 130 sequentially transfers the image data for one frame to the data bus 130C. The image data guided to the top and the data bus 130C is supplied to the digital signal processing unit 142 one after another.

  Here, the flow of processing of moving image data (hereinafter referred to as “through image data”) for displaying on the LCD panel 117A (see FIG. 2) of the LCD 117 an object captured by the photographing optical system when the power is turned on. explain.

  The CPU 130 instructs the image pickup device driving unit 1303 to repeatedly drive the image pickup device 131 at every predetermined timing to perform rolling shutter shooting, thereby thinning out the image pickup device and outputting an image signal, and the analog signal processing circuit 132. The through image data after being output one after another and converted into digital image data by the ADC 133 is instructed to the image processing unit, and the data bus is successively controlled under the control of the memory control circuit 141 in the image processing unit 140. Let 130C lead.

  The through image data guided onto the data bus 130C by the memory control circuit 131 is first supplied to the digital signal processing circuit 142 via the data bus 130C. In the digital signal circuit 142, the live view data is subjected to YC conversion processing after shading correction and the like. The YC conversion process is performed to generate a YC signal from the through image data, and the generated YC signal is supplied to the display control circuit 1170 via the data bus 130C under the control of the CPU 130. A through image based on the YC signal is displayed on 117A.

  On the other hand, the image signal supplied to the digital signal processing circuit 142 is also supplied to the integrating circuit 144, and a focus position detection process or a photometric process is performed based on the image signal. In response to the integration result in the integration circuit 144, the CPU 130 instructs the lens driving unit 1301 to move the focus lens 1111 to the in-focus position, or instructs the aperture driving unit 1302 to change the aperture diameter. Yes. Therefore, a through image based on the through image data adjusted to the appropriate exposure is displayed on the LCD panel 117A at any time.

  Here, when the photographer presses the release button 115 by referring to the through image on the LCD panel 117A, the CPU 130 receives the pressing operation of the release button 115 and starts shooting processing.

  The CPU 130 instructs the image sensor driving unit 1303 to cause the image sensor 131 to start rolling shutter shooting, and outputs an image signal composed of all pixels to analog signal processing. Image data composed of all pixels output from the image sensor 131 is reduced in noise by the analog signal processing circuit 132 and converted into digital image data by the ADC 133.

  The digital image data output from the ADC 133 is guided to the data bus 130C by the memory control circuit 141 under the control of the CPU 130, and the image data guided to the data bus 130C is supplied to the digital signal processing circuit 142. . The digital signal processing circuit 142 performs shading correction and then performs conversion processing to a YC signal, and the YC signal generated by the conversion processing is supplied to the compression / expansion circuit 143. The compression / decompression circuit 143 compresses the YC signal and supplies the compressed YC signal to the external memory control circuit 150.

  At this time, the CPU 130 adds shooting information and compression information to the compressed YC signal to form a compressed YC signal, and the image file is recorded on the recording medium 151.

  3 shows a light emitting unit ILL that emits flash light from the light emitting window 112 shown in FIG. Since the light emitting unit ILL has a dimming function, it includes a light emitting unit DET for receiving light.

  Here, a continuous shooting process performed in cooperation with the CPU 130 and the image processing unit 140 when the continuous shooting mode is designated by operating the menu / OK button 118 or the cross key 122 will be described.

  FIG. 4 is a diagram illustrating an initial state of a subject for which continuous shooting is performed.

  FIG. 5 is a diagram for explaining the continuous shooting process performed by the digital camera of FIGS. 1 and 2 during continuous shooting.

  FIG. 4 is a diagram showing a photographic subject indicated by a through image displayed on the LCD panel 117A. The automobile in the image is a moving object and is traveling at a predetermined speed. In the case of a live view image, pixels are thinned out and read out, so that rolling distortion is small and hardly noticeable.

  Here, when continuously photographing the photographic subject, all pixels are read out from the image sensor 131, so that the rolling distortion of the subject increases every time the photographing is performed, and the image quality deteriorates.

  Therefore, in this embodiment, the distortion correction unit according to the present invention is provided in the digital signal processing unit 142 included in the image processing unit 140, and the distortion correction unit can correct the distortion. This distortion correction means corrects an image with rolling distortion so as to match the reference image originally taken, and more specifically, each pixel between the reference image and the distortion correction target image. Alternatively, the motion vector for each small area is detected, and the rolling distortion and background of the moving object are adaptively corrected based on the motion vector.

  FIG. 5 is a diagram for explaining a distortion correction state performed by the distortion correction means in the digital signal processing circuit 142.

  As shown in FIG. 5, when the release button 115 is pressed when attempting to shoot while viewing the through image shown in FIG. 4, the CPU 130 controls the CPU 130 according to the integration by the integration circuit 144 when the release button 115 is half-pressed. After the focus adjustment and exposure adjustment are performed by the lens driving unit 1301 and the aperture driving unit 1302, the image sensor 131 performs reference photographing by driving the image sensor driving unit 1303 to generate a reference image. At this time, high-speed reading is performed from the image sensor 131 so that continuous shooting is not affected, and the reference image is supplied into the digital signal processing circuit 142 and stored in the internal frame memory within a short time.

  In this example, the same processing as in the case of a through image is performed, and the CPU 130 serving as a photographing control unit according to the present invention is added to the digital camera shown in FIGS. The reference image is generated by reference photographing that performs relatively high-speed reading by thinning out the number of pixels from the image sensor 131. The image processing unit 140 reads the reference image from the image sensor at high speed by the memory control circuit 141, guides it to the data bus 130 </ b> C side, and stores it in the frame memory in the digital signal processing unit 142.

  In the second and subsequent shootings, the CPU 130 causes the digital camera 100 in FIGS. 1 and 2 to generate an image with rolling shutter distortion by normal shooting in which the number of pixels is read from the image sensor 131. The image processing unit 140 instructs the internal memory control circuit 141 to guide the image signal composed of all the pixels converted into a digital signal by the ADC 133 onto the data bus 130C, and the digital signal processing circuit 142 in the image processing unit 140. And the distortion correction means of the digital signal processing circuit 142 performs distortion correction.

  FIG. 5 shows an output image after the distortion correction is performed by the digital signal processing circuit 142. As shown in the center and right side of FIG. 5, distortion correction is performed so as to match the reference image, and the same image as the reference image is generated. At this time, the background is not corrected together with the moving object according to the movement of the moving object as before, but according to the respective motion vectors of the moving object part and the background part detected between the distortion correction target image and the reference image. Since the correction is performed so as to match the reference image obtained by the first reference photographing, the background is prevented from being distorted.

  Note that the present invention is not limited to continuous shooting, and may be configured to perform shooting processing in the continuous shooting mode when the main subject is a moving object in still image shooting.

  As described above, the image capturing control apparatus and the image capturing method that do not cause distortion in the background even when the rolling distortion is corrected when the continuous image capturing by the rolling shutter imaging is performed, and the image capturing control apparatus are provided. Realized imaging device.

  Here, in the above embodiment, since high-speed reading from the image sensor is performed by thinning out the number of pixels from the image sensor 131, the distortion correction target when all pixels are read from the reference image and the image sensor 131 is read out. The image size is different.

  FIG. 6 is a diagram for explaining processing for adjusting the size of the reference image to the size of the distortion correction target image.

As shown in FIG. 6, when performing high-speed read by thinning out the number of pixels, the image size of the reference image and the distortion correction target image are different, the enlargement process in the resizing circuit in the digital signal processing unit 142 (not shown) After that, the processing for matching the distortion correction target image with the reference image may be performed to correct the rolling distortion.

  Here, the configuration of the first embodiment may be used. However, when performing reference imaging, it is desirable to simplify the process as much as possible and finish the process in a short time.

  For this purpose, the image size after the high-speed reading is distorted by performing the relative high-speed reading from the image sensor by the bit number obtained by relatively reducing the number of bits representing the pixel value per pixel. It is conceivable to increase the speed by simplifying the processing in the digital signal processing so as to be the same as the size of the correction target image.

  FIG. 7 shows a reference image obtained by reference photographing in which the CPU 130 causes the digital camera 100 to perform relative high-speed reading from the image sensor 131 by using the number of bits obtained by relatively reducing the number of bits representing a pixel value per pixel. It is a figure explaining the process at the time of setting it as the structure which produces | generates an image.

When the configuration of the second embodiment of FIG. 7, the image processing unit 140, the enlargement process of the same size as the image having a rolling strain by enlargement processing the reference image are omitted, the distortion in the digital signal processing unit 142 It is possible to further increase the speed by omitting the expansion process when performing the process of matching the correction target image with the reference image.

  In addition, even in rolling shutter shooting, the rolling distortion of the reference image can be reduced by performing high-speed reading. However, for the reference shooting, if a mechanical shutter is used to perform global shutter shooting, the reference image Rolling distortion disappears.

  FIG. 8 is a diagram corresponding to FIG. 5 and is a diagram for explaining rolling distortion correction processing in a case where a reference image is generated using a mechanical shutter at the time of reference photographing. FIG. 9 is a diagram corresponding to FIG. 6 and is a diagram for explaining rolling distortion correction processing in a case where a reference image is generated using a mechanical shutter during reference photographing. Further, FIG. 10 is a diagram corresponding to FIG. 7, and is a diagram for explaining rolling distortion correction processing in a case where a reference image is generated using a mechanical shutter during reference photographing.

  FIG. 8 to FIG. 10 show a rolling distortion correction process in a case where a reference image is generated using a mechanical shutter during reference photographing. Except for the use of the mechanical shutter, the same processing as in FIGS. 5 to 7 is performed.

  When the mechanical shutter is used in this way, the rolling distortion of the reference image is eliminated, and the rolling distortion due to the rolling shutter photographing is accurately corrected by the distortion correcting means in the digital signal processing circuit 142 provided in the image processing unit, and the background is the reference image. Reproduced faithfully.

  As described above, the shooting control apparatus and shooting method that do not cause distortion in the background even if the rolling distortion is corrected when continuous shooting is performed by rolling shutter shooting, and shooting that realizes the shooting method. An imaging apparatus including the program and the imaging control apparatus is realized.

  By the way, when the motion of the moving subject as the main subject is slow and the level of rolling distortion is less than the first reference value, it may not be necessary to correct the rolling distortion.

  In addition, when the moving object that is the main subject moves fast and the level of rolling distortion is equal to or higher than the second reference value that is larger than the first reference value, the correction cannot be completed even if correction is performed by the distortion correction means. The image may be broken.

  FIG. 11 is a diagram showing how much rolling distortion is caused according to the speed of the moving object.

  FIG. 11A shows a situation where the speed of the moving object is low, and FIG. 11C shows a situation where the speed of the moving object is high, and FIG. FIG. 11 shows a situation in the case of an intermediate speed between FIG. 11 (a) and FIG. 11 (c).

  In the present embodiment, in addition to the distortion correction means, a motion detection means for detecting the movement of the subject between two images obtained by adjacent photographing in the digital signal processing circuit 142 provided in the image processing section 140. And a distortion level determination means, and the distortion level determination means determines that the level of rolling distortion is between the first range and the second range according to the speed of the moving object detected by the moving object detection means. The distortion correction unit is configured to correct the rolling distortion of the distortion correction target image only in the case where the distortion correction is performed. As described above, the digital signal processing unit 142 included in the image processing unit 140 includes an example of the distortion correction unit according to the present invention.

  With this configuration, the correction processing of the distortion correction means in the digital signal processing circuit 142 is performed only when the distortion level determination means is in the second range, so that the first range and the third range are set. Processing at a certain time is omitted, and the configuration of the digital signal processing circuit 142 and, consequently, the configuration of the image processing unit 140 are simplified.

  FIG. 12 is a flowchart illustrating a procedure of processing relating to distortion level determination executed by the image processing unit 140. In the present embodiment, an example of the distortion level determination unit according to the present invention is configured by the determination processing in steps S1201 and S1202 during the processing of the flow executed by the image processing dedicated processor constituting the image processing unit.

  The program of FIG. 12 is stored in the main memory MEM.

  In step S1201, the image processing unit 140 determines whether the level of rolling distortion of the distortion correction target image with respect to the reference image is greater than or equal to the first reference value based on the result of motion detection by the motion detection unit. Here, when it is determined that the level of rolling distortion is in the first range that is less than the first reference value, the process proceeds to No and the process of this flow is terminated. When it is determined in step S1201 that the value is equal to or greater than the first reference value, the process proceeds to Yes, and in step S1202, the level of rolling distortion of the distortion correction target image with respect to the reference image is equal to or greater than the second reference value. It is determined whether or not. When it is determined in step S1202 that the rolling distortion level is in the second range that is less than the first reference value and less than the second reference value, the process proceeds to the No side. In step S1203, rolling distortion correction is performed, and the processing of this flow ends.

  When it determines with it being in the 3rd range beyond a 2nd reference value by step S1202, it progresses to Yes side and complete | finishes the process of this flow.

  With this configuration, the distortion correction process is performed by the distortion correction unit only when the distortion correction unit can perform the distortion correction based on the determination result of the distortion level determination unit in the image processing unit 140. Some steps of the correction process in the distortion correction unit are omitted, and the configuration of the digital processing circuit 142 and, consequently, the configuration of the image processing unit 140 are simplified. When simplification is achieved in this way, speeding up of processing is achieved.

  Here, in addition to the distortion correction unit, the digital signal processing unit 142 included in the image processing unit 140 includes a distortion correction availability determination unit that determines whether or not the image can be corrected for rolling distortion. A configuration in which the distortion correction unit corrects the rolling distortion of the distortion correction target image only when the distortion correction possibility determination unit determines that the distortion correction target image is an image capable of correcting the rolling distortion. However, the same effect can be obtained.

  FIG. 13 is a diagram illustrating a state where the motion detection unit in the digital signal processing circuit 142 included in the image processing unit 140 cannot perform motion detection.

  As shown in FIG. 13, when the image when all pixels are read out from the reference image protrudes from the shooting angle of view, the motion cannot be detected by the motion detection means. In this way, when the image protrudes from the angle of view, there is a high possibility that the image will fail without being corrected even if the distortion correction of the rolling distortion is performed.

  Therefore, it is conceivable that the image processing unit 140 does not cause the distortion correction unit to perform distortion correction when the movement detection unit cannot detect movement. In this case, the image is prevented from failing when the rolling distortion is forcibly corrected.

  FIG. 14 is a flowchart showing the procedure of the distortion correction process executed by the image processing unit 140 provided with an example of the distortion correction availability determination unit according to the present invention. In the present embodiment, an example of the distortion correction availability determination unit according to the present invention is configured by the processing in step S1401 of the processing in the flow executed by the image processing unit 140 configured by a processor dedicated to image processing.

  In step S1401, the image processing unit 140 determines whether the distortion correction target image is an image that can correct rolling distortion based on the detection result of the motion detection unit. If it is determined in step S1402 that the image is out of the angle of view and cannot be corrected as shown in FIG. 13B, the process of this flow is terminated without correction.

  If it is determined in step S1401 that correction is possible, the process proceeds to Yes, and rolling distortion is corrected in step S1403, and the process of this flow ends.

  When the image processing unit executes the above flow, an image having a rolling distortion is recorded as it is without correction for an image that causes a failure of the image when the rolling distortion correction is performed according to the detection result of the motion detection unit. .

  In this way, an image that can be seen to some extent is acquired even if the image is broken or has rolling distortion.

  Here, when the CPU 130 instructs the image sensor driving unit 1303 to perform high-speed reading from the image sensor 131, the number of pixels is thinned out, the number of bits expressing one pixel is reduced, and only the moving object part that is the main subject is displayed. That is, high-speed reading can be performed even if the central portion of the image is cut out and read out.

  FIG. 15 is a diagram for explaining a state in the case of performing so-called trimming that cuts out the center portion of the angle of view.

  As shown in FIG. 15, when the center portion of the image is cut out, the number of pixels to be read is reduced and high-speed reading can be performed.

  FIG. 16 is a diagram similar to FIGS. 5 to 7 and FIGS. 8 to 10 and is a diagram for explaining the continuous shooting process performed by the digital camera of FIGS. 1 and 2 during continuous shooting.

  As shown in FIG. 16, the reference image may be generated by cutting out the center of the angle of view.

  With the configuration shown in FIG. 16, for example, it is possible to correct rolling distortion as long as the moving object that is the main subject protrudes from the central frame and remains within the angle of view.

  Therefore, when the above processing is performed inside the digital camera, it is preferable to display a frame (dotted line) on the LCD panel so that the user can take a picture so that the main subject is within the frame.

  As a result, since it is possible to quickly process whether or not the moving object protrudes, it is possible to more reliably prevent the image from failing.

  By the way, when continuous shooting is performed when the subject is a moving object, the moving object always moves away from the angle of view as the number of shots increases. In other words, if the number of images is limited to a predetermined number or less, the risk that the moving object is out of the angle of view and the image is broken is avoided.

  FIG. 17 is a diagram illustrating a situation when continuous shooting is performed.

  As shown in FIG. 17, up to the second image is within the angle of view, and shooting is preferably performed. However, the third image is taken as if the shooting failed because the image was out of the angle of view. End up. If continuous shooting is performed and the third image is in the continuously shot image file, some photographers have the impression that shooting has failed.

  Therefore, even if a predetermined number of images are taken, the number of images taken is set to two or three, and an image that has failed to be shot can be deleted later. Alternatively, it may be configured such that shooting is terminated when the motion detector cannot detect a moving object based on a reference image.

  FIG. 18 is a diagram illustrating a configuration when the number of shots is set to a predetermined number.

  In the configuration of FIG. 17, since continuous shooting is performed, an image as if shooting failed in the third shooting is obtained, but the unit of continuous shooting is set to two as shown in FIG. If the configuration is such that the standard shooting is performed for every two images and the image file is created for each of the two images, the photographer fails to shoot after shooting (the third image in FIG. 17). You can do that. In addition, the continuous shooting can be stopped when the reference image protrudes using the motion detection means.

  Further, the number of units as the unit may be made variable according to the movement of the moving object.

  FIG. 19 is a diagram for explaining the situation when the number of shots is made variable according to the movement of the movement detecting means.

  FIG. 19A shows the situation when the amount of movement is small, FIG. 19C shows the situation when the amount of movement is large, and FIG. 19B shows the situation. A situation between the situation of FIG. 19A and the situation of FIG. 19C is shown.

  As shown in FIG. 19, the number of continuous shots is controlled according to the speed of the moving object that is the main subject, and an image as if shooting failed is not shot.

  Finally, an example in which the imaging control device of the present invention is applied to a monitoring camera on a production line equipped with a conveyor will be described.

  FIG. 20 is a diagram illustrating a configuration of a monitoring camera system provided in the production line.

  As shown in FIG. 20, a program for operating a personal computer 30 (hereinafter referred to as a PC) 30 as a photographing control device (a program for executing the contents described in the drawings and drawings) is installed in the PC, An example will be described in which the present invention is applied to a system in which an imaging unit 31 is provided to monitor whether a product is flowing on a conveyor without any problem.

  It is assumed that an image processing board including an image processing dedicated processor is inserted in the slot of the PC 30, and further, an image sensor driving unit that drives the imaging unit 31 is provided on the image processing board.

  As shown in FIG. 20, the PC 30 has a CPU that constitutes the photographing control means according to the present invention, and an image processing board that constitutes an example of the distortion correction means according to the present invention is inserted into the slot of the PC 30. Yes.

  For this reason, when the product 300 is flowing on the conveyor 3, it is possible to satisfactorily monitor whether the production line is operating normally by performing continuous shooting.

  As described above, the shooting control apparatus and shooting method that do not cause distortion in the background even if the rolling distortion is corrected when continuous shooting is performed by rolling shutter shooting, and shooting that realizes the shooting method. An imaging apparatus including the program and the imaging control apparatus is realized.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is an external perspective view of a digital camera that is a first embodiment of a photographing apparatus according to the present invention when viewed obliquely from the front top. It is an external view which shows the back surface of the digital camera shown in FIG. FIG. 3 is a block diagram illustrating a circuit configuration of the digital camera 100 illustrated in FIGS. 1 and 2. It is a figure which shows the first state of the to-be-photographed object in which continuous imaging | photography is performed. It is a figure explaining the process of the continuous imaging performed with the digital camera of FIG. 1, FIG. 2 at the time of continuous imaging. It is a figure explaining the process which matches the size of a reference | standard image with the size of the distortion correction object image. The CPU 130 causes the digital camera 100 to generate a reference image by reference photographing that causes the image sensor 131 to perform relatively high-speed reading with the number of bits obtained by relatively reducing the number of bits representing the pixel value per pixel. It is a figure explaining the process at the time of having comprised. FIG. 6 is a diagram corresponding to FIG. 5, illustrating a rolling distortion correction process when a reference image is generated using a mechanical shutter during reference shooting. FIG. 7 is a diagram corresponding to FIG. 6, illustrating a rolling distortion correction process when a reference image is generated using a mechanical shutter during reference shooting. FIG. 8 is a diagram corresponding to FIG. 7, illustrating a rolling distortion correction process when a reference image is generated using a mechanical shutter during reference shooting. It is a figure which shows how much rolling distortion becomes according to the movement amount of a moving body. It is a flowchart which shows the procedure of the process regarding the distortion level determination which the motion detection means in the digital signal processing circuit 142 with which the image process part 140 is provided performs. It is a figure explaining the state when a motion cannot be detected with the motion detection means in the digital signal processing circuit 142 with which the image process part 140 is provided. It is a flowchart which shows the procedure of the distortion correction process which the image process part 140 performs based on the detection result of the motion detection part which comprises an example of the distortion correction | amendment propriety determination means said to this invention. It is a figure explaining the state in the case of performing what is called trimming which cuts out the center part of an angle of view. FIG. 11 is a diagram similar to FIGS. 5 to 7 and FIGS. 8 to 10, and is a diagram illustrating a process of continuous shooting performed by the digital camera of FIGS. 1 and 2 during continuous shooting. It is a figure explaining the condition when continuous imaging is performed. It is a figure explaining the structure at the time of setting the unit of imaging | photography number to a predetermined number. It is a figure explaining the situation at the time of changing the number of imaging | photography according to the motion of a motion detection means. It is a figure which shows the structure of the surveillance camera system arrange | positioned at the manufacturing line.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Digital camera 1110 Shooting optical system 1111 Shooting lens 1112 Aperture 111 Zoom lens barrel 112 Flash emission window 113 AF auxiliary light lamp 114 Power switch 115 Release button 116 Mode dial 1170 Display control circuit 117 LCD
117A LCD panel 118 MENU / OK button 119 DISP / BACK button 120 Photo mode button 121 Playback button 122 Cross button 123 Zoom button 124 Viewfinder 130 CPU
130B Address bus 130C Data bus 1301 Lens drive unit 1302 Aperture drive unit 1303 Image sensor drive unit 1304 Subject determination circuit 140 Image processing unit 141 Memory control circuit 142 Digital signal processing circuit 143 Compression / decompression circuit 144 Integration circuit 131 Image sensor 132 Analog signal processing Circuit 133 ADC
150 External media control circuit

Claims (10)

Imaging that consists of a plurality of pixel lines, and that can perform rolling shutter photography in which the electronic shutter is opened at different timings for each pixel line and image signals are read at different timings for each pixel line after a predetermined exposure time has elapsed. An imaging control device that controls imaging in an imaging device including an element,
The photographing apparatus has a continuous photographing mode for continuously photographing a plurality of sheets,
When the imaging device is in front Symbol continuous shooting mode, to the imaging apparatus, along with causing performed a plurality of rolling shutter photographing consecutive, for at least one of the plurality number photographed from the image sensor due to the relative high speed reading of relatively rolling strain rather small, and a photographing control means for relatively image size makes perform standard photography to produce a reference image has smaller,
The image size of the reference image is set so that the image size of the reference image is the same as the image size of the distortion correction target image obtained by shooting excluding the reference shooting among the plurality of rolling shutter shootings. Means for enlarging the image size,
The image size enlargement unit reference image picture size is enlarged by, between the front Kiibitsu correction target image, detects each pixel or the motion vector for each small region, based on the motion vector, the An imaging control apparatus comprising: distortion correction means for correcting rolling distortion of the distortion correction target image so as to match the reference image .   The photographing control unit is configured to generate the reference image by reference photographing that causes the photographing apparatus to perform relatively high-speed reading by thinning out a relatively small number of pixels from the imaging device. The imaging control apparatus according to claim 1. A first range in which a level of rolling distortion of the distortion correction target image with respect to the reference image is less than a first reference value, the first reference value, and a second reference value greater than the first reference value; Distortion level determination means for determining which range is a second range in the middle of the second range or a third range that is equal to or greater than the second reference value ,
The distortion correcting unit according to claim 1, wherein the one in which it is in said second range by the distortion level determining means for correcting the rolling distortion of the distortion correction image only when it is determined or 2. The photographing control device according to 2 . The distortion correction means comprises distortion correction availability determination means for determining whether or not the image is capable of correcting rolling distortion;
The distortion correcting unit only if said distortion correction target image is an image that can correct the rolling strain is determined by the distortion correction Allowed determination means, rolling distortion of the distortion correction image imaging control apparatus according to claim 1 or 2, wherein the corrects the. Said imaging control means, at least the first one shot control according to any one of the preceding claims, characterized in 4 that is intended to perform the standard photography for one of the plurality of continuous shooting apparatus. The photographing control unit, imaging control apparatus according to any one of claims 1 to 5, characterized in that to let out the said reference shot by shooting a predetermined number. A motion detecting means for detecting a motion of a subject between two images obtained by photographing at adjacent times;
2. The photographing control unit, which causes the photographing apparatus to perform the reference photographing for every number of photographings according to the speed of movement of a subject detected by the motion detecting unit. 5. The photographing control device according to any one of items 1 to 4 . Imaging that consists of a plurality of pixel lines, and that can perform rolling shutter photography in which the electronic shutter is opened at different timings for each pixel line and image signals are read at different timings for each pixel line after a predetermined exposure time has elapsed. An imaging control method for controlling imaging in an imaging apparatus including an element,
The photographing apparatus has a continuous photographing mode for continuously photographing a plurality of sheets,
When the photographing apparatus is in the continuous photographing mode, the photographing apparatus performs a plurality of continuous rolling shutter photographing, and at least one of the plurality of photographings is taken from the image sensor. due to the relative high speed reading, a photographing control step of causing performed relatively rolling strain rather small, and the reference imaging relatively image size to generate the reference image has smaller,
The image size of the reference image is set so that the image size of the reference image is the same as the image size of the distortion correction target image obtained by shooting excluding the reference shooting among the plurality of rolling shutter shootings. An image size enlargement step to enlarge,
The image size enlargement steps by reference image the image size is enlarged, between the front Kiibitsu correction target image, detects each pixel or the motion vector for each small region, based on the motion vector, the And a distortion correction step of correcting a rolling distortion of the distortion correction target image so as to match the reference image . The information processing apparatus is executed by an information processing apparatus that executes a program. The information processing apparatus includes a plurality of pixel lines, and the electronic shutter is opened at different timings sequentially for each of the plurality of pixel lines. A shooting control program that operates as a shooting control device that controls shooting in a shooting device that includes a rolling shutter shooting that reads out image signals at different timings sequentially for each pixel line after a lapse,
The photographing apparatus has a continuous photographing mode for continuously photographing a plurality of images,
The information processing apparatus;
When the photographing apparatus is in the continuous photographing mode, the photographing apparatus performs a plurality of continuous rolling shutter photographing, and at least one of the plurality of photographings is taken from the image sensor. by relative high-speed reading, relatively rolling strain rather small, and a photographing control means for relatively image size makes perform standard photography to produce a reference image has smaller,
The image size of the reference image is set so that the image size of the reference image is the same as the image size of the distortion correction target image obtained by shooting excluding the reference shooting among the plurality of rolling shutter shootings. Means for enlarging the image size,
The image size enlargement unit reference image picture size is enlarged by, between the front Kiibitsu correction target image, detects each pixel or the motion vector for each small region, based on the motion vector, the A shooting control program that is operated as a shooting control device including a distortion correction unit that corrects rolling distortion of an image to be corrected so as to match a reference image . In a photographing device that forms an image of a subject on an image sensor with a photographing optical system and generates an image signal representing the subject with the image sensor,
The image sensor is composed of a plurality of pixel lines, and the plurality of pixel lines are opened at different timings for each pixel line at different timings, and after a predetermined exposure time has elapsed, at different timings for each pixel line. Rolling shutter photography that reads out image signals is possible,
Furthermore, the imaging device
It has a continuous shooting mode for continuously shooting multiple images,
Upon receiving a shooting instruction when in the continuous shooting mode, the image sensor performs a plurality of continuous rolling shutter shooting, and at least one of the plurality of images is taken from the image sensor. due to the relative high speed reading of relatively rolling strain rather small, and a photographing control means for relatively image size makes perform standard photography to produce a reference image has smaller,
The image size of the reference image is set so that the image size of the reference image is the same as the image size of the distortion correction target image obtained by shooting excluding the reference shooting among the plurality of rolling shutter shootings. Means for enlarging the image size,
The image size enlargement unit reference image picture size is enlarged by, between the front Kiibitsu correction target image, detects each pixel or the motion vector for each small region, based on the motion vector, the An imaging apparatus comprising: distortion correction means for correcting rolling distortion of the distortion correction target image so as to match a reference image .

Images