JP2006058945A - Method and device for correcting rolling shutter image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and device for correcting a rolling shutter image for accurately correcting the distortion of a photographic image to be generated according to the conveyance of a product at the time of photographing a product under line conveyance by using a CMOS image sensor with a rolling shutter. <P>SOLUTION: The correction image of a photographic image generated by successively changing coefficients B and D of X1=X+BY and Y1=DY of a coordinate conversion system (1) is compared with a preliminarily registered reference image, and the difference of those images is decided, and coefficients B and D when the difference of the correction image and the registered reference image is the minimum value are detected so that even when the moving speed of the product is not measured, the distortion of a rolling shutter image to be generated according to the conveyance of the product to an X direction or Y direction can be accurately corrected so as to be restored to the original image. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

      The present invention relates to a method and apparatus for correcting a rolling shutter image obtained by photographing a product in line conveyance with a CMOS image sensor with a rolling shutter.

  When shooting products that are transported one after another on the production line and comparing the captured images with registered reference images to determine whether the shape of the product or the characters and figures that appear on the surface of the product is good or bad When the product is imaged by a CMOS image sensor (imaging device) with a rolling shutter (ERS), for example, the shutters are sequentially opened from the top to the bottom (in the Y direction) as shown in FIG. The product is usually conveyed in one of the X direction and the Y direction by a conveyor or the like. For this reason, when the product is photographed in a state where the product is conveyed in the X direction or the Y direction, as shown in FIG. 4B, direction 1 (from top to bottom), direction 2 (from bottom to top), direction 3 (left) When the product moves in the direction 4 (from right to left) and in the direction 4 (from right to left), as shown in FIG. For example, in the case of the direction 3, since the lower right corner 30a of the product 30 has been moved to the right when the (n-1) th row is photographed, it is distorted into a rhombus as shown in FIG.

  If image distortion occurs, correct determination cannot be made when determining whether the product is good or bad (OK / NG). For this reason, conventionally, a CCD image sensor whose shutter is opened simultaneously on all screens (all pixels) or a CMOS image sensor having a global shutter function that exposes all pixels at the same timing has been used.

  On the other hand, in the case of a hand-held image scanner, images are sequentially captured by moving the scanner manually (for example, Patent Document 1). However, when a CMOS image sensor with a rolling shutter is used for such a purpose, the image is similarly deformed. End up.

  However, the CCD image sensor has a drawback that the element itself is expensive, requires a plurality of power supplies, and consumes a large amount of power. In addition, a CMOS image sensor with a global shutter function is equipped with a global shutter function on the chip, so that the area of the light receiving element is reduced, resulting in a screen with a lot of sensitivity and noise, and the chip area is increased. Get higher.

  The present invention solves the above-described problems, and can accurately correct a distortion of a photographed image caused by product transportation when photographing a product during line transportation using a CMOS image sensor with a rolling shutter. It is an object of the present invention to provide a shutter image correction method and apparatus.

In order to achieve the above object, a rolling shutter image correction method and apparatus according to one configuration of the present invention sequentially exposes a product being conveyed in a Y (vertical) direction using a CMOS image sensor with a rolling shutter. When the image is taken, the image distortion caused by the conveyance of the product in the X (horizontal) direction or the Y direction is corrected by coordinate conversion. When the image is corrected by converting the read coordinates X or Y into coordinates X1 or Y1 based on the coefficients B and D,
X1 = X + BY, Y1 = DY (1)
The correction image of the photographed image generated by sequentially changing the coefficients B and D is compared with the reference image registered in advance to determine the difference between the two images, and the difference between the correction image and the registered reference image is minimized. The coefficients B and D when they are values are detected.

  According to this configuration, the correction image of the captured image generated by sequentially changing the coefficients B and D of the coordinate conversion formula (1) is compared with the reference image registered in advance, and the difference between the two images is determined. Since the coefficients B and D when the difference between the corrected image and the registered reference image is the minimum value are detected, the rolling shutter that occurs when the product is transported in the X or Y direction without measuring the moving speed. It is possible to accurately correct the image distortion so as to restore the original image.

A rolling shutter image correction method and apparatus according to another configuration of the present invention, when shooting a product in line conveyance by sequentially exposing in the Y (vertical) direction using a CMOS image sensor with a rolling shutter, The image distortion caused by the conveyance of the product in the X (horizontal) direction or the Y direction is corrected by coordinate transformation, and the coordinates X or When correcting the image by converting Y into coordinates X1 or Y1 based on the coefficients B and D,
When the transfer time for one row of the CMOS image sensor is T and the time for a product corresponding to the product conveyance speed to pass one pixel is P, the coefficients B and D are calculated by the following equation (2). is there.
B = T / P, D = T / (TP) (2)

  According to this configuration, when the transfer time for one row of the CMOS image sensor is T and the time for the product corresponding to the moving speed of the product to pass one pixel is P, the coordinate conversion formula ( Since the coefficients B and D of 1) are calculated, the distortion of the rolling shutter image caused by the conveyance of the product in the X direction or the Y direction can be accurately corrected according to the moving speed of the product so as to return to the original image. .

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a rolling shutter image correction apparatus according to an embodiment of the present invention. This device, for example, captures products that are transported one after another on a production line, compares the captured images with registered reference images, and determines whether the product shape or the characters and figures that appear on the surface of the product are good / bad. When determining (OK / NG), using a CMOS image sensor with a rolling shutter, for example, in the Y (vertical) direction, for example, exposure is performed sequentially from top to bottom in the X (horizontal) direction or Y Correction is performed to return a captured image that is distorted in accordance with direction conveyance to an original shape image.

  1 includes a CMOS image sensor 1 with a rolling shutter, an image memory 2, a registered image memory 3, a read address calculation means 4, a coordinate counter 5, a moving direction set value storage memory 6, a coefficient counter 7, and an image superimposing means. 8. A difference determination unit 9 and a difference minimum value detection unit 10 are provided.

  In the CMOS image sensor 1 with a rolling shutter, the shutter is sequentially opened from the top to the bottom in the Y (vertical) direction to expose and image the product. The image memory 2 stores an image obtained by photographing the product being conveyed. The registered image memory 3 stores a registered reference image obtained by photographing a reference product in advance according to a registration instruction.

The read address calculation means 4 uses the following coordinate conversion formula (1) to calculate the coordinates based on the coefficients B and D for each coordinate (X, Y) that is the read address of the captured image counted by the coordinate counter 5. A calculation is performed to correct the captured image stored in the image memory 2 by converting the coordinates to X1 or Y1. When the product does not move in the X direction and moves only in the Y direction, the coefficient B = 0, and when the product does not move in the Y direction and moves only in the X direction, the coefficient D = 1.
X1 = X + BY, Y1 = DY (1)

FIG. 2 shows the coordinates (X, Y) of the read address of the captured image. As shown in the figure, it is represented by the coordinates (X _m−1 , Y _n−1 ) of the read address in the ( _{m −1} ) _th column in the X direction and the ( _n−1 ) _th row in the Y direction. Through the above coordinate conversion calculation, a corrected image of the captured image generated by sequentially changing the coefficients B and D is read from the image memory 2.

  The moving direction set value storage memory 6 stores setting values for the moving direction and moving speed in the X direction or Y direction, and the coefficients B and D that are sequentially changed corresponding to the set values are read by the coefficient counter 7. This is given to the address calculation means 4.

  The image superimposing means 8 superimposes both images in order to compare the corrected image generated by sequentially changing the coefficients B and D with the reference image registered in advance. Then, the difference determination unit 9 determines the difference between the two images. The read address calculation means 4, the coordinate counter 5, the moving direction set value storage memory 6, the coefficient counter 7, the image superposition means 8 and the difference determination means 9 constitute a determination means 20.

  Then, the minimum value of the difference between the corrected image of the captured image generated by sequentially changing the coefficients B and D by the coefficient counter 5 and the registered reference image is detected by the difference minimum value detecting means (coefficient detecting means) 10, Coefficients B and D at this time are detected.

The operation of the rolling shutter image correcting apparatus having the above-described configuration will be described below with reference to FIG. In this example, it is assumed that the product does not move in the X direction, but only moves from the top to the bottom in the Y direction (direction 1 in FIG. 4).
(1) Comparison between the corrected image and the registered reference image First, a product is photographed by a CMOS image sensor with a rolling shutter, and an image obtained by photographing the product 30 being conveyed into the image memory 2 as shown in FIG. Remembered. As shown in FIG. 3A, the registered image memory 3 stores a registered reference image obtained by photographing the product 30 as a reference in advance.

  Since the product 30 moves from top to bottom in the Y direction, Y1 = DY in the coordinate conversion formula (1) above, and based on the coefficient D obtained by sequentially changing the coordinates Y read out of the captured image, A corrected image of the captured image is generated by converting the coordinate to the coordinate Y1, and is read from the image memory 2. In this case, the coefficient D is 1 or more. In this example, since the product 30 does not move in the X direction, in the coordinate conversion formula (1), X1 = X + BY has a coefficient B of 0 and X1 = X.

  3C is a corrected image read at D = 1, FIG. 3D is a corrected image read at D = 1.1, and FIG. 3E is read at D = 1.2. FIG. 3F shows a corrected image read out with D = 1.3. Each of the corrected images in FIGS. 3C to 3F and the registration reference image in FIG. 3A are overlaid and compared, and the difference between the two images is determined.

(2) Coefficient detection As a result of determining the difference between each corrected image and the registered reference image and obtaining each difference, the difference between the registered reference image and the corrected image in FIG. Therefore, the coefficient D = 1.2 at this time is detected.

  Similarly, when the product moves from bottom to top in the Y direction, the coefficient D is detected when the coefficient B = 0 and the coefficient D = 1 or less.

  When the product moves from left to right in the X direction, the coefficient B = 0 or more and the coefficient D = 1, and when the product moves from right to left in the X direction, the coefficient B = 0 or less (negative value) ), The coefficient D = 1, and the coefficient B is detected.

  As described above, in the present invention, the correction image of the captured image generated by sequentially changing the coefficients B and D of the coordinate conversion equation (1) is compared with the pre-registered image, and the difference between the two images is determined. Judgment and detection of the coefficients B and D when the difference between the corrected image and the registered image is the minimum value, so that the rolling caused by the conveyance of the product in the X direction or the Y direction can be performed without measuring the moving speed of the product. The distortion of the shutter image can be accurately corrected so as to return to the original image.

In the above-described embodiment, the coefficients B and D are detected. However, when the moving speed of the product is clearly known, the coefficients B and D are calculated based on the moving speed of the product as follows. You may calculate. That is, when the coefficient calculation means is T for the transfer time of one row of the CMOS image sensor and P is the time for the product corresponding to the moving speed of the product to pass one pixel, the coefficient B, D is calculated.
B = T / P, D = T / (TP) (2)
Using the calculated coefficients B and D, an accurate corrected image of the captured image can be generated by the coordinate conversion equation (1).

  Thus, when the transfer time for one row of the CMOS image sensor is T and the time for the product corresponding to the moving speed of the product to pass through one pixel is P, the coordinate conversion equation (1) is obtained by the above equation (2). Therefore, the distortion of the rolling shutter image caused by the conveyance of the product in the X direction or the Y direction can be accurately corrected so as to return to the original image according to the moving speed of the product.

  In the present invention, a CMOS image sensor with a rolling shutter is used for sequential exposure in the Y (vertical) direction. However, exposure may be performed for sequential exposure in the X (horizontal) direction.

It is a block diagram which shows the correction apparatus of the rolling shutter image which concerns on one Embodiment of this invention. It is a figure which shows a coordinate (X, Y). It is a figure which shows the operation | movement which detects the coefficient of a coordinate transformation type | formula. It is a figure which shows the distortion of the picked-up image accompanying the movement direction of a product.

Explanation of symbols

1: CMOS image sensor with rolling shutter 3: Registered image memory 10: Coefficient detection means 20: Determination means

Claims (4)

When a product in line is photographed by sequentially exposing the product in the Y (vertical) direction using a CMOS image sensor with a rolling shutter, the image generated when the product is conveyed in the X (horizontal) or Y direction. A method of correcting distortion by coordinate transformation,
When correcting the image by converting the coordinate X or Y read out of the captured image into the coordinate X1 or Y1 based on the coefficients B and D by the following coordinate conversion formula (1),
X1 = X + BY, Y1 = DY (1)
The correction image of the photographed image generated by sequentially changing the coefficients B and D is compared with the reference image registered in advance to determine the difference between the two images, and the difference between the correction image and the registered reference image is minimized. A method for correcting a rolling shutter image, wherein the coefficients B and D when the value is a value are detected. When a product in line is photographed by sequentially exposing the product in the Y (vertical) direction using a CMOS image sensor with a rolling shutter, the image generated when the product is conveyed in the X (horizontal) or Y direction. A method of correcting distortion by coordinate transformation,
When correcting the image by converting the coordinate X or Y read out of the captured image into the coordinate X1 or Y1 based on the coefficients B and D by the following coordinate conversion formula (1),
X1 = X + BY, Y1 = DY (1)
When the transfer time for one row of the CMOS image sensor is T and the time for the product corresponding to the moving speed of the product to pass one pixel is P, the coefficients B and D are calculated by the following equation (2). A method for correcting a shutter image.
B = T / P, D = T / (TP) (2) When a product in line conveyance is imaged by sequentially exposing the product in the Y (vertical) direction using a CMOS image sensor with a rolling shutter, the image generated when the product is conveyed in the X (horizontal) direction or the Y direction. An apparatus for correcting distortion by coordinate transformation,
When correcting the image by converting the coordinate X or Y read out of the captured image into the coordinate X1 or Y1 based on the coefficients B and D by the following coordinate conversion formula (1),
X1 = X + BY, Y1 = DY (1)
A determination unit that compares a corrected image of a captured image generated by sequentially changing the coefficients B and D of the coordinate conversion formula and a reference image registered in advance, and determines a difference between the two images;
A rolling shutter image correction apparatus, comprising: coefficient detection means for detecting the coefficients B and D when the difference between the correction image and the registered reference image is a minimum value. When a product in line conveyance is imaged by sequentially exposing the product in the Y (vertical) direction using a CMOS image sensor with a rolling shutter, the image generated when the product is conveyed in the X (horizontal) direction or the Y direction. An apparatus for correcting distortion by coordinate transformation,
When correcting the image by converting the coordinate X or Y read out of the captured image into the coordinate X1 or Y1 based on the coefficients B and D by the following coordinate conversion formula (1),
X1 = X + BY, Y1 = DY (1)
Coefficient calculation for calculating coefficients B and D according to the following equation (2), where T is the transfer time for one row of the CMOS image sensor and P is the time for the product corresponding to the moving speed of the product to pass one pixel. A rolling shutter image correction device comprising means.
B = T / P, D = T / (TP) (2)

Images