JP4222238B2 - Imaging apparatus and imaging control method - Google Patents

Description

  The present invention relates to a photographing apparatus such as a digital camera and a photographing control method.

In recent years, with the advancement of electronic technology, not only mere photography but also digital cameras capable of various editing with respect to taken images have become widely used. The electronic camera disclosed in Patent Document 1 includes an image processing circuit and can generate a front image obtained by photographing a document from the front from the photographed image even when a document or the like is photographed from an oblique direction.
JP 2002-354331 A

  In order to generate a front image from a captured image, a great deal of calculation is required, and the processing time becomes longer as the number of pixels increases to 4 million or 5 million pixels. In order to shorten the processing time, a method such as using a faster CPU must be taken.

  However, the power of the CPU mounted in the digital camera is naturally limited. If a front image is generated and saved at the time of shooting, it takes time until the saving is completed because the CPU is slow, and quick shooting cannot be performed. That is, convenience and operability as a camera are impaired. In addition, the CPU power is occupied during that time and cannot be used for other processing, and there is a problem that the operability of the user is deteriorated such that the response of the key operation is extremely deteriorated.

  On the other hand, a digital camera has a low-resolution display device (LCD screen) for browsing. When displaying an image on a display device (LCD screen) for viewing, it is conceivable to make a corrected image with a low resolution for correction on the front image and display it on the LCD screen. However, since a front image is created and displayed every time, there is a problem that improvement in response cannot be expected so much for the operation of a digital camera in which many images are browsed sequentially.

  The present invention has been made in view of the current situation as described above, and an object thereof is to provide a photographing apparatus capable of generating a front image without deteriorating the operability, convenience, and response characteristics of a digital camera. To do.

In order to achieve the above object, a photographing apparatus according to a first aspect of the present invention includes mode switching means for switching between a photographing mode and a reproduction mode,
A shooting unit that captures a shooting target and acquires a shot image when the shooting mode is set;
Storage means for storing the captured image;
Contour forming means for obtaining a contour of the photographing object in the photographed image;
Projection parameter obtaining means for obtaining a projection parameter indicating a relationship between the photographed image and the actual photographing object based on the obtained contour;
A front image forming unit that forms a front image of the photographed image with respect to the photographed image using the projection parameter;
Display means for displaying the photographed image and the front image formed for the photographed image;
A control means for controlling the contour forming means, parameter acquiring means, front image forming means and display means;
The control means includes
Information related to the contour obtained by the contour forming unit and a reduced image of the captured image are stored in the storage unit in association with the captured image,
When the reproduction mode is set, a reduced front image corresponding to a reduced image of the photographed image is created based on the information related to the contour stored in the storage unit and displayed by the display unit, and thereafter The front image corresponding to the photographed image is created and displayed by the display means .

By adopting such a configuration, a front image is not formed in the shooting mode, and for example, continuous shooting is possible, and convenience is not impaired. In the playback mode, first, the reduced front image is displayed, and when the reduced front image is viewed, if the photographed image is not desired to be corrected to the front image, the previous / next image is quickly advanced. Therefore, a front image can be generated and confirmed without deteriorating operability, convenience, and response characteristics.

Incidentally, when the reproduction mode is set, when displaying the front image on the display means reads the information on the contour stored in the storage means, to the contour by the projection parameter acquiring unit The projection parameter may be obtained based on the information, the front image may be formed by the front image forming unit, and the front image may be displayed by the display unit.

Further, the control means creates the reduced front image and displays it on the display means . When instructed , the control means creates the front image corresponding to the captured image and displays it on the display means. Good.

  In addition, the image processing apparatus includes an editing instruction input unit that inputs an instruction to edit the photographed image, the reduced image, the reduced front image, or the front image, and the control unit responds to the content of the input pre-editing instruction. An image to be edited may be selected from the photographed image, the reduced image, the reduced front image, or the front image, displayed on the display means, and the editing may be performed using data corresponding to the image.

In order to solve the above-described problem, a shooting control method according to a second aspect of the present invention includes a mode switching unit that switches between a shooting mode and a playback mode,
A shooting unit that captures a shooting target and acquires a shot image when the shooting mode is set;
Storage means for storing the captured image;
Contour forming means for obtaining a contour of the photographing object in the photographed image;
Projection parameter obtaining means for obtaining a projection parameter indicating a relationship between the photographed image and the actual photographing object based on the obtained contour;
A front image forming means for forming a front image when the photographing object is photographed using the projection parameter;
Display means for displaying the front image formed for the previous photographed image;
In the camera device comprising the contour forming means, the parameter acquiring means, the front image forming means, and the control means for controlling the display means,
Information related to the contour obtained by the contour forming unit and a reduced image of the captured image are stored in the storage unit in association with the captured image,
When the reproduction mode is set, a reduced front image corresponding to a reduced image of the photographed image is created based on the information related to the contour stored in the storage unit and displayed by the display unit, and thereafter A process of creating the front image corresponding to the photographed image and displaying it by the display means ;
It is characterized by implementing.

When the reproduction mode is set, when the front image is displayed again on the display unit, the information on the contour stored in the storage unit is read out, and the projection parameter acquisition unit performs a process on the contour. The projection parameter may be obtained based on the information, the front image may be formed by the front image forming unit, and the front image may be displayed by the display unit.

Further, when there is an instruction as a result of creating the reduced front image and displaying it by the display means, the front image corresponding to the photographed image may be created and displayed by the display means.
The photographing apparatus includes an editing instruction input means for inputting an instruction to edit the photographed image, the reduced image, the reduced front image, or the front image,
An image to be edited is selected from the photographed image, the reduced image, the reduced front image, or the front image according to the content of the input pre-edit instruction, and is displayed on the display unit, and corresponds to the image. The editing may be performed using data.

  According to the present invention, in the shooting mode, the front image is not formed, and the front image is formed in the playback mode. Therefore, the convenience in shooting does not deteriorate, and for example, continuous shooting is possible. It is.

(Embodiment 1)
FIG. 1 is a configuration diagram of a digital camera according to an embodiment of the present invention.

  The digital camera 100 includes a CPU and the like. The digital camera 100 includes a control unit 101 that controls the entire system, an optical lens 102 that includes a lens that collects an image, a CCD, and the like. Digitized and captured image sensor 103, storage unit 104, display unit 105 such as an LCD screen that displays captured images and the like, and image processing unit 106 that performs image processing in combination with control unit 101. And an input unit 107 in which various keys are arranged.

  The storage unit 104 stores an image memory for storing an image from the image sensor 103, a ROM (Read Only Memory) for storing a program to be executed by the control unit 101, and when the control unit 101 performs an operation or the like. A RAM (Random Access Memory) for storing data, a non-volatile memory for storing processed images, and the like are included.

The input unit 107 includes, for example, a shutter key, a mode setting key for setting a mode, and a key for selecting and executing various editing processing functions.
Next, the operation of this digital camera will be described with reference to a plurality of drawings.

FIG. 2 is a schematic diagram illustrating a shooting state of the digital camera.
FIG. 3 is a diagram illustrating an example of a captured image.

  When the user performs shooting, the mode setting key of the input unit 107 is operated to set the shooting mode, and the shutter key is pressed with the optical lens 102 facing the shooting target T as shown in FIG. When the shutter key is pressed, an image including a subject to be photographed is condensed on the image sensor 103 via the optical lens 102.

  The image sensor 103 photoelectrically converts the collected image, and generates a captured image M1 as shown in FIG. 3 composed of image data for each pixel. When the shooting target T is, for example, a document, if the shooting target is shot from an oblique direction, the shooting target T in the shot image M1 is distorted.

  As shown in FIG. 4, the control unit 101 stores, in the storage unit 104, a captured image M1, a reduced image M2 obtained by thinning out pixels of the captured image M1, and “contour undetected” IR1 as information regarding a contour to be described later. Correspond and memorize.

FIG. 4 is a schematic diagram showing the contents stored in the storage unit 104.
It is desirable that the reduced image M2 has a resolution (number of pixels) sufficient to be displayed on the display unit 105 and grasp the entire image.

For example, there are three types of information related to the contour stored in the storage unit 104 as shown in FIG.
FIG. 5 is a diagram showing information about the contour.
If the contour of the imaging target T is not obtained, “contour not detected” IR1 is stored. If it is not necessary to obtain the contour, “contour detection not required” IR2 is stored. When the contour is obtained, “contour detection− (x1, y1) − (x2, y2) − (x3, y3) − (x4, y4)]” IR3 including the coordinate information of the contour is stored.

  After shooting, when the shooting / playback switching key is pressed to switch to the playback mode, the control unit 101 reads the reduced image M2 stored in the storage unit 104 and displays it on the display unit 105 as shown in FIG. . The time for performing this process is assumed to be t1.

FIG. 6 is a diagram illustrating an image displayed on the display unit 105.
When the photographing target T is photographed from an oblique direction, the photographing target T is distorted and difficult to see. Therefore, when the reduced image M2 is displayed on the display unit 105 and there is an instruction from the user, the correction shown in FIGS. 7 and 8 is performed.

FIG. 7 is a diagram showing an outline of correction.
If there is a user instruction from the input unit 107 at the stage D2 after the display process of the reduced image M2 (D1 in FIG. 7) and it is necessary to perform correction, the contour detection process (stage in FIG. 7) is performed. D3) to high-definition front image display processing (step D6 in FIG. 7) are performed.

  In step D3, the control unit 101 reads the captured image M1 from the storage unit 104, and detects an outline by the image processing unit 106 as described later. When the contour is detected, the control unit 101 associates the captured image M1 and the reduced image M2 with “contour detection− (x1, y1) − (x2, y2) − (x3, y3) − (x4, y4)”. The information of IR3 is written in the storage unit 104. When the contour is not detected, the control unit 101 writes “contour detection unnecessary” IR2 in the storage unit 104 in association with the captured image M1 and the reduced image M2.

  The control unit 101 sets time required for this contour detection to t2. In step D5 after step D4, the control unit 101 uses the detected contour to create an image corrected to a front image by the image processing device 106. The time required for this processing is assumed to be t3. In step D6, the image corrected to the front image is displayed on the display unit 105. The time required for this processing is assumed to be t4. The time required for this series of processing is t1 + t2 + t3 + t4.

When the front image is displayed again, a low-resolution front image is obtained in step D7, and the processes in steps D8 to D11 are performed.
In step D7, the control unit 101 reads the reduced image M2 stored in the storage unit 104, and further reads information related to the contour. If this information is “contour detection− (x1, y1) − (x2, y2) − (x3, y3) − (x4, y4)]” IR3, the control unit 101 uses the image processing unit 106 to detect the contour. A low-resolution front image corrected from the reduced image M2 to the front image is created using the coordinates of the contour reduced by the ratio of the reduced image M2 / captured image M1 from the vertex coordinates. The time required for this processing is assumed to be t5. The control unit 101 displays the low-resolution front image on the display unit 105 as shown in FIG. 6B (step D8 in FIG. 7).

As a result of displaying the low-resolution front image in step D8, if an instruction from the user is input from the input unit 107 in step D9 and a high-resolution front image is required, the control unit 101 captures the image in step D10. A high-resolution front image corrected to a front image is created by the image processing unit 106 using the image M1 and information about the contour. In step D <b> 11, the image corrected to the front image is displayed on the display unit 105.
The time required for this series of processing from stage D7 to stage D11 is t5 + t6 + t3 + t4.

FIG. 8 is an explanatory diagram showing the resolution of an image.
As shown in FIG. 8, when the low-resolution image M3 corrected from the reduced image M2 to the front image is created, since the resolution (number of pixels) of the reduced image M2 before correction is originally small, the resolution of the image M3 after correction is also high. The display is low. On the other hand, when the high-resolution image M4 corrected to the captured image M1 or the front image is created, the corrected image M4 has a high resolution because the captured image M1 before correction has a sufficient resolution (number of pixels). Display.
FIG. 9 shows a processing flow for efficiently performing the steps D1 to D11.

FIG. 9 is a flowchart for performing the correction process.
In step S <b> 1 of FIG. 9, the control unit 101 reads out information related to the contour (contour detection information) from the storage unit 104. When the information regarding the contour is “contour not detected” IR1 or “contour detection not necessary” IR2 (step S2: NO), the control unit 101 reads the reduced image M2 and displays it on the display unit 105 (step S3).

  In step S4, when there is a display stop instruction from the user via the input unit 107 (step S4: YES), the control unit 101 ends the process. That is, step S4 corresponds to stage D2 in FIG.

  If there is no instruction to stop the display (step S4: NO), the control unit 101 determines whether or not the information regarding the contour is “contour not detected” IR1 (step S5). If it is not “contour undetected” IR1 (step S5: NO), the control unit 101 ends the process, and if it is “contour not detected” IR1 (step S5: YES), the contour detection of step S6 is performed. If the contour is not obtained as a result of the contour detection (step S7: NO), the control unit 101 writes the information of “no contour detection necessary” IR2 in the storage unit 104 (step S8). When the coordinates of the vertex of the contour are obtained as the contour (step S7: YES), the control unit 101 includes “contour detection− (x1, y1) − (x2, y2) − (x3, y3) − (x4, y4) ”IR3 information is written in the storage unit 101.

  In step S10 after step S9, the control unit 101 reads the captured image M1 from the storage unit 104, corrects the captured image M1 to a high-resolution front image by the image processing unit 106, and displays it on the display unit 105. .

Here, the details of step S7 and step S10 will be described.
First, a basic concept (implementation method) of affine transformation used by the image processing unit 106 for image processing will be described.

Affine transformation is widely applied to image spatial transformation. In this embodiment, projective transformation is performed using two-dimensional affine transformation without using three-dimensional camera parameters. This is because the coordinates (u, v) before conversion are related to the coordinates (x, y) after conversion by movement, enlargement / reduction, rotation, etc. Become. Projective transformation can also be performed by this affine transformation.
Expression 2 is an expression for projective transformation, and the coordinates x and y are reduced toward 0 according to the value of z ′. That is, the parameter included in z ′ affects the projection. This parameter is a13, a23, a33. Since other parameters can be normalized by the parameter a33, a33 may be 1.

FIG. 10 shows the coordinates of the vertices of the rectangular imaging target T in the captured image.
The relationship between the quadrangle photographed with the digital camera 100 and the actual photographing object will be described with reference to FIG.
In FIG. 11, the U-V-W coordinate system is a three-dimensional coordinate system of an image obtained by photographing with the digital camera 100. The A (Au, Av, Aw) vector and the B (Bu, Bv, Bw) vector represent the object to be imaged as vectors in the three-dimensional coordinate system UV-W.
The S (Su, Sv, Sw) vector indicates the distance between the origin of the three-dimensional coordinate system UV-W and the imaging target T.
The projection screen shown in FIG. 11 is for projecting an image of the photographing target T.

If the coordinate system on the projection screen is x, y, the image projected on the projection screen may be considered as an image photographed by the digital camera 100. It is assumed that the projection screen is positioned vertically at a distance f from the W axis. An arbitrary point P (u, v, w) of the object to be photographed is connected to the origin by a straight line, and there is a point that intersects the straight line and the projection screen, and the XY coordinate of the intersection is p (x, y ). At this time, the coordinate p is expressed by the following equation 3 by projective transformation.

From Equation 3, the relationship shown in the following Equation 4 is obtained from the relationship between the points P0, P1, P2, P3 and the projection points p0, p1, p2, p3 on the projection screen as shown in FIG.
At this time, the projection coefficients α and β are expressed by the following equation (5).

Next, projective transformation will be described.
An arbitrary point P on the object to be imaged is expressed by the following equation 6 using S, A, and B vectors.
When the relational expression of Expression 4 is substituted into Expression 6, the coordinates x and y are expressed by the following Expression 7.
When this relationship is applied to the affine transformation formula, the coordinates (x ′, y ′, z ′) are expressed by the following equation (8).

  By substituting m and n into Equation 8, the corresponding point (x, y) of the captured image is obtained. Since the corresponding point (x, y) is not necessarily an integer value, the pixel value may be obtained using an image interpolation method or the like.

For m and n, an image size (0 ≦ u <umax, 0 ≦ v <vmax) for outputting the corrected image p (u, v) is given in advance, and a method of adjusting the image according to the image size is considered. It is done. According to this method, m and n are expressed by the following equation (9).

However, the aspect ratio of the corrected image to be created does not match the aspect ratio of the shooting target. Here, the relationship between the corrected image p (u, v) and the values of m and n is expressed by the following Expression 10 from Expression 3, Expression 4, and Expression 5.
If the focal length f of the lens, which is a camera parameter, is known, the aspect ratio k can be obtained according to Equation 10. Therefore, if the image size of the corrected image p (u, v) is (0 ≦ u <umax, 0 ≦ v <vmax), m and n are obtained according to the following equation 11 to obtain the same as the object to be imaged. An aspect ratio k can be obtained.

If the camera has a fixed focus, the value of the focal length f of the lens can be obtained in advance. When a zoom lens or the like is present, the value of the focal length f of the lens changes depending on the zoom magnification of the lens. Therefore, a table indicating the relationship between the zoom magnification and the focal length f of the lens is created and stored in advance. Then, the focal length f can be read based on the zoom magnification, and projective transformation can be performed according to Equations 10 and 11.
In order to perform such affine transformation, the image processing unit 106 first extracts projection parameters from the captured image of the photographic subject.

Projection parameter extraction processing executed by the image processing unit 106 will be described with reference to the flowchart shown in FIG.
When the shooting target T is a square document, the image processing unit 106 extracts the four corner coordinates (rectangular outline) of the image of the shooting target T from the shot image of the shooting target T (step S31). The image processing unit 106 extracts a rectangular outline shown in the flowchart of FIG.

That is, the image processing unit 106 reads the reduced image M2 from the storage unit 104 in order to reduce the number of operations for image processing. (Step S41 in FIG. 13).
The image processing unit 106 creates an edge image of the shooting target T from the reduced image M2 (step S42).

The image processing unit 106 detects a straight line parameter included in the edge image from the edge image of the photographing target T (step S43).
The image processing unit 106 creates a quadrangle that is a candidate for forming the contour of the object to be photographed from the detected straight line parameters (step S44).
The image processing unit 106 generates candidate rectangles in this way, and assigns priorities to the generated rectangles (step S32 in FIG. 12).

The image processing unit 106 selects a rectangle according to the priority order, and determines whether or not the selected rectangle has been extracted (step S33).
If it is determined that the quadrangle could not be extracted (step S33: NO), the control unit 101 ends the projection parameter extraction process.

On the other hand, if it is determined that the quadrangle has been extracted (step S33: YES), the control unit 101 acquires the extracted quadrangle from the image processing unit 106 and sends it to the display unit 105 to display a quadrangle preview image ( Step S34).
Here, the control unit 101 determines which of the YES key and the NO key has been pressed based on the operation information transmitted from the input unit 107 (step S35).

If the control unit 101 determines that the NO key has been pressed (step S35: NO), the image processing unit 106 designates the next candidate rectangle (step S36).
On the other hand, if it is determined that the YES key is pressed (step S35: YES), the control unit 101 calculates an affine parameter from the extracted quadrangular vertex (step S37).

Next, the projection parameter extraction process will be described more specifically.
An example of the reduced image M2 generated by the image processing unit 106 in step S41 is shown in FIG. The image processing unit 106 generates an edge image as shown in FIG. 14B from the reduced image M2 using a filter for edge detection called a Roberts filter (step S42). The Roberts filter is a filter that detects an edge of an image by weighting two 4-neighboring pixels, obtaining two filters Δ1 and Δ2, and averaging them.

FIG. 15A shows the coefficients of the filter Δ1, and FIG. 15B shows the coefficients of the filter Δ2. When the coefficients of the two filters Δ1 and Δ2 are applied to the pixel value f (x, y) of a certain coordinate (x, y), the converted pixel value g (x, y) Represented by
The edge image shown in FIG. 14B includes a straight line parameter. The image processing unit 106 performs radon transformation from the edge image to detect straight line parameters (processing in step S43).

The Radon transform is an integral transform that associates n-dimensional data with (n-1) -dimensional projection data. Specifically, as shown in FIGS. 16A and 16B, an r-θ coordinate system rotated by an angle θ from the xy coordinate system with image data f (x, y) is considered. Image projection data p (r, θ) in the θ direction is defined by the following equation (13).

This conversion by Equation 13 is called Radon conversion.
Image data f (x, y) as shown in FIG. 16A is converted into image projection data p (r, θ) as shown in FIG. 16B by Radon transformation.
According to such a Radon transform, a straight line L in the xy coordinate system as shown in FIG. 17A is represented by ρ = x cos α + y sin α in the polar coordinate system. Since this straight line L is entirely projected on the point p (ρ, α), it is possible to detect the straight line by detecting the peak of p (r, θ). Using this principle, the image processing unit 106 generates data p (r, θ) from the edge image by Radon transform.

  Next, the image processing unit 106 extracts the peak point from the generated data p (r, θ). For this reason, the image processing unit 106 executes this peak point extraction processing according to the flowchart shown in FIG.

The image processing unit 106 searches for the maximum value pmax at p (r, θ) (step S51).
The image processing unit 106 obtains a threshold value pth = pmax * k (k is a constant value from 0 to 1) (step S52).
The image processing unit 106 generates a binary image B (r, θ) from p (r, θ) using pth as a threshold (step S53).

The image processing unit 106 performs image labeling of B (r, θ). The number of labels obtained at this time is set to N1 (step S54).
The image processing unit 106 examines r and θ where p (r, θ) has the maximum value in each label area. The image processing unit 106 acquires these values as ri, θi (i = 1 to Nl), respectively (step S55). This is a straight line parameter.
Next, when the quadrilateral candidate is generated using the detected straight line parameter (the process of step S44 in FIG. 13), the image processing unit 106 forms four quadrilaterals as shown in FIG. 19A. In this case, for example, the opposing straight lines a1 and a3 have two straight lines a2 and a4 other than the opposing straight lines a1 and a3 and intersections p1, p4, p2, and p3, respectively. In this case, the control unit 101 determines that the quadrangle has been extracted (step S33 in FIG. 12: YES).

On the other hand, as shown in FIGS. 19B and 19C, if there are no four intersections, the control unit 101 determines that the quadrangle could not be extracted (step S33: NO).
Next, the image processing unit 106 selects the most suitable rectangle as the one representing the side of the object to be photographed from the extracted rectangle candidates.

Several methods are conceivable. In the present embodiment, the outermost quadrangle is selected from the captured quadrangles.
If the coordinates of the four vertices of the rectangle Ri are (x0, y0), (x1, y1), (x2, y2), and (x3, y3), the square area Si is expressed by the following equation (14). Is done.

The image processing unit 106 selects this rectangle according to the flowchart shown in FIG.
The image processing unit 106 selects one of the plurality of candidate Nr quadrangles (step S61).

The image processing unit 106 calculates an area Si of the selected quadrangle according to Equation 14 (step S62).
The image processing unit 106 decrements the number of candidates Nr (step S63).
The image processing unit 106 determines whether the number of candidates Nr has become 0 (step S64).

When it is determined that the number of candidates Nr is not 0 (step S64: NO), the image processing unit 106 executes the processes of steps S61 to S63 again.
When it is determined that the number of candidates Nr has become 0 by repeating such processing (step S64: YES), the image processing unit 106 arranges candidate quadrilateral data in descending order of the calculated area Si. Change (step S65).
Then, the image processing unit 106 sets the first rectangle as the highest-priority rectangle outline. Thus, even if there are a plurality of quadrangle candidates, the quadrilateral with the maximum outline is always preferentially selected. In this way, the preferential selection of the quadrangular shape of the maximum outline is usually performed by zoom adjustment and adjustment of the photographing position so that the object to be photographed becomes the maximum within the photographing angle of view. This is because the quadrangle is considered as the outline of the object to be imaged.

  Therefore, in general, it is expected that the outline of the imaging target T is automatically extracted. In addition, even if a quadrangle is extracted by mistake, the user generally confirms sequentially from the quadrangle of the maximum outline. For this reason, if a straight line indicating the outline of the photographing target T is extracted, the true square of the photographing target T is selected by sequentially pressing the NO key.

  In this way, using the coordinates (x0, y0), (x1, y1), (x2, y2), (x3, y3) of the four points of the quadrangle of the selected photographing object, According to Equation 8, affine parameters that are each element in the matrix shown in Equation 8 can be obtained.

(1) Extraction of Affine Parameter from Image to be Captured Based on such a concept, the image processing unit 106 acquires an affine parameter from a rectangular vertex. This process will be described based on the flowchart shown in FIG.
The image processing unit 106 calculates projection coefficients α and β according to Equation 5 from the four vertex coordinates (x0, y0), (x1, y1), (x2, y2), and (x3, y3) of the quadrangle. (Step S71).

The image processing unit 106 calculates the aspect ratio k of the object to be photographed according to Equation 10 (step S72). The image processing unit 106 designates the center point (uc, vc) of the image (step S73). The image processing unit 106 compares the maximum image size vmax / umax with the aspect ratio k expressed by Equation 10 (step S74).
When vmax / umax ≦ k (step S74: NO), the image processing unit 106 assumes that the maximum image size umax on the U-axis side (lateral) is larger than the image size of the object to be photographed, assuming that the aspect ratio k is not changed. Is also determined to be large. Then, the image processing unit 106 obtains the values of m and n according to the condition (1) of Formula 11 so that the maximum image size on the V-axis side matches the image size of the photographing target (Step S75).

  If vmax / umax> k (step S74: YES), assuming that the aspect ratio k is not changed, the image processing unit 106 determines that the maximum image size vmax on the V-axis side (vertical) is larger than the image size of the object to be photographed. Is also determined to be large. Then, the image processing unit 106 obtains the values of m and n according to the condition (2) of Equation 11 so that the maximum image size on the U-axis side matches the image size of the photographing target (Step S76).

  The image processing unit 106 calculates an affine transformation from the calculated vertex coordinates (x0, y0), (x1, y1), (x2, y2), and (x3, y3) of the four points of the quadrangle according to Equation 8 A matrix Af is obtained (step S77).

The image processing unit 106 acquires the affine parameter A using each element of the affine transformation matrix Af as the affine parameter A (step S78).
Note that, as in the case where the quadrangle cannot be recognized (step S33 in FIG. 12: NO), the image capturing condition or the like is inappropriate, and the projection parameter may not be obtained. In such a case, for example, a warning text is displayed on the display unit 105 to appropriately warn the photographer that the area could not be detected.

(2) Image Conversion Using Extracted Affine Parameters Next, an image processing method for creating a corrected image using the obtained affine parameters will be described.

  First, when projective transformation or other affine transformation is performed using affine parameters, as shown in FIG. 22, the point p (x, y) of the original image is transformed by projective transformation using the transformation matrix Ap (after). Assume that it corresponds to the point P (u, v) of the image. In this case, it is better to obtain the point p (x, y) of the original image corresponding to the point P (u, v) of the converted image than to obtain the point P of the converted image corresponding to the point p of the original image. preferable.

Note that when obtaining the coordinates of the point P of the converted image, an interpolation method based on the bilinear method is used. In the bilinear interpolation method, the coordinate point of the other image (transformed image) corresponding to the coordinate point of one image (original image) is searched, and the (pixel) values of four points around the coordinate point of the one image are found. Is a method for obtaining a (pixel) value of a point P (u, v) of the converted image. According to this method, the pixel value P of the point P of the converted image is calculated according to the following Expression 15.

In order to obtain the point p (x, y) of the original image corresponding to the point P (u, v) of the converted image, the image processing unit 106 executes the processing of the flowchart shown in FIG.
The image processing unit 106 initializes the pixel position u of the converted image to 0 (step S91).

  The image processing unit 106 initializes the pixel position v of the converted image to 0 (step S92). The image processing unit 106 substitutes the pixel position (u, v) of the converted image using the affine parameter A obtained by Equations 5 and 8, and according to Equation 2, the pixel position (x, y) of the original image Is obtained (step S93).

The image processing unit 106 obtains the pixel value P (u, v) from the obtained pixel position (x, y) according to the formula 15 by the bilinear method (step S94). The image processing unit 106 increments the coordinate v of the corrected image by one (step S95).
The image processing unit 106 compares the coordinate v of the corrected image with the maximum value vmax of the coordinate v, and determines whether or not the coordinate v of the corrected image is greater than or equal to the maximum value vmax (step S96). When it is determined that the coordinate v is less than the maximum value vmax (step S96: NO), the image processing unit 106 executes steps S93 to S95 again.

When it is determined that the coordinate v has reached the maximum value vmax by repeating the processes in steps S93 to S95 (step S96: YES), the image processing unit 106 increments the coordinate u of the corrected image by one ( Step S97).
The image processing unit 106 compares the coordinate u with the maximum value umax of the coordinate u, and determines whether the coordinate u is equal to or greater than the maximum value umax (step S98). When it is determined that the coordinate u is less than the maximum value umax (step S98: NO), the image processing unit 106 executes the processes of steps S92 to S97 again.

  If it is determined that the coordinate u has reached the maximum value umax by repeating the processes of steps 92 to S97 (step 98: YES), the image processing unit 106 ends the image conversion process.

The description returns to the flowchart of FIG. 9 again.
When the information regarding the contour is “contour detection− (x1, y1) − (x2, y2) − (x3, y3) − (x4, y4)]” IR3 in step S2 (step S2: YES), photographing corresponding to the contour is performed. The vertex coordinates of the target T have already been obtained. In this case, the control unit 101 uses the image processing unit 106 to correct the reduced image M2 to a front image in the same manner as in step S10, and displays the low-resolution front image obtained as a result on the display unit 105 (step S10). S11).

  The stage in which step S11 is performed corresponds to stage D8 in FIG. Here, the control part 101 complete | finishes a process, when there exists a display cancellation instruction | indication from a user (step S12: YES). When there is no display stop instruction from the user (step S12: NO), the control unit 101 reads the captured image M1 from the storage unit 105, corrects the captured image M1 to a front image in the same manner as in step S10, and results thereof. The obtained high-resolution front image is displayed on the display unit 105 (step S13).

  As described above, the digital camera 100 according to the present embodiment does not perform the process of forming the front image when shooting in the shooting mode. Therefore, even when a CPU is mounted as the control unit 101, continuous shooting is sufficiently possible even if the power is low, and the convenience of the digital camera is not impaired.

  Further, when a front image is formed in the reproduction mode, the processing can be stopped in a series of processing steps D2 and D9. Thereby, for example, when the digital camera 100 inputs an instruction to cancel in step D2, it is possible to proceed to display of the next / previous image. Therefore, when the user does not desire the display corrected to the front image at the stage of viewing the reduced image M2, it means that the user quickly proceeds to the previous / next image.

Further, when an instruction to cancel is input in step D9, it is possible to proceed to display of the next / previous image. This means that when a coarse display (low-resolution image) is viewed, fine display is not desired, and the image is quickly advanced to the previous / next image.

  Therefore, even in the operation of sequentially switching and displaying a plurality of images using the keys of the input unit 107, the user corrects the front image when the reduced image M2 before correction is first displayed. If it is desired to display the image, it waits for a while until the processing of the step D6 is completed. If not, it is simply input a stop instruction (only at the time t1), and the display is shifted to the next / previous image display. All you have to do is a natural operation.

On the other hand, for the image that has been corrected to the front image once, if you want to display a finer high-resolution front image at the stage of displaying the low-resolution front image from the beginning, wait a little, If there is not, it is only necessary to input a cancellation instruction (with a processing time of t5 + t6) and to perform a natural operation of shifting to display of the next / previous image. That is, a front image can be generated without deteriorating the operability, convenience, and response characteristics of the digital camera.
Even when a large number of images are stored, even a digital camera with a powerless CPU can be used to view images taken with the cursor key, etc. There is an effect that can be raised.

(Embodiment 2)
FIG. 24 is a flowchart showing editing processing according to the second embodiment of the present invention.
FIG. 25 is an explanatory diagram showing a navigation table stored in the storage unit.

The digital camera 100 according to the first embodiment can be provided with various editing functions such as trimming, resizing, and DPOF (Digital Print Order Format) setting for the captured image M1.
Trimming, resizing, and the like have different processing results depending on whether the front image is corrected or not.

  Therefore, in this embodiment, the navigation table shown in FIG. 25 is stored in the storage unit 104 of the digital camera 100 shown in FIG. In the first column of the navigation table, resizing, trimming, DPOF setting, and the like indicating the contents of editing are stored. The second column of the navigation table indicates whether or not the image displayed on the display unit 105 at that time is to be edited, and “inherit” indicates that the display unit 105 before editing is performed. Indicates that the display state is continued for processing.

  “None” in the second column of the navigation table indicates that correction display is not performed on the front image regardless of the display state of the display unit 105. “Inheritance” in the third column of the navigation table indicates that editing is performed by inheriting the resolution of the image displayed on the display unit 105 as it is. “High resolution” in the third column indicates that editing is performed on a high resolution pixel regardless of the display state of the display unit 105 at that time.

  When the user designates editing using the key of the input unit 107, the control unit 101 refers to the navigation table in the storage unit 104 (step S100 in FIG. 24). Then, the editing contents corresponding to the instruction are searched from the first column of the navigation table, and information such as whether correction to the front image is necessary or resolution change is acquired from the second and third columns.

  In step S <b> 101, the control unit 101 determines whether or not the second column of the navigation table is “inherit”. If it is “inherit” (step S <b> 101: YES), the control unit 101 displays on the display unit 105. It is determined whether or not the image is a front image (step S102). In the case of a front image (step S102: YES), the control unit 101 determines whether there is a designation for high-resolution display (step S103).

  When the control unit 101 designates high-resolution display according to the stored contents of the third column of the navigation table (step S103: YES), the control unit 101 displays the image currently displayed on the display unit 105 as follows: It is determined whether or not the image is a high resolution image (step S104). When the image currently displayed on the display unit 105 is a low-resolution image (step S104: NO), the control unit 101 reads the captured image M1 from the storage unit 104 and uses the image processing unit 106 to perform high-resolution. Is displayed on the display unit 105 (step S105). In step S106, the control unit 101 performs editing corresponding to the instruction from the user.

  If the image displayed on the current display unit 105 in step S102 is not a front image, the image displayed on the display unit 105 in step S103 is a low resolution image, and the current display unit 105 in step S104. If it is determined that the display has a high resolution, the control unit 101 edits the image currently displayed on the display unit 105 (step S106).

  If “inheritance” is not stored in the second column of the navigation table (step S101: NO), the control unit 101 determines whether “none” is stored in the second column of the navigation table. (Step S107). When “No” is stored in the second column of the navigation table and it is indicated that the front image is not displayed (step S107: YES), the control unit 101 indicates that the image currently displayed on the display unit 105 is the front side. It is determined whether or not it is an image (step S108). When the front image is displayed (step S108: YES), the control unit 101 reads the reduced image M2 from the storage unit 104 and displays it on the display unit 105 (step S109). After step S109, the control unit 101 edits the image currently displayed on the display unit 105 (step S106).

  When it is necessary to save the edited front image (step S110: YES), the control unit 101 that has performed the editing in step S106 writes “contour detection unnecessary” IR2 as information regarding the contour in the storage unit 104 ( Step S111). After step S111 or when there is no need for storage in step S110, the control unit 101 ends the process.

  By performing the above steps S100 to S111, for example, when the display unit 105 is displayed in a state where the front image correction is not performed, a key for instructing trimming of the input unit 107 is pressed. A screen for designating the trimming position, size, etc. is displayed on the display unit 105 as shown in FIG.

  At this time, since the trimming is performed in a state where the front image is not corrected, the screen for designating the trimming position and size is displayed in a state where the front image is not corrected. When the trimming is executed, the control unit 101 reads the captured image M1 stored in the storage device 104, sets the coordinates and size of the trimming frame designated on the screen for designating the trimming position, size, etc., to the captured image M1 / reduced image M2. The trimming process is performed with the value converted at the ratio of the above and stored in the storage unit 104. FIG. 27A shows the contents of the image after trimming.

  On the other hand, when a key for instructing the trimming process of the input unit 107 is pressed in a state where the front image is displayed after correction is performed on the display unit 105, the trimming position, size, and the like are designated as shown in FIG. The screen to be displayed is displayed. At this time, since the trimming is performed with the front image corrected, the screen for designating the trimming position, size, and the like is displayed with the front image corrected. When the trimming is executed, the control unit 101 reads the captured image M1 and information regarding the contour stored in the storage unit 104, creates a front image using the image processing unit 106, and stores the front image as a temporary image in the storage unit 104. To remember.

  Next, trimming processing is performed on the temporary image with a value obtained by converting the position and size designated on the screen for designating the trimming position and size by the ratio of the temporary image / reduced image M2, and the result is stored in the storage unit 104. FIG. 27B shows the contents of the image after trimming.

  When the key for instructing the resizing process in the input unit 107 is pressed, the control unit 101 inherits the correction state of the front image before activation as it is in the trimming as shown in FIGS. 28A and 28B. The size selection screen after resizing shown in FIG. Whether or not to correct the front image at the time of resizing is important for the user, but whether the display is high resolution or low resolution does not affect the processing.

  However, in the case of the above-described trimming processing, since the designation of the coordinate position and size is involved, there may be a case where the error in the correction process becomes large in the display subjected to the rough front image correction, and the screen of FIG. It is assumed that the difference between the image and the image of FIG. It is preferable to specify the coordinate position and size on a finer display. Therefore, when display accuracy is required during editing, a high-resolution image is automatically displayed at the same time as the editing process starts.

  When a key instructing DPOF setting in the input unit 107 is pressed, the control unit 101 displays a DPOF setting screen on the display unit 105 as shown in FIG. In this case, whether or not the image is a front image, in the print processing after DPOF setting, the content of the captured image M1 stored in the storage unit 104 is printed as it is, so that the image corrected to the front image remains as it is. If it is displayed, it may give a misunderstanding that the corrected image is printed on the front image. Therefore, the DPOF setting screen is a screen displaying the reduced image M2.

  Further, even though the front image can be displayed in the shooting mode and the playback mode, it is not stored in the storage unit. Therefore, a process of forming a front image by correction and recording it is also used as a kind of editing. FIG. 28D shows an example of an image displayed with this processing setting.

  By newly creating an image of the corrected content in the front image and editing it for storage in the storage unit 104, a new image corrected to the front image with respect to the reduced image M2 and the captured image M1 is formed based on the contour, By storing this in the storage unit 105, the image corrected to the front image can be browsed.

  However, if the image processed in this way is detected in the same way as the image taken during playback with this digital camera and corrected to the front image, the image itself has already been corrected to the front image. In spite of this, there may be a case where the display corrected to the front image is newly performed by the contour included in the image. Since this is different from the result desired by the user, when storing the corrected image in the front image in the storage unit 104, information on “contour detection required” IR2 is written as information on the contour.

  On the other hand, when displaying, if “contour detection required” IR2 in the storage unit 104 is stored, the contour detection and the front image correction display are not performed, so the stored image is taken as a captured image or a reduced image. And the reduced image thus regarded is displayed as it is.

  As described above, in the digital camera of the present embodiment, the image displayed on the display unit 105 is appropriately switched according to the content of editing. Therefore, there is an advantage that the user can grasp the contents after processing and expect improvement in convenience.

In addition, this invention is not limited to the said embodiment, A various deformation | transformation is possible.
For example, when the contour is detected, a rectangular contour is obtained assuming that the photographing target T is a document, but a contour corresponding to the shape of the photographing target T may be obtained.

1 is a configuration diagram of a digital camera according to an embodiment of the present invention. It is a schematic diagram which shows the imaging | photography state of a digital camera. It is a figure which shows an example of a picked-up image. It is a schematic diagram which shows the memory content of a memory | storage part. It is a figure which shows the information regarding an outline. It is a figure which shows the image displayed on a display part. It is a figure which shows the outline | summary of correction | amendment. It is explanatory drawing which shows the resolution of an image. It is a flowchart for performing a correction process. The coordinates of each vertex of the shooting target in the shot image are shown. It is a figure which shows a projection screen. It is a flowchart which shows the content of the projection parameter extraction process which an image process part performs. It is a flowchart which shows the content of the square outline extraction process which an image process part performs. It is explanatory drawing of a reduction image and an edge image. It is explanatory drawing of the function of Roberts filter. It is an image figure for demonstrating the principle of radon conversion. It is a figure for demonstrating the operation | movement which acquires the data of a polar coordinate system by radon-transforming the straight line of a X, Y coordinate system. It is a flowchart which shows the content of the peak point detection process from the data of the polar coordinate system which an image process part performs. It is explanatory drawing which shows the view which detects a square from the straight line which detected and extracted the peak point. It is a flowchart which shows the content of the selection process of the detected square which an image process part performs. It is a flowchart which shows the content of the process which calculates | requires an affine parameter from the square vertex performed by the image processing apparatus. It is explanatory drawing of reverse conversion. It is a flowchart which shows the content of the image conversion by an affine transformation. It is a flowchart which shows the edit process which concerns on Embodiment 2 of this invention. It is explanatory drawing which shows the navigation table memorize | stored in the memory | storage part. It is a figure which shows the screen which sets trimming. It is a figure which shows the trimmed image. It is a figure which shows the screen for a setting of resizing, DPOF, and a copy.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 ... Digital camera, 101 ... Control part, 102 ... Optical lens, 103 ... Image sensor, 104 ... Memory | storage part, 105 ... Display part, 106 ... Image processing part, 107 ... Input part, D1-D11 ... Stage, M1 ... Photographing Image, M2 ... Reduced image, IR1-IR3 ... Information about contour

Claims (8)

Mode switching means for switching between shooting mode and playback mode;
A shooting unit that captures a shooting target and acquires a shot image when the shooting mode is set;
Storage means for storing the captured image;
Contour forming means for obtaining a contour of the photographing object in the photographed image;
Projection parameter obtaining means for obtaining a projection parameter indicating a relationship between the photographed image and the actual photographing object based on the obtained contour;
A front image forming unit that forms a front image of the photographed image with respect to the photographed image using the projection parameter;
Display means for displaying the photographed image and the front image formed for the photographed image;
A control means for controlling the contour forming means, parameter acquiring means, front image forming means and display means;
The control means includes
Information related to the contour obtained by the contour forming unit and a reduced image of the captured image are stored in the storage unit in association with the captured image,
When the reproduction mode is set, a reduced front image corresponding to a reduced image of the photographed image is created based on the information related to the contour stored in the storage unit and displayed by the display unit, and thereafter An imaging apparatus, wherein the front image corresponding to the captured image is created and displayed by the display means . The control means includes
When the reproduction mode is set, when displaying the front image on the display means, information on the contour stored in the storage means reads, on the information on the contour by the projection parameter acquiring unit Obtaining the projection parameter based on the front image forming means, forming the front image, and displaying the front image by the display means;
The imaging apparatus according to claim 1, wherein: The control means includes
When the playback mode is set, the reduced front image is created and displayed by the display means, and when instructed, the front image corresponding to the captured image is created and displayed by the display means. The photographing apparatus according to claim 1 or 2 , wherein Editing instruction input means for inputting an instruction to edit the captured image, the reduced image, the reduced front image or the front image;
The control unit selects an image to be edited from the photographed image, the reduced image, the reduced front image, or the front image according to the content of the input pre-edit instruction and displays the selected image on the display unit. The photographing apparatus according to any one of claims 1 to 3 , wherein the editing is performed using data corresponding to the image. Mode switching means for switching between shooting mode and playback mode;
A shooting unit that captures a shooting target and acquires a shot image when the shooting mode is set;
Storage means for storing the captured image;
Contour forming means for obtaining a contour of the photographing object in the photographed image;
Projection parameter obtaining means for obtaining a projection parameter indicating a relationship between the photographed image and the actual photographing object based on the obtained contour;
A front image forming means for forming a front image when the photographing object is photographed using the projection parameter;
Display means for displaying the front image formed for the previous photographed image;
In the camera device comprising the contour forming means, the parameter acquiring means, the front image forming means, and the control means for controlling the display means,
Information related to the contour obtained by the contour forming unit and a reduced image of the captured image are stored in the storage unit in association with the captured image,
When the reproduction mode is set, a reduced front image corresponding to a reduced image of the photographed image is created based on the information related to the contour stored in the storage unit and displayed by the display unit, and thereafter A process of creating the front image corresponding to the photographed image and displaying it by the display means ;
An imaging control method comprising: When the playback mode is set, when displaying the front image on the display unit, the information on the contour stored in the storage unit is read, and based on the information on the contour by the projection parameter acquisition unit. Obtaining the projection parameter, forming the front image by the front image forming means, and displaying the front image by the display means,
The imaging control method according to claim 5 , wherein: When the playback mode is set, when the reduced front image is created and displayed by the display means, and there is an instruction, the front image corresponding to the captured image is created and displayed. The shooting control method according to claim 5, wherein the shooting control method is displayed. The photographing apparatus includes an editing instruction input means for inputting an instruction to edit the photographed image, the reduced image, the reduced front image, or the front image,
An image to be edited is selected from the photographed image, the reduced image, the reduced front image, or the front image according to the content of the input pre-edit instruction, and is displayed on the display unit, and corresponds to the image. The imaging control method according to claim 5, wherein the editing is performed using data.