JP4363153B2 - Imaging apparatus, image processing method thereof, and program - Google Patents

Description

  The present invention relates to a photographing apparatus, an image processing method thereof, and a program.

  With the development of digital cameras and the low price of storage-type memory, digital cameras are not only used for taking pictures of landscapes and people, but are also used for paper documents, business cards, and blackboards written at meetings. An application is being considered in which what is displayed is photographed, and these images are digitally stored in a personal computer or the like for management.

  When photographing a document or a blackboard in this way, it is desirable to photograph the photographing object in front and vertically. However, it is difficult to shoot a document placed on the desk vertically and vertically. Also, when shooting a blackboard, it may be difficult to capture the blackboard from the front depending on the position of the photographer. Even if the image can be taken from the front, it may be preferable to avoid taking the image from the front for reasons such as reflection of light. Thus, when an object such as a document is photographed from an oblique direction, characters and the like are distorted obliquely or trapezoidally. Of course, these characters can be interpreted. However, even if it can be read, it is very tiring to read. Moreover, since these images have distortion, it is difficult to reuse these images.

  As a solution to such a problem, for example, there is one in which image distortion is corrected using an image having two parallaxes (see, for example, Patent Document 1).

This image processing unit creates a corrected image that is synthesized by performing image conversion so that parallax is eliminated with respect to two or more images and combining them. Then, the image composition unit stores the created corrected image in the image memory.
JP 2001-134751 (5th page, FIG. 1)

  However, in such a conventional apparatus, when image editing is performed again, the image deteriorates if the corrected image stored in the image memory is edited. If the original image before correction is to be corrected again, it is necessary to store the original image and the corrected image in the image memory, and it is necessary to repeat similar image processing on the original image.

  The present invention has been made in view of such conventional problems, and an object of the present invention is to provide a photographing apparatus capable of easily correcting an image of a photographing target, an image processing method thereof, and a program. .

In order to achieve this object, an imaging device according to the first aspect of the present invention provides:
In a photographing device that photographs a photographing object by a photographing unit,
A photographing unit for photographing the photographing object;
An image processing unit that performs image processing on the photographed image so as to correct distortion of the photographed image of the photographing object obtained by photographing with the photographing unit;
The image processing unit
A shape acquisition unit that acquires the contour of the image from the image of the shooting target, and acquires the shape of the image from the acquired contour;
The shape of the image acquired by the shape acquisition unit and the actual shape of the object to be photographed are associated with each other by setting a three-dimensional coordinate system. A projection parameter obtaining unit for obtaining a projection parameter indicating a relationship with the object to be imaged,
An aspect ratio acquisition unit for obtaining an aspect ratio of the photographing object based on a focal length of a lens included in the photographing unit and a vertex position of an image of the photographing object;
Using the projection parameter obtained by the projection parameter acquisition unit and the aspect ratio obtained by the aspect ratio acquisition unit, an image conversion parameter for image conversion is obtained, and the image conversion parameter is used to calculate the image of the subject to be imaged. An image conversion unit that performs image conversion of the image;
The image conversion unit includes a storage control unit that stores an image conversion parameter used for image conversion of the image of the photographing object in an image file of the storage unit.

The storage control unit may store the image data of the photographing object obtained by photographing with the photographing unit in the image file of the storage unit, and store the image conversion parameter in the image file. Good.

  The storage control unit may store the image converted by the image conversion unit in the image file of the storage unit.

  The storage unit may be configured by an external storage medium.

From the shape of the image acquired by the shape acquisition unit, an actual target part representing the content of the photographing object is determined, and an image cutout unit that cuts out the image of the determined real target part;
A correction parameter acquisition unit for acquiring an image effect correction parameter for correcting an image effect from the image of the real target part cut out by the image cutout unit;
An image effect processing unit that performs image effect processing of the image of the object to be photographed using the image effect correction parameter acquired by the correction parameter acquisition unit;
The storage control unit may store an image effect correction parameter used by the image effect processing unit in the image effect processing of the image of the shooting target in the image file of the storage unit.

  The image conversion unit may read the stored image effect correction parameter from the image file in the storage unit, and perform the image conversion using the read image effect correction parameter.

  You may provide the display part which displays the said image.

An image processing method of the photographing apparatus according to the second aspect of the present invention is as follows.
An image processing method of a photographing apparatus that performs image processing on an image of the photographing target so as to correct distortion of the image of the photographing target obtained by photographing the photographing target by a photographing unit. ,
Acquiring a contour of the image from the image of the photographing object, and acquiring a shape of the image from the acquired contour;
The acquired shape of the image and the shape of the actual shooting target are associated with each other by setting a three-dimensional coordinate system, and from the vertex position of the image of the shooting target, the image of the shooting target and the actual shooting target Obtaining a projection parameter indicating a relationship with the object to be imaged;
Obtaining an aspect ratio of the photographing object based on a focal length of a lens included in the photographing unit and a vertex position of an image of the photographing object;
Using the obtained projection parameters and aspect ratio to determine image conversion parameters for image conversion, and using the image conversion parameters to perform image conversion of the image of the object to be photographed;
Storing image conversion parameters used for the image conversion in an image file storing image data.

The program according to the third aspect of the present invention is:
In the computer of the photographing device that photographs the object to be photographed by the photographing unit,
A procedure for acquiring the contour of the image from the image of the shooting target obtained by shooting the shooting target, and acquiring the shape of the image of the shooting target from the acquired contour,
The acquired shape of the image and the shape of the actual shooting target are associated with each other by setting a three-dimensional coordinate system, and from the vertex position of the image of the shooting target, the image of the shooting target and the actual shooting target A procedure for obtaining a projection parameter indicating the relationship with the object to be photographed;
A procedure for obtaining an aspect ratio of the photographing object based on a focal length of a lens of the photographing unit and a vertex position of an image of the photographing object;
An image conversion parameter for image conversion is obtained using the obtained projection parameter and aspect ratio, and the image is imaged with respect to the image so as to correct the distortion of the image of the object to be photographed using the image conversion parameter. The steps to do the conversion,
A procedure for storing image conversion parameters used for the image conversion in an image file storing image data;
Is to execute.

  According to the present invention, it is possible to easily correct an image of a photographing object.

Hereinafter, a photographing apparatus according to an embodiment of the present invention will be described with reference to the drawings.
In the embodiment, the photographing apparatus is described as a digital camera.

A configuration of a digital camera 1 according to the present embodiment is shown in FIG.
The digital camera 1 according to the present embodiment captures characters such as a white board 2, a figure, a photograph, and the like as an object to be photographed, detects and corrects image distortion from an image obtained by photographing, and is as if from the front. It generates an image as if it was taken. The digital camera 1 includes a photographic lens unit 11, a liquid crystal monitor 12, and a shutter button 13.

  The photographic lens unit 11 includes a lens that collects light and the like, and collects light from characters such as the white plate 2, drawings, and photographs.

The liquid crystal monitor 12 is for projecting an image taken in through the photographing lens unit 11.
The shutter button 13 is pressed when shooting a shooting target.

  As shown in FIG. 2, the digital camera 1 includes an optical lens device 21, an image sensor 22, a memory 23, a display device 24, an image processing device 25, an operation unit 26, a computer interface unit 27, An external storage IO device 28 and a program code storage device 29 are provided.

  The optical lens device 21 includes a photographic lens unit 11 and a driving unit thereof, and forms an image on the image sensor 22 by condensing light from characters, drawings, photographs, and the like on the white plate 2. .

  The image sensor 22 is for capturing a formed image as digitized image data, and is constituted by a CCD or the like. If the image sensor 22 is controlled by the CPU 30 and the shutter button 13 is not pressed, digital image data having a low preview resolution is generated, and the image data is periodically stored in the memory 23 at intervals of about 30 sheets per second. To send. Further, when the shutter button 13 is pressed, the image sensor 22 generates image data with high resolution and sends the generated image data to the memory 23.

  The memory 23 temporarily stores low-resolution preview images from the image sensor 22, high-resolution image data, original image data processed by the image processing device 25, and processed image data. The memory 23 sends the temporarily stored image data to the display device 24 or the image processing device 25.

  The display device 24 includes the liquid crystal monitor 12 and displays the image on the liquid crystal monitor 12. The display device 24 displays a low-resolution preview image or a high-resolution image temporarily stored in the memory 23 on the liquid crystal monitor 12.

  The image processing device 25 is for performing image processing such as image data compression, image distortion correction, and image effect processing on the image data temporarily stored in the memory 23.

  As shown in FIG. 3A, when the digital camera 1 shoots characters on the white plate 2 from the left direction and the right direction, the liquid crystal monitor 12 displays the characters shown in FIGS. 3B and 3C. As shown, the white plate 2 and images such as letters, diagrams, and photographs are distorted and displayed. The image processing device 25 performs image processing on the images as shown in FIGS. 3B and 3C to generate an image taken from the front as shown in FIG. 3D. To do.

  In order to correct image distortion, the image processing device 25 cuts a square from the distorted image, and performs projective transformation of the captured image into the cut quadrangle.

More specifically, the image processing device 25 is controlled by the CPU 30 and mainly performs the following processing.
(1) Extraction of affine parameters from photographed image (2) Image conversion using extracted affine parameters (3) Adjustment processing of image conversion (4) Extraction of image effect correction parameters relating to luminance or color difference and image effect processing The processing contents of will be described later.

  The operation unit 26 includes switches and keys for controlling the document projection function. When the user presses these keys and switches, the operation unit 26 responds and transmits the operation information at this time to the CPU 30.

  As shown in FIG. 4, the operation unit 26 includes an upper reduction key 111, a lower reduction key 112, a right reduction key 113, a left reduction key 114, a right rotation key 115, and a left rotation key 116. .

  The upper reduction key 111, the lower reduction key 112, the right reduction key 113, and the left reduction key 114 are projective conversion keys for performing projective conversion. The upper reduction key 111 is a key that is pressed when the upper part of the image is compared with the lower part around the X axis and the upper part is large and the upper part is rotated downward toward the paper surface.

  The lower reduction key 112 is a key that is pressed when the upper part and the lower part of the image are compared around the X axis, and the lower part is rotated downward toward the paper surface when the lower part is large.

  The right reduction key 113 and the left reduction key 114 are keys that are pressed when adjusting left and right distortion around the Y-axis side, and the right reduction key 113 is a key that is pressed when the right is large, and left The reduction key 114 is a key that is pressed when the left side is large.

  The right rotation key 115 and the left rotation key 116 are rotation correction keys for adjusting the rotation of the image. The right rotation key 115 is a key that is pressed when the image is rotated to the right, and the left rotation key 116 is a key that is pressed when the image is rotated to the left.

  In addition, the operation unit 26 includes a photographing key, a reproduction key, a cursor key, a control key, and the like (not shown). The shooting key is a key for selecting a shooting mode when shooting a shooting object. The reproduction key is a key for selecting a reproduction mode for reproducing an image to be photographed obtained by photographing. The control key is a key having functions such as a YES key for confirming the operation, a NO key for canceling the operation, and an editing key for performing editing.

  The computer interface unit 27 operates as a USB store registration class driver when the digital camera 1 is connected to a computer (not shown). Thus, when the computer is connected to the digital camera 1, the computer handles the memory card 31 as an external storage device of the computer.

  The external storage IO device 28 inputs and outputs image data and the like with the memory card 31. The memory card 31 stores image data and the like supplied from the external storage IO device 28.

  The program code storage device 29 is for storing a program executed by the CPU 30, and is configured by a ROM or the like.

  The CPU 30 controls the entire system according to a program stored in the program code storage device 29. The memory 23 is also used as a work memory for the CPU 30.

  When operation information is transmitted from the operation unit 26 by pressing a switch or key of the operation unit 26, the CPU 30 performs image sensor 22, memory 23, display device 24, image processing device based on the operation information. 25 etc. are controlled.

  Specifically, when the operation information indicating that the photographing key is pressed is transmitted from the operation unit 26, the CPU 30 sets each unit to the photographing mode. The CPU 30 sets the image sensor 22 to the preview mode if the shutter button 13 is not pressed in the state set to the shooting mode, and reads the high-resolution shooting target image if the shutter button 13 is pressed. Set to. Further, when the operation information indicating that the reproduction key is pressed is transmitted, the CPU 30 sets each unit to the reproduction mode.

  Further, when the operation information indicating that the projection conversion key and the rotation correction key are pressed is transmitted from the operation unit 26, the CPU 30 transmits the operation information to the image processing device 25, and causes the image processing device 25 to operate. Control.

  When the image data is temporarily stored in the memory 23, the CPU 30 records the preview image and the high-resolution image data in different storage areas.

  Further, the CPU 30 records preview image and high-resolution image data on the memory card 31 via the external storage IO device 28, and reads the recorded image data from the memory card 31. For this reason, the CPU 30 creates an image file for storing image data in the memory card 31.

  Then, for example, the CPU 30 records the corrected image data compressed in the JPEG format into image files. Further, the CPU 30 creates a header information storage area of the image file for each image file and records the image data, such as the projection parameters and image effect processing correction parameters obtained by the image processing device 25 when the image data is recorded. Is recorded in this header information storage area. This header information will be described later.

Next, the operation of the digital camera 1 according to this embodiment will be described.
When the user turns on (turns on) the digital camera 1, the CPU 30 acquires program data stored in the program code storage device 29. When the user presses the shooting button, the operation unit 26 transmits this operation information to the CPU 30. The CPU 30 receives this operation information, and the CPU 30, the image processing device 25, etc. execute the photographing process according to the flowchart shown in FIG.

The CPU 30 sets the image sensor 22 to the preview mode (step S11).
The CPU 30 determines whether or not the shutter button 13 has been pressed based on the operation information transmitted from the operation unit 26 (step S12).

  If it is determined that the shutter button 13 has been pressed (Yes in step S12), the CPU 30 controls the image sensor 22 by switching the image sensor 22 from the preview mode to the high resolution mode (step S13).

  The CPU 30 records the high-resolution photographic target image data generated by the image sensor 22 in a storage area different from the preview image on the memory 23 (step S14).

The CPU 30 determines whether the reading of the image data has been completed (step S15).
If it is determined that the reading has not been completed (No in step S15), the CPU 30 continues to control the image sensor 22 to read the image data.

  If it is determined that all the image data has been read and the image transfer has been completed (Yes in step S15), the CPU 30 generates a low-resolution preview image from the captured image (high-resolution image), and uses the preview image for the preview image in the memory 23. The preview image data is written in the storage area (step S16).

The CPU 30 controls the image processing device 25 so as to create compressed data, and the image processing device 25 creates compressed data (step S17).
The CPU 30 records this compressed data on the memory card 31 via the external storage IO device 28, and stores the compressed data created by the image processing device 25 (step S18).

  Next, the image processing device 25 extracts projection parameters for creating a front image from the captured image under the control of the CPU 30 (step S19).

The CPU 30 and the image processing device 25 determine whether or not the projection parameters have been extracted (step S20).
If it is determined that the extraction has been completed (Yes in step S20), the image processing device 25 creates a projected transformation image based on the extracted projection parameters (step S21).

  When the projection conversion key and the rotation correction key of the operation unit 26 are pressed, the operation unit 26 transmits this operation information to the CPU 30. The CPU 30 transmits operation information from the operation unit 26 to the image processing device 25, and the image processing device 25 performs image conversion adjustment processing in accordance with the transmitted operation information (step S22).

  The image processing device 25 extracts image effect correction parameters (step S23) and performs image effect processing (step S24).

  The image processing device 25 performs compression processing on the image data that has been subjected to the image effect processing, and creates compressed data (step S25).

  The image processing device 25 records the created compressed data in the image file of the memory card 31 (step S26). Further, the CPU 30 stores the projection parameters used for the projective conversion and the image effect correction parameters used for the image effect processing in the image file of the memory card 31.

  On the other hand, when it is determined that the projection parameters could not be extracted (No in step S20), the CPU 30 performs a warning process (step S27).

  The CPU 30 and the like end the photographing process in this way. Note that the CPU 30 or the like repeatedly executes this photographing process unless the user operates the key.

Next, image processing performed by the image processing device 25 will be described.
First, the basic concept (implementation method) of affine transformation used by the image processing apparatus 25 for image processing will be described.

Affine transformation is widely applied to image spatial transformation. In this embodiment, projective transformation is performed using two-dimensional affine transformation without using three-dimensional camera parameters. This is because the coordinates (u, v) before conversion are related to each other by the following equation (1) by conversion such as movement, enlargement / reduction, and rotation. Projective transformation can also be performed by this affine transformation.

The final coordinates (x, y) are calculated by the following formula 2.

Expression 2 is an expression for projective transformation, and the coordinates x and y are reduced toward 0 according to the value of z ′. That is, the parameter included in z ′ affects the projection. This parameter is a13, a23, a33. Since other parameters can be normalized by the parameter a33, a33 may be 1.

  FIG. 6 shows the coordinates of each vertex of a rectangular captured image. The relationship between the quadrangle photographed with the digital camera 1 and the actual photographing object (white plate 2) will be described with reference to FIG.

In FIG. 7, the UVW coordinate system is a three-dimensional coordinate system of an image obtained by photographing with the digital camera 1. The A (Au, Av, Aw) vector and the B (Bu, Bv, Bw) vector represent the object to be imaged as vectors in the three-dimensional coordinate system UV-W.
The S (Su, Sv, Sw) vector indicates the distance between the origin of the three-dimensional coordinate system UV-W and the object to be imaged.

The projection screen shown in FIG. 7 is for projecting an image of the object to be photographed.
If the coordinate system on the projection screen is x, y, the image projected on the projection screen may be considered as the image photographed by the digital camera 1. It is assumed that the projection screen is positioned vertically at a distance f from the W axis. An arbitrary point P (u, v, w) of the object to be photographed is connected to the origin by a straight line, and there is a point that intersects the straight line and the projection screen, and the XY coordinate of the intersection is p (x, y ). At this time, the coordinate p is expressed by the following equation 3 by projective transformation.

From Equation 3, the relationship shown in the following Equation 4 is obtained from the relationship between the points P0, P1, P2, P3 and the projection points p0, p1, p2, p3 on the projection screen as shown in FIG.

At this time, the projection coefficients α and β are expressed by the following equation (5).

Next, projective transformation will be described.
An arbitrary point P on the object to be imaged is expressed by the following equation 6 using S, A, and B vectors.

When the relational expression of Expression 4 is substituted into Expression 6, the coordinates x and y are expressed by the following Expression 7.

When this relationship is applied to the affine transformation formula, the coordinates (x ′, y ′, z ′) are expressed by the following equation (8).

  By substituting m and n into Equation 8, the corresponding point (x, y) of the captured image is obtained. Since the corresponding point (x, y) is not necessarily an integer value, the pixel value may be obtained using an image interpolation method or the like.

For m and n, an image size (0 ≦ u <umax, 0 ≦ v <vmax) for outputting the corrected image p (u, v) is given in advance, and a method of adjusting the image according to the image size is considered. It is done. According to this method, m and n are expressed by the following equation (9).

However, the aspect ratio of the corrected image to be created does not match the aspect ratio of the shooting target. Here, the relationship between the corrected image p (u, v) and the values of m and n is expressed by the following Equation 10 from Equation 3.

If the focal length f of the lens, which is a camera parameter, is known, the aspect ratio k can be obtained according to Equation 10. Therefore, if the image size of the corrected image p (u, v) is (0 ≦ u <umax, 0 ≦ v <vmax), m and n are obtained according to the following equation 11 to obtain the same as the object to be imaged. An aspect ratio k can be obtained.

If the camera has a fixed focus, the value of the focal length f of the lens can be obtained in advance. When a zoom lens or the like is present, the value of the focal length f of the lens changes depending on the zoom magnification of the lens. Therefore, a table indicating the relationship between the zoom magnification and the focal length f of the lens is created and stored in advance. Then, the focal length f can be read based on the zoom magnification, and projective transformation can be performed according to Equations 10 and 11.

In order to perform such affine transformation, the image processing device 25 first extracts projection parameters from the image of the photographed subject (step S19 in FIG. 5).
Projection parameter extraction processing executed by the image processing device 25 will be described with reference to the flowchart shown in FIG.

  The image processing device 25 extracts coordinate points (rectangular contours) at the four corners of the image of the photographed object from the photographed image of the photographed object (step S31). The image processing device 25 extracts a rectangular outline shown in the flowchart of FIG.

That is, the image processing device 25 generates a reduced luminance image from the input image in order to reduce the number of operations for image processing (step S41).
The image processing device 25 generates an edge image of the shooting target from the generated reduced luminance image (step S42).

The image processing device 25 detects a straight line parameter included in the edge image from the edge image of the photographing object (step S43).
The image processing device 25 creates a quadrangle that is a candidate for forming the contour of the object to be photographed from the detected straight line parameters (step S44).

The image processing device 25 generates the candidate rectangles in this way, and assigns priorities to the generated rectangles (step S32 in FIG. 8).
The image processing device 25 selects a rectangle according to the priority order, and determines whether or not the selected rectangle has been extracted (step S33).

  If it is determined that the quadrangle could not be extracted (No in step S33), the CPU 30 ends the projection parameter extraction process.

  On the other hand, if it is determined that the quadrangle has been extracted (Yes in step S33), the CPU 30 acquires the extracted quadrangle from the image processing device 25, sends it to the display device 24, and displays the quadrangle preview image on the liquid crystal monitor 12. It is displayed (step S34).

  The CPU 30 determines whether the YES key or the NO key has been pressed based on the operation information transmitted from the operation unit 26 (step S35).

  If the CPU 30 determines that the NO key has been pressed (No in step S35), the image processing device 25 designates the next candidate rectangle (step S36).

  On the other hand, if it is determined that the YES key has been pressed (Yes in step S35), the CPU 30 calculates affine parameters from the extracted quadrangular vertices (step S37).

Next, the projection parameter extraction process will be described more specifically.
An example of the reduced luminance image generated by the image processing device 25 in step S41 is shown in FIG. The image processing device 25 generates an edge image as shown in FIG. 10B from such a reduced luminance image using an edge detection filter called a Roberts filter (step S42). The Roberts filter is a filter that detects an edge of an image by weighting two 4-neighboring pixels, obtaining two filters Δ1 and Δ2, and averaging them.

11A shows the coefficient of the filter Δ1, and FIG. 11B shows the coefficient of the filter Δ2. When the coefficients of the two filters Δ1 and Δ2 are applied to the pixel value f (x, y) of a certain coordinate (x, y), the converted pixel value g (x, y) Represented by

  The edge image shown in FIG. 10B includes straight line parameters. The image processing device 25 performs a Radon transform from this edge image to detect a straight line parameter (processing in step S43).

The Radon transform is an integral transform that associates n-dimensional data with (n-1) -dimensional projection data. Specifically, as shown in FIGS. 12A and 12B, an r-θ coordinate system rotated by an angle θ from the xy coordinate system with image data f (x, y) is considered. Image projection data p (r, θ) in the θ direction is defined by the following equation (13).

This conversion by Equation 13 is called Radon conversion.
Image data f (x, y) as shown in FIG. 12A is converted into image projection data p (r, θ) as shown in FIG. 12B by Radon transformation.

  According to such a Radon transform, a straight line L in the xy coordinate system as shown in FIG. 13A is represented by ρ = x cos α + y sin α in the polar coordinate system. Since this straight line L is entirely projected on the point p (ρ, α), it is possible to detect the straight line by detecting the peak of p (r, θ). Using this principle, the image processing device 25 generates data p (r, θ) from the edge image by Radon transform.

  Next, the image processing device 25 extracts the peak point from the generated data p (r, θ). Therefore, the image processing device 25 executes this peak point extraction process according to the flowchart shown in FIG.

The image processing device 25 searches for the maximum value pmax at P (r, θ) (step S51).
The image processing apparatus 25 obtains a threshold value pth = pmax * k (k is a constant value from 0 to 1) (step S52).

The image processing device 25 generates a binary image B (r, θ) from P (r, θ) using pth as a threshold (step S53).
The image processing device 25 performs image labeling of B (r, θ). The number of labels obtained at this time is set to N1 (step S54).

  The image processing device 25 examines r and θ where P (r, θ) has the maximum value in each label area. The image processing device 25 acquires these values as ri, θi (i = 1 to Nl), respectively (step S55). This is a straight line parameter.

  Next, when the quadrilateral candidate is generated using the detected straight line parameter (the process of step S44 in FIG. 9), the image processing device 25 forms a quadrilateral with four straight lines as shown in FIG. In this case, for example, the opposing straight lines a1 and a3 have two straight lines a2 and a4 other than the opposing straight lines a1 and a3 and intersections p1, p4, p2, and p3, respectively. In this case, the CPU 30 determines that the quadrangle has been extracted (Yes in step S33 in FIG. 8).

  On the other hand, as shown in FIGS. 15B and 15C, if there are no four intersections, the CPU 30 determines that the quadrangle could not be extracted (No in step S33).

Next, the image processing device 25 selects the most suitable rectangle as the one representing the side of the object to be photographed from the extracted rectangle candidates.
Several methods are conceivable, and in the present embodiment, the outermost quadrangle is selected from the photographed quadrilaterals. As shown in FIG. 6, the outermost quadrangle is a rectangle that is formed by surrounding a candidate quadrangle with parallel lines of the X and Y axes, and has the largest area.

If the coordinates of the four vertices of the rectangle Ri are (x0, y0), (x1, y1), (x2, y2), and (x3, y3), the square area Si is expressed by the following equation (14). Is done.

The image processing device 25 selects this rectangle according to the flowchart shown in FIG.
The image processing device 25 selects one of the candidate number Nr of rectangles (step S61).
The image processing apparatus 25 calculates the area Si of the selected square according to Equation 10 (step S62).

The image processing device 25 decrements the candidate number Nr (step S63).
The image processing device 25 determines whether the number of candidates Nr has become 0 (step S64).
When it is determined that the number of candidates Nr is not 0 (No in step S64), the image processing device 25 executes the processes in steps S61 to S63 again.

  When it is determined that the number of candidates Nr has become 0 by repeating such processing (Yes in step S64), the image processing device 25 arranges the candidate quadrilateral data in descending order of the calculated area Si. Change (step S65).

  Then, the image processing apparatus 25 sets the first quadrangle as the highest priority quadrilateral outline. Thus, even if there are a plurality of quadrangle candidates, the quadrilateral with the maximum outline is always preferentially selected. In this way, the preferential selection of the quadrangular shape of the maximum outline is usually performed by zoom adjustment and adjustment of the photographing position so that the object to be photographed becomes the maximum within the photographing angle of view. This is because the quadrangle is considered as the outline of the object to be imaged.

  Therefore, it is generally expected that the outline of the object to be imaged is automatically extracted. In addition, even if a quadrangle is extracted by mistake, the user generally confirms sequentially from the quadrangle of the maximum outline. For this reason, if a straight line indicating the outline of the object to be imaged is extracted, the true rectangle of the object to be imaged is selected by sequentially pressing the NO key.

  In this way, using the coordinates (x0, y0), (x1, y1), (x2, y2), (x3, y3) of the four points of the quadrangle of the selected photographing object, According to Equation 8, affine parameters that are each element in the matrix shown in Equation 8 can be obtained.

(1) Extraction of Affine Parameter from Image to be Captured Based on such a concept, the image processing device 25 acquires an affine parameter from a rectangular vertex. This process will be described based on the flowchart shown in FIG.

  The image processing device 25 calculates projection coefficients α and β according to Equation 5 from the vertex coordinates (x0, y0), (x1, y1), (x2, y2), and (x3, y3) of the four points of the rectangle. (Step S71).

The image processing device 25 calculates the aspect ratio k of the object to be photographed according to Equation 10 (step S72).
The image processing device 25 designates the center point (uc, vc) of the image (step S73).
The image processing device 25 compares the maximum image size vmax / umax with the aspect ratio k expressed by Equation 10 (step S74).

  When vmax / umax ≦ k (No in step S74), the image processing apparatus 25 assumes that the maximum image size umax on the U-axis side (lateral) is larger than the image size of the object to be photographed, assuming that the aspect ratio k is not changed. Is also determined to be large. Then, the image processing device 25 obtains the values of m and n according to the condition (1) of Equation 11 so that the maximum image size on the V-axis side and the image size of the object to be photographed coincide (step S75).

  When vmax / umax> k (Yes in step S74), assuming that the aspect ratio k is not changed, the image processing apparatus 25 determines that the maximum image size vmax on the V-axis side (vertical) is larger than the image size of the object to be photographed. Is also determined to be large. Then, the image processing device 25 obtains the values of m and n according to the condition (2) of Equation 11 so that the maximum image size on the U-axis side matches the image size of the object to be photographed (step S76).

The image processing device 25 calculates the affine transformation from the calculated vertex coordinates (x0, y0), (x1, y1), (x2, y2), (x3, y3) of the four points of the quadrangle according to Equation 8 A matrix Af is obtained (step S77).
The image processing apparatus 25 acquires the affine parameter A using each element of the affine transformation matrix Af as the affine parameter A (step S78).

  Note that, as in the case where the quadrangle cannot be recognized (No in step S33 in FIG. 8), the image capturing condition or the like is inappropriate, and the projection parameter may not be obtained. In such a case, a warning message as shown in FIG. 18A is displayed on the liquid crystal monitor 12 to appropriately warn the photographer that the area could not be detected, and to the photographer, FIG. It is preferable to change the shooting conditions to the camera shooting setting mode as shown in FIG. Furthermore, it may be preferable to warn again to change the shooting conditions.

  When the proper projection parameter cannot be obtained as described above, the CPU 30 performs a warning process according to the flowchart shown in FIG. 19 (step S27 in FIG. 5).

  The CPU 30 controls the display device 24 to display a warning text as shown in FIG. 18A on the liquid crystal monitor 12 (step S81).

  The CPU 30 determines which of the YES key and the NO key is pressed based on the operation information transmitted from the operation unit 26 (step S82).

  When it is determined that the NO key has been pressed (No in step S82), the CPU 30 ends the warning process.

  On the other hand, if it is determined that the YES key has been pressed (Yes in step S82), the CPU 30 switches to the shooting mode (step S83) and ends this warning process.

(2) Image Conversion Using Extracted Affine Parameters Next, an image processing method for creating a corrected image using the obtained affine parameters will be described.
First, when projective transformation or other affine transformation is performed using affine parameters, as shown in FIG. 20, the point p (x, y) of the original image is transformed by projective transformation using the transformation matrix Ap (after). Assume that it corresponds to the point P (u, v) of the image. In this case, it is better to obtain the point p (x, y) of the original image corresponding to the point P (u, v) of the converted image than to obtain the point P of the converted image corresponding to the point p of the original image. preferable.

Note that when obtaining the coordinates of the point P of the converted image, an interpolation method based on the bilinear method is used. In the bilinear interpolation method, the coordinate point of the other image (transformed image) corresponding to the coordinate point of one image (original image) is searched, and the (pixel) values of four points around the coordinate point of the one image are found. Is a method for obtaining a (pixel) value of a point P (u, v) of the converted image. According to this method, the pixel value P of the point P of the converted image is calculated according to the following Expression 15.

In order to obtain the point p (x, y) of the original image corresponding to the point P (u, v) of the converted image, the image processing device 25 executes the process of the flowchart shown in FIG.
The image processing device 25 initializes the pixel position u of the converted image to 0 (step S91).

The image processing device 25 initializes the pixel position v of the converted image to 0 (step S92).
The image processing device 25 substitutes the pixel position (u, v) of the converted image using the affine parameter A obtained by Equations 5 and 8, and according to Equation 2, the pixel position (x, y) of the original image Is obtained (step S93).

The image processing device 25 obtains the pixel value P (u, v) from the obtained pixel position (x, y) according to the formula 15 by the bilinear method (step S94).
The image processing device 25 increments the corrected image coordinate v by one (step S95).

  The image processing device 25 compares the coordinate v of the corrected image with the maximum value vmax of the coordinate v, and determines whether or not the coordinate v of the corrected image is greater than or equal to the maximum value vmax (step S96).

  When it is determined that the coordinate v is less than the maximum value vmax (No in step S96), the image processing device 25 executes steps S93 to S95 again.

  When it is determined that the coordinate v has reached the maximum value vmax by repeating the processes in steps S93 to S95 (Yes in step S96), the image processing device 25 increments the coordinate u of the corrected image by one ( Step S97).

The image processing device 25 compares the coordinate u with the maximum value umax of the coordinate u, and determines whether or not the coordinate u is equal to or greater than the maximum value umax (step S98).
When it is determined that the coordinate u is less than the maximum value umax (No in step S98), the image processing device 25 executes the processes of steps S92 to S97 again.

  When it is determined that the coordinate u has reached the maximum value umax by repeating the processes of steps 92 to S97 (Yes in step 98), the image processing apparatus 25 ends the image conversion process.

(3) Image Conversion Adjustment Next, adjustment (step S22 in FIG. 5) performed on an image that has been once converted will be described.
In the case where some errors are included in the coordinates of the extracted quadrangular vertices, as shown in FIG. 22A, the result of projection with the obtained affine parameters may not be preferable. For this reason, the digital camera 1 of the present embodiment is configured so that the user can adjust the projective transformation in order to adjust the image once converted and obtain an image as shown in FIG. Has been.

  When the user operates the projection conversion key and the rotation correction key of the operation unit 26, the operation unit 26 transmits this operation information to the CPU 30 in response to the user's operation. The CPU 30 determines this operation information and controls the image processing device 25 according to the determination result.

  As shown in FIG. 23, when the interpolation pixel Q (u ′, v ′) of the corrected image is obtained, an inverse transformation Ai is performed on the interpolation pixel Q (u ′, v ′) to obtain the interpolation pixel Q. A corrected image P (u, v) corresponding to (u ′, v ′) is obtained, and further the inverse transformation is performed on the corrected image P (u, v) to obtain p (x, y) of the original image. Then, pixel interpolation is performed on the image p (x, y). Further, when performing two-stage image conversion such as projective conversion and enlargement conversion, a conversion matrix obtained by combining the two conversions is obtained, and the original image is converted at a time. In this case, an image can be obtained at a higher speed than when conversion is performed twice, and image deterioration can be reduced.

The rotation inverse transformation matrix Ar for obtaining the image before conversion from the image after conversion obtained by rotating the image before conversion by the angle θ about the X axis and the Y axis is expressed by the following equation (16). expressed.

An enlargement matrix Asc in the case of obtaining an image before conversion from an image after conversion obtained by enlarging the image before conversion by Sc magnification around the X axis and the Y axis is expressed by the following Expression 17. The

  Note that once an image is enlarged, a rounding error may be processed by adjusting or calculating an affine parameter. For this reason, before enlarging the image, it is necessary to restore the original affine parameters of the same magnification.

The movement matrix As for obtaining the image before conversion from the image after conversion obtained by moving the image before conversion in the X and Y directions by Tx and Ty, respectively, is expressed by the following equation (18). Is done.

The projection effect matrix Ap in the case of acquiring the image before conversion from the image after conversion obtained by inclining the image before correction in the X and Y directions by α and β, respectively, is given by expressed.

When performing two-stage inverse transformation, the inverse transformation matrix A is expressed by the following equation (20).

  In this embodiment, the projection effect parameters α and β are set in increments of 0.1, and the angle correction parameter θ is set in increments of 1 degree. However, the adjustment range of each parameter can be determined while checking the actual correction effect.

Image conversion adjustment processing executed based on this concept will be described based on the flowchart shown in FIG.
The CPU 30 sets the center coordinates Xc and Yc of the image (step S101).
The CPU 30 determines whether or not the projection correction key is pressed based on the operation information transmitted from the operation unit 26 (step 102).
If it is determined that the projection conversion key has been pressed (Yes in step S102), the CPU 30 has pressed one of the upper reduction key 111, the lower reduction key 112, the right reduction key 113, and the left reduction key 114 as the projection conversion key. The key type is determined (steps S103 to S106).

  When determining that the pressed projective transformation key is the up-reduction key 111, the CPU 30 assigns α = 0.1 and β = 0 to the projective effect matrix Ap shown in Equation 19 to obtain an inverse transformation matrix Ai = Ap. Is acquired (step S103).

  If it is determined that the pressed projective transformation key is the lower reduction key 112, the CPU 30 assigns α = −0.1 and β = 0 to the projective effect matrix Ap shown in Equation 19 to obtain an inverse transformation matrix Ai = Ap is acquired (step S104).

  If it is determined that the pressed projective transformation key is the right reduction key 113, the CPU 30 substitutes α = 0 and β = 0.1 for the projection effect matrix Ap shown in Equation 19 to obtain an inverse transformation matrix Ai = Ap. Is acquired (step S105).

  When determining that the pressed projective transformation key is the left reduction key 114, the CPU 30 assigns α = 0 and β = −0.1 to the projective effect matrix Ap shown in Equation 19, respectively, and the inverse transformation matrix Ai = Ap is acquired (step S106).

If it is determined that the projection conversion key has not been pressed (No in step 102), the CPU 30 determines whether or not the rotation correction key has been pressed (step S107).
If it is determined that the rotation correction key has been pressed (Yes in step S107), the CPU 30 determines the type of the rotation correction key that has been pressed (steps S108 and S109).

  When determining that the pressed rotation correction key is the right rotation key 115, the CPU 30 substitutes θ = −1 into the rotation inverse transformation matrix Ar shown in Expression 16 to obtain the inverse transformation matrix Ai = Ar. (Step 108).

  If it is determined that the pressed rotation correction key is the left rotation key 116, the CPU 30 substitutes θ = 1 for the rotation inverse transformation matrix Ar shown in Equation 16 to obtain the inverse transformation matrix Ai = Ar ( Step 109).

  When it is determined that the projective transformation key or the rotation correction key has been pressed and the inverse transformation matrix Ai is set (steps S103 to S106, S108, S109), the CPU 30 obtains the inverse transformation matrix A according to Equation 20 ( Step S110).

The CPU 30 supplies the obtained conversion matrix A to the image processing device 25, and controls the image processing device 25 to perform image conversion based on the inverse conversion matrix A. The image processing device 25 performs image conversion by affine transformation based on the supplied inverse transformation matrix A (step S111), and ends this image transformation adjustment processing.
On the other hand, if it is determined that the rotation correction key has not been pressed (No in step S107), the CPU 30 ends the image conversion adjustment process as it is.

  By performing image conversion by the above-described method using the affine parameters obtained in this manner, the image can be further manually adjusted with respect to the corrected image.

  For example, when adjusting an image as shown in FIG. 22, since the image is distorted leftward, first, when the right rotation key 115 is pressed, the image processing device 25 rotates the image to the right. If the right rotation key 115 is pressed as it is and the character string is correctly displayed and the right rotation key 115 stops being pressed, the image processing device 25 stops the right rotation of the image.

  Next, since the left side of the image is larger than the right side, when the left reduction key 114 is pressed, the image processing device 25 adjusts the left and right sides of the image. If the left reduction key 114 is pressed and the left and right balances are aligned and the pressing of the left reduction key 114 stops, the image processing device 25 stops the left reduction process.

  In this method, an image can be obtained by performing only one image conversion on the original image, so that an image with better image quality than rotating and projecting the image once subjected to projection correction can be obtained. Can do.

(3) Extraction of image effect correction parameters relating to luminance or color difference and image effect processing Next, processing for extracting image effect correction parameters from the image thus obtained, and image effects performed using these parameters Processing will be described. The image effect process is a process for obtaining a clearer image.

  The image thus obtained is an image obtained by cutting out the white board 2 or the document as a result. When performing image effects such as histogram correction, more effective parameters are expected to be obtained by obtaining parameters from the cut out image than by obtaining correction parameters from the original image and performing correction.

  In the present embodiment, a histogram is created from luminance (Y) from the corrected image data, and image effect processing is performed according to the histogram.

  The image effect correction parameters are variables necessary for image effect processing such as the maximum value, minimum value, and peak value of the luminance histogram.

  In order to extract the image effect correction parameter, it is necessary to generate a luminance histogram. First, processing necessary for extracting image effect correction parameters will be described.

  The luminance histogram indicates the distribution of luminance values (Y) existing in the image, and is generated by counting the number of pixels for each luminance value. FIG. 25 shows an example of a luminance histogram. In FIG. 25, the horizontal axis indicates the luminance value (Y), and the vertical axis indicates the number of pixels. In order to correct the image effect, it is necessary to obtain a maximum value (Ymax), a minimum value (Ymin), and a peak value (Ypeak) as image effect correction parameters.

  The maximum value counts the number of pixels for each luminance value, and is a value indicating the maximum luminance among luminance values having a predetermined number or more set in advance, and the minimum value is a predetermined number or more This is a value indicating the minimum luminance among the luminance values having the counted value. The peak value is a luminance value that maximizes the count value. The peak value is considered to indicate the luminance value of the background color of the photographing object.

  In order to obtain an image with excellent visibility by correcting the image effect, the correction effect differs depending on the background color of the object to be imaged. Therefore, it is necessary to change the image effect correction method depending on the background color of the object to be imaged. is there. For this reason, it is necessary to determine the background color of the object to be photographed. The background color of the object to be photographed is determined from the peak values of the luminance histogram and the color difference histogram.

  Here, the background color of the photographing object is classified into three. The first is a case where the background color is white, such as a whiteboard or notebook. The second is a case where the background color is black, such as a blackboard. The third is a case where the background color is other than white or black, such as a magazine or a pamphlet.

Specifically, the background color of the shooting target is determined according to the following discriminant.
(2-a) White Determination Condition The white determination condition is expressed by the following Expression 21, and when the condition shown in Expression 21 is satisfied, the background color of the photographing target is determined to be white (W).

(2-b) Black Determination Condition The black determination condition is represented by the following Expression 22, and when the condition shown in Expression 22 is satisfied, the background color of the shooting target is determined to be black (b).

  If the conditions shown in Equations 21 and 22 are not satisfied, the background color of the object to be photographed is determined to be color (C). For example, the color threshold is set to 50 and 128, the white determination threshold is set to 128, and the black determination threshold is set to 50, for example.

Based on such a concept, the image processing device 25 executes image effect correction parameter extraction processing according to the flowchart shown in FIG.
The image processing device 25 counts the number of pixels having each luminance (Y) value, and generates a luminance histogram as shown in FIG. 25 (step S121).

  The image processing device 25 acquires the maximum value (Ymax), the minimum value (Ymin), and the peak value (Ypeak) of the luminance from the generated luminance histogram (step S122).

The image processing device 25 discriminates the background color of the object to be photographed from the peak value (Ypeak) of the luminance histogram according to the judgment condition formulas shown in Equations 21 and 22 (step S123).
The image processing device 25 stores the image effect correction parameter and the background color data of the shooting target in the memory 23 (step S124).

Next, the image processing device 25 performs image effect processing (step S24 in FIG. 5) using the image effect correction parameters extracted in this way.
As described above, in order to effectively perform the image effect processing, it is necessary to change the processing contents depending on the background color.

  When the background color is white, such as a whiteboard or notebook, luminance conversion as shown in FIG. When the background color is black, such as a blackboard, luminance conversion as shown in FIG. When the background color is other than white or black, such as a magazine or pamphlet, the conversion shown in FIG. 27C is performed. In FIGS. 27A, 27B, and 27C, the horizontal axis indicates the input value of the pixel value, and the vertical axis indicates the output value of the pixel value.

  When the background color is white, as shown in FIG. 27A, the inclination angle of the luminance conversion line is changed with the peak value as a boundary. The predetermined luminance value is set to 230, for example, and the input luminance peak value is raised to the luminance value 230. Then, the maximum value is raised to the maximum brightness. Therefore, the luminance conversion line is represented by two line segments as shown in FIG.

  When the background color is black, luminance conversion is performed so that the peak value becomes a certain luminance value (20) as shown in FIG. Also in this case, as shown in FIG. 27B, the luminance conversion line is represented by two line segments.

  When the background color is a color other than white or black, as shown in FIG. 27C, the minimum value or less and the maximum value or more are cut and expressed as one line segment as in the normal enlargement process. The luminance conversion line is set as follows.

  It should be noted that a conversion table of background luminance (Y) and output (Y ′) may be set in advance and stored in the memory card 31 as described above. According to the created conversion table, each output value is obtained from the input pixel value, and image effect processing is performed. In the image thus converted, bright pixels are brighter and dark pixels are darker, so that the luminance distribution is widened and the image is excellent in visibility.

Based on such a concept, the image processing device 25 executes image effect processing according to the flowchart shown in FIG.
The image processing device 25 reads the stored image effect correction parameters from the memory 23 (step S131).

The image processing device 25 determines whether or not the background is white (step S132).
When the background is determined to be white (Yes in step S132), the image processing device 25 performs luminance conversion as illustrated in FIG. 27A so that the background is whiter and the visibility is improved. The brightness histogram is adjusted (step S133).

  If it is determined that the background is not white (No in step S132), the image processing apparatus 25 determines whether the background is black (step S134).

  When it is determined that the background is black (Yes in step S134), when the background is black, the image processing apparatus 25 performs luminance conversion as illustrated in FIG. 27B and adjusts the luminance histogram (step). S135).

  If it is determined that the background is not black (No in step S134), the image processing apparatus 25 performs luminance conversion as shown in FIG. 27C and performs histogram adjustment according to the background color of the shooting target. (Step S136).

In the digital camera 1 of the present embodiment, the mode can be set not only to the shooting mode but also to the playback mode.
When the playback key of the operation unit 26 is pressed, the operation unit 26 transmits this operation information to the CPU 30.

When the CPU 30 determines that the mode is the playback mode based on the operation information, the CPU 30 executes the playback process according to the flowchart shown in FIG.
The CPU 30 selects one image selected by the user from the image files recorded in the memory card 31 (step S141).

The CPU 30 reads the selected image file from the memory card 31 and writes it in the memory 23 (step S142).
The CPU 30 creates a reduced image from the read image (step S143).

  The CPU 30 writes in the preview image storage area on the memory 23 (step S144). Thus, the reduced image is displayed from the display device 24 as in the photographing mode. The user can confirm the reproduced image by viewing this display.

If the user visually recognizes this display and presses the NO key to view another image, the operation unit 26 transmits this operation information to the CPU 30.
The CPU 30 determines whether or not the next image has been designated according to the operation information (step S145).

  When it is determined that the next image has been designated (Yes in step S145), the CPU 30 selects the next image (step S146) and executes steps S142 to S144 again.

If it is determined that the next image has not been designated (No in step S145), the CPU 30 determines whether or not the reproduction mode has ended (step S147).
If the reproduction key is not pressed, the CPU 30 determines again whether or not the next image has been designated (step S145).

  On the other hand, when the playback key is pressed, the CPU 30 receives this operation information from the operation unit 26 and determines that the playback mode has ended (Yes in step S147), and ends the playback process.

Next, writing parameters to the file header will be described.
When both the original image and the corrected image are recorded as JPEG format images, the file name of the original image is recorded in the header (option data) area of the image file. When it is desired to correct the corrected image again, if the corrected image file is corrected, an image with little deterioration can be generated by generating the corrected image file from the original image. This image editing may be performed in the digital camera 1. However, image editing can also be performed on a computer. If image editing is performed on a computer, it is expected that more advanced image correction can be performed.

  Further, by recording the parameters when image processing is performed in the header, the computer can easily create a corrected image again using the parameters for the original image. For this reason, it becomes easy for the user to make a change immediately from the previous corrected image.

As shown in FIG. 30, the header information includes the type of data (data name), the number of bytes of each data, and the contents thereof.
The CPU 30 sequentially stores the header information in the header information storage area, the image file name of the original image data, the image size of the corrected image, the affine parameters, and an input / output data set for creating a histogram table.

  The input / output data set is a set of data indicating the relationship between the input data and the output data, and is arranged in order from the smallest of the input data for each change point where the slope of the straight line indicating the input / output relationship changes. This is a set of output data.

If the input data at the change point is x1, x2, ..., xm in order from the smallest, and the corresponding output data is y1, y2, ..., ym, the data set is (x1, y1), ( x2, y2),..., (xm, ym). In this case, if the input data x is xi <x <xi + 1, the output data y is expressed by the following equation (23).

  For example, when the relationship between the input data and the output data is as shown in FIG. 27A, the CPU 30 determines (0, 0), (minimum value, 0), (peak value, constant luminance value ( = 230)), (maximum value, 255), and (255, 255) are stored in the memory card 31.

  Similarly, when the relationship between the input and output data is as shown in FIG. 27B, the CPU 30 determines (0, 0), (minimum value, 0), (peak value, constant luminance value (= 20). )), (Maximum value, 255), and (255, 255) are stored in the memory card 31.

  When the relationship between the input and output data is as shown in FIG. 27C, the CPU 30 determines (0, 0), (minimum value, 0), (maximum value, 255), (255, 255). Four data sets are stored in the memory card 31.

  By recording such a brightness value data set, various histograms can be described. The CPU 30 records such data on the memory card 31 simultaneously with the recording of the corrected image data (step S26 in FIG. 5).

Next, correction image editing processing will be described.
In some cases, after image data is stored once, it is desired to correct the image data again. In this case, as described above, it is preferable to collectively correct the original image data in that image deterioration is small.

When the user operates the control key to designate an image editing mode, the CPU 30 executes an image editing process according to the flowchart shown in FIG.
The CPU 30 selects a corrected image according to the operation information transmitted from the operation unit 26 (step S151).

The CPU 30 reads header information from the header information storage area of the memory card 31 (step S152).
The CPU 30 reads the original image (step S153).
The CPU 30 sends the read corrected image, header information, and original image to the image processing device 25, and controls the image processing device 25 to correct again (step S154).

  The image processing device 25 acquires a corrected image, header information, and an original image from the CPU 30, creates a projective conversion image (step S155), and performs image effect processing (step S156). The image processing device 25 sends the data of the created projective transformation image to the display device 24, and the display device 24 displays this image on the liquid crystal monitor 12.

  When the user operates the projection conversion key and the rotation correction key, the operation unit 26 transmits this operation information to the CPU 30. The CPU 30 acquires this operation information and controls the image processing apparatus 25 to perform the projection manual correction process.

The image processing device 25 performs a projection manual correction process according to the flowchart shown in FIG. 24 (step S157).
The image processing device 25 records the corrected image subjected to the projection manual correction process together with the header information on the memory card 31 (step S158), and ends the editing process.

  As described above, according to the present embodiment, when the image processing device 25 performs the projective transformation and the image effect processing, the corrected image data is recorded in the image file of the memory card 31, and the CPU 30 The image data name, the obtained projection parameter, and the image processing correction parameter are stored in the image file of the memory card 31. Therefore, when image editing is performed again, image editing can be easily performed and the image can be corrected again.

  Further, the image processing device 25 acquires a contour from the image of the shooting target (white plate 2), acquires the shape of the shooting target, obtains a projection parameter from the position of the quadrangle of the shooting target, and acquires a shooting parameter. Projective transformation of object images.

  Therefore, if the object to be photographed is a quadrangle, the image distortion can be automatically corrected easily. As a result, it is possible to obtain an image with high legibility that is equal to the image of the original object to be photographed.

  Further, even when the object to be photographed is a quadrangle whose aspect ratio is unknown, and an arbitrary object to be photographed is photographed from an arbitrary viewpoint / posture, the focal length of the digital camera 1 is used to distort the image. Can be corrected.

  In addition, since a reduced image having a low resolution used for preview or the like is used to obtain the projection parameters, the number of operations can be reduced, the operation processing can be performed efficiently, and the image processing can be performed at high speed.

  In addition, when a plurality of rectangles are extracted from the captured image, the largest outline rectangle is preferentially selected, and the rectangle is selected in order from the larger one by the NO key, the cursor key, or the like. Therefore, even if there are a plurality of candidates for the contour of the object to be photographed, the contour of the object to be photographed can be quickly acquired, and as a result, a highly legible image can be obtained by a simple method.

  In addition, since the image processing correction parameters are obtained from the distortion-corrected image, better image effect correction can be performed.

In carrying out the present invention, various forms are conceivable and the present invention is not limited to the above embodiment.
For example, in the above embodiment, the parameters used to create the corrected image are stored in the corrected image file. However, this parameter can also be stored in the original image file. If the parameters are saved in the original image file, the corrected image can be easily created from the original image without saving the corrected image.

  As shown in FIG. 32, the header information in this case includes only the image size, projection parameters, and image effect correction parameters.

Further, when performing the photographing process, the digital camera 1 immediately performs the projection parameter extraction process without storing the converted image. The contents of this processing are shown in the flowchart of FIG.
The image processing device 25 creates a projection corrected image (steps S161 to S169), performs image conversion adjustment, extraction of image effect correction parameters, and image effect processing (steps S170 to S171).

The CPU 30 creates header information shown in FIG. 32 (step S172).
The image processing device 25 compresses the original image data, and the CPU 30 records the created header information together with the compressed original image data in the memory card 31 (steps S173 and 174).

Also, the editing process is different from the process shown in FIG. That is, the editing process is performed not on the corrected image but on the original image. This processing content is shown in the flowchart of FIG.
The CPU 30 selects the original image and reads header information from the original image file (steps S181 and S182).

The image processing device 25 immediately creates a projection corrected image and performs image effect processing (step S183).
The CPU 30 and the image processing device 25 perform a projection manual correction process. After the correction is completed, the CPU 30 creates header information based on the changed projection correction parameter, and records the created header information in the original image file again. (Steps S184 to S187).

  In the above embodiment, the image correction is performed by the digital camera 1. However, image correction can also be performed by a computer. In this case, a computer is connected to the computer interface unit 27, and the computer executes image editing processing according to the flowchart shown in FIG. If the computer performs such image editing processing, the computer includes a mouse and the like, and it becomes easier to input operation information than the digital camera 1, so that the operability is improved. Further, since the display device of the computer is generally larger than the liquid crystal monitor 12 of the digital camera 1, the image can be visually confirmed in detail and image correction can be performed with high accuracy.

  In the above embodiment, a warning is given when a quadrangle cannot be acquired. However, instead of issuing a warning, it is also possible to display the captured image and allow the user to designate four rectangular points using a control key or the like. Then, the affine parameters can be obtained using the designated four points.

  In the above-described embodiment, the program has been described as being stored in advance in a memory or the like. However, a program for causing a computer to operate as the whole or a part of the apparatus or to execute the above-described processing is a flexible disk, a CD-ROM (Compact Disk Read-Only Memory), a DVD (Digital Versatile Disk), The information may be stored in a computer-readable recording medium such as an MO (Magneto Optical disk) and distributed, installed in another computer, operated as the above-described means, or the above-described steps may be executed.

  Furthermore, the program may be stored in a disk device or the like of a server device on the Internet, and may be downloaded onto a computer by being superimposed on a carrier wave, for example.

It is explanatory drawing which shows a state when image | photographing a white board with the digital camera which concerns on embodiment of this invention. It is a block diagram which shows the structure of the digital camera shown in FIG. It is explanatory drawing for demonstrating the function of the image processing apparatus shown in FIG. It is explanatory drawing for demonstrating each key with which the operation part shown in FIG. 2 is provided. It is a flowchart which shows the content of the imaging | photography process which a digital camera performs. It is explanatory drawing which shows the square of the clipping object of the image processing apparatus shown in FIG. It is explanatory drawing for demonstrating the basic concept of extraction of a projection parameter, and an affine transformation. It is a flowchart which shows the content of the projection parameter extraction process which the image processing apparatus shown in FIG. 2 performs. It is a flowchart which shows the content of the square outline extraction process which the image processing apparatus shown in FIG. 2 performs. It is explanatory drawing for demonstrating a reduced luminance image and an edge image. It is explanatory drawing for demonstrating the function of a Roberts filter. It is an image figure for demonstrating the principle of radon conversion. It is explanatory drawing for demonstrating the operation | movement which acquires the data of a polar coordinate system by radon-transforming the straight line of a X, Y coordinate system. It is a flowchart which shows the content of the peak point detection process from the data of the polar coordinate system which the image processing apparatus shown in FIG. 2 performs. It is explanatory drawing which shows the view which detects a square from the straight line which detected and extracted the peak point. It is a flowchart which shows the content of the selection process of the detected square which the image processing apparatus shown in FIG. 2 performs. It is a flowchart which shows the content of the process which calculates | requires an affine parameter from the square vertex performed by the image processing apparatus shown in FIG. It is explanatory drawing which shows the content of the warning when a square cannot be extracted. It is a flowchart which shows the content of the warning process which CPU shown in FIG. 2 performs. It is explanatory drawing for demonstrating the inverse transformation for obtaining the original image from the image after projective transformation. It is a flowchart which shows the content of the image conversion process by the affine transformation which the image processing apparatus shown in FIG. 2 performs. It is explanatory drawing which shows the example which was unable to correct | amend image distortion by projective transformation. It is explanatory drawing for demonstrating the correspondence with an original image, a projection conversion image, and the enlarged projection conversion image. 3 is a flowchart showing details of correction adjustment and image conversion processing executed by the image processing apparatus shown in FIG. 2. It is explanatory drawing which shows an example of a brightness | luminance histogram. 3 is a flowchart showing the contents of image effect correction parameter extraction processing executed by the image processing apparatus shown in FIG. 2. It is explanatory drawing which shows an image effect process, (a) shows the image effect process in case a background color is white, (b) shows the image effect process in case a background is black, (c), The image effect processing when the background is other than white or black is shown. It is a flowchart which shows the content of the image effect process which the image processing apparatus shown in FIG. 2 performs. It is a flowchart which shows the content of the reproduction | regeneration processing which CPU and image processing apparatus which are shown in FIG. 2 perform. It is explanatory drawing which shows the content of header information. 3 is a flowchart showing the content of a correction image editing process executed by the CPU and image processing apparatus shown in FIG. It is explanatory drawing which shows the content of the header information in the case of preserve | saving only an original image. FIG. 3 is a flowchart showing the contents of a photographing process executed by the CPU and image processing apparatus shown in FIG. 2 when only an original image is stored. FIG. 3 is a flowchart showing details of a correction image re-editing process executed by the CPU and the image processing apparatus shown in FIG. 2 when only the original image is stored.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Digital camera, 2 ... White board, 11 ... Shooting lens part, 12 ... Liquid crystal monitor, 13 ... Shutter button, 21 ... Optical lens apparatus, 22 ... Image sensor, 23 ... Memory, 24 ... Display device, 25 ... Image processing device, 30 ... CPU, 31 ... Memory card

Claims (9)

In a photographing device that photographs a photographing object by a photographing unit,
A photographing unit for photographing the photographing object;
An image processing unit that performs image processing on the photographed image so as to correct distortion of the photographed image of the photographing object obtained by photographing with the photographing unit;
The image processing unit
A shape acquisition unit that acquires the contour of the image from the image of the shooting target, and acquires the shape of the image from the acquired contour;
The shape of the image acquired by the shape acquisition unit and the actual shape of the object to be photographed are associated with each other by setting a three-dimensional coordinate system. A projection parameter obtaining unit for obtaining a projection parameter indicating a relationship with the object to be imaged,
An aspect ratio acquisition unit for obtaining an aspect ratio of the photographing object based on a focal length of a lens included in the photographing unit and a vertex position of an image of the photographing object;
Using the projection parameter obtained by the projection parameter acquisition unit and the aspect ratio obtained by the aspect ratio acquisition unit, an image conversion parameter for image conversion is obtained, and the image conversion parameter is used to calculate the image of the subject to be imaged. An image conversion unit that performs image conversion of the image;
A storage control unit that stores image conversion parameters used by the image conversion unit for image conversion of the image of the object to be captured in an image file of a storage unit;
An imaging apparatus characterized by that. The storage control unit stores the image data of the photographing object obtained by photographing with the photographing unit in the image file of the storage unit, and stores the image conversion parameter in the image file.
The imaging apparatus according to claim 1, wherein: The storage control unit stores the image converted by the image conversion unit in the image file of the storage unit.
The imaging apparatus according to claim 2, wherein: The storage unit is configured by an external storage medium.
The photographing apparatus according to any one of claims 1 to 3, wherein From the shape of the image acquired by the shape acquisition unit, an actual target part representing the content of the photographing object is determined, and an image cutout unit that cuts out the image of the determined real target part;
A correction parameter acquisition unit for acquiring an image effect correction parameter for correcting an image effect from the image of the real target part cut out by the image cutout unit;
An image effect processing unit that performs image effect processing of the image of the object to be photographed using the image effect correction parameter acquired by the correction parameter acquisition unit;
The storage control unit stores, in the image file of the storage unit, image effect correction parameters used by the image effect processing unit for image effect processing of the image of the shooting target.
The photographing apparatus according to claim 1, wherein the photographing apparatus is characterized in that: The image conversion unit reads the stored image effect correction parameters from the image file in the storage unit, and performs the image conversion using the read image effect correction parameters.
The imaging device according to claim 5, wherein: A display unit for displaying the image;
The photographing apparatus according to any one of claims 1 to 6, wherein the photographing apparatus includes: An image processing method of a photographing apparatus that performs image processing on an image of the photographing target so as to correct distortion of the image of the photographing target obtained by photographing the photographing target by a photographing unit. ,
Acquiring a contour of the image from the image of the photographing object, and acquiring a shape of the image from the acquired contour;
The acquired shape of the image and the shape of the actual shooting target are associated with each other by setting a three-dimensional coordinate system, and from the vertex position of the image of the shooting target, the image of the shooting target and the actual shooting target Obtaining a projection parameter indicating a relationship with the object to be imaged;
Obtaining an aspect ratio of the photographing object based on a focal length of a lens included in the photographing unit and a vertex position of an image of the photographing object;
Using the obtained projection parameters and aspect ratio to determine image conversion parameters for image conversion, and using the image conversion parameters to perform image conversion of the image of the object to be photographed;
Storing image conversion parameters used for the image conversion in an image file storing image data,
An image processing method for a photographing apparatus. In the computer of the photographing device that photographs the object to be photographed by the photographing unit,
A procedure for acquiring the contour of the image from the image of the shooting target obtained by shooting the shooting target, and acquiring the shape of the image of the shooting target from the acquired contour,
The acquired shape of the image and the shape of the actual shooting target are associated with each other by setting a three-dimensional coordinate system, and from the vertex position of the image of the shooting target, the image of the shooting target and the actual shooting target A procedure for obtaining a projection parameter indicating the relationship with the object to be photographed;
A procedure for obtaining an aspect ratio of the photographing object based on a focal length of a lens of the photographing unit and a vertex position of an image of the photographing object;
An image conversion parameter for image conversion is obtained using the obtained projection parameter and aspect ratio, and the image is imaged with respect to the image so as to correct the distortion of the image of the object to be photographed using the image conversion parameter. The steps to do the conversion,
A procedure for storing image conversion parameters used for the image conversion in an image file storing image data;
A program for running