JP4363154B2 - Imaging apparatus, image processing method thereof, and program - Google Patents

Description

  The present invention relates to a photographing apparatus, an image processing method thereof, and a program.

  With the development of digital cameras and the low price of storage-type memory, digital cameras are not only used for taking pictures of landscapes and people, but are also used for paper documents, business cards, and blackboards written at meetings. Applications are being considered in which the displayed images are photographed, and these images are digitally stored in a personal computer or the like for management.

  When photographing a document or a blackboard in this way, it is desirable to photograph the photographing object in front and vertically. However, it is difficult to shoot a document placed on the desk vertically and vertically. Also, when shooting a blackboard, it may be difficult to capture the blackboard from the front depending on the position of the photographer. Even if the image can be taken from the front, it may be preferable to avoid taking the image from the front for reasons such as reflection of light. Thus, when an object such as a document is photographed from an oblique direction, characters and the like are distorted obliquely or trapezoidally. Of course, these characters can be interpreted. However, even if it can be read, it is very tiring to read. Moreover, since these images have distortion, it is difficult to reuse these images.

  In order to solve such a problem, there is a digital camera provided with an image processing device that corrects image distortion due to a relative displacement between a document and a digital camera at the time of shooting (for example, see Patent Document 1).

  Such a digital camera includes an image processing device. The image processing device creates four upper, lower, left, and right partial images of the input image, creates an unequal side rectangle based on the inclination of each partial image, Rotational transformation is performed so that two opposite sides of the equilateral square are parallel to each other.

Next, the image processing apparatus obtains the horizontal tilt angle α, and then obtains the angle between a straight line perpendicular to two parallel sides and a vertical line as the skew rotation angle β. The image processing apparatus obtains the vertical tilt angle γ by rotating in the vertical direction so that the remaining two opposite sides are parallel to each other. The image processing apparatus corrects image distortion by performing geometric transformation on the input image using the horizontal tilt angle α, the skew rotation angle β, and the vertical tilt angle γ.
Japanese Unexamined Patent Publication No. 2000-341501 (pages 6 to 14, FIG. 1 etc.)

  However, in such a conventional digital camera, many rotation conversion processes must be performed, the number of operations increases, and the process takes time. In addition, an expensive arithmetic unit is required to reduce the processing time.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a photographing apparatus capable of speeding up correction of an image of a photographing target, an image processing method thereof, and a program.

In order to achieve this object, an imaging device according to the first aspect of the present invention provides:
In a photographing device that photographs a photographing object by a photographing unit,
An image processing unit that performs image processing on the photographed image so as to correct distortion of the image of the photographing object obtained by photographing;
The image processing unit
A reduced image with a reduced amount of data is generated by selecting pixels of the image of the photographing object, distortion correction of the image of the photographing object is performed on the selected pixels, and the reduced image is generated from the generated reduced image. A shape acquisition unit that generates an edge image of the image of the shooting target and acquires the shape of the image of the shooting target from the generated edge image;
The shape of the image acquired by the shape acquisition unit and the actual shape of the object to be photographed are associated with each other by setting a three-dimensional coordinate system. A projection parameter obtaining unit for obtaining a projection parameter indicating a relationship with the object to be imaged,
An aspect ratio acquisition unit for obtaining an aspect ratio of the photographing object based on a focal length of a lens included in the photographing unit and a vertex position of an image of the photographing object;
An image conversion unit that performs image conversion of the image of the object to be photographed using the projection parameter obtained by the projection parameter acquisition unit and the aspect ratio obtained by the aspect ratio acquisition unit.

The shape acquisition unit
An edge that selects a pixel that generates a reduced image of the image of the shooting object, and performs an edge correction of the image of the shooting object by performing distortion correction on the image of the shooting object using the selected pixel as an investigation target. An image generator;
A straight line acquisition unit that acquires a straight line forming the edge image from the edge image generated by the edge image generation unit;
A shape candidate generation unit that generates a shape candidate of the image of the object to be photographed by the straight line acquired by the straight line acquisition unit;
A shape selection unit that selects a shape of the image of the shooting target from a shape candidate of the image of the shooting target generated by the shape candidate generation unit.

  The straight line acquisition unit obtains the length of the straight line obtained from the edge image generated by the edge image generation unit, and based on the obtained straight line length, forms a straight line that forms the shape of the image of the photographing object. You may select and acquire.

  The straight line acquisition unit obtains an angle of a straight line obtained from the edge image generated by the edge image generation unit, and selects and obtains a straight line that forms the shape of the image of the photographing object based on the obtained angle. May be.

  The shape selection unit selects and selects the image of the object to be imaged generated by the shape candidate generation unit that has the largest image area size surrounded by the shape. May be acquired as the shape of the image of the object to be photographed.

  You may provide the memory | storage part which memorize | stores the image data of the said imaging | photography target object.

An image processing method of the photographing apparatus according to the second aspect of the present invention is as follows.
An image processing method of a photographing apparatus that performs image processing on an image of the photographing target so as to correct distortion of the image of the photographing target obtained by photographing the photographing target by a photographing unit. ,
Generating a reduced image with a reduced amount of data by selecting pixels of the image of the shooting object, and correcting distortion of the image of the shooting object for the selected pixels;
Generating an edge image of the image of the object to be photographed from the generated reduced image;
Obtaining the shape of the image of the object to be photographed from the generated edge image;
The shape of the object to be photographed and the shape of the image of the object to be photographed acquired by the shape acquisition unit are associated with each other by setting a three-dimensional coordinate system, and from the vertex position of the image of the object to be photographed, Obtaining a projection parameter indicating a relationship between an image and the object to be photographed;
Obtaining an aspect ratio of the photographing object based on a focal length of a lens included in the photographing unit and a vertex position of an image of the photographing object;
Performing image conversion of the image of the object to be photographed using the obtained projection parameters and aspect ratio.

The program according to the third aspect of the present invention is:
In the computer of the photographing device that photographs the object to be photographed by the photographing unit,
A procedure for generating a reduced image with a reduced amount of data by selecting pixels of the image of the shooting object, and correcting distortion of the image of the shooting object for the selected pixel;
A procedure for generating an edge image of the image of the object to be photographed from the generated reduced image;
A procedure for acquiring the shape of the image of the photographing object from the generated edge image,
The shape of the object to be photographed and the shape of the image of the object to be photographed acquired by the shape acquisition unit are associated with each other by setting a three-dimensional coordinate system, and from the vertex position of the image of the object to be photographed, A procedure for obtaining a projection parameter indicating a relationship between an image and the object to be imaged;
A procedure for obtaining an aspect ratio of the photographing object based on a focal length of a lens of the photographing unit and a vertex position of an image of the photographing object;
A procedure for performing image conversion of the image of the object to be photographed using the obtained projection parameters and aspect ratio;
Is to execute.

  According to the present invention, it is possible to speed up the process of correcting an image of a photographing object.

Hereinafter, a photographing apparatus according to an embodiment of the present invention will be described with reference to the drawings.
In the embodiment, the photographing apparatus is described as a digital camera.

A configuration of a digital camera 1 according to the present embodiment is shown in FIG.
The digital camera 1 according to the present embodiment captures characters such as a white board 2, a figure, a photograph, and the like as an object to be photographed, detects and corrects image distortion from an image obtained by photographing, and is as if from the front. It generates an image as if it was taken. The digital camera 1 includes a photographic lens unit 11, a liquid crystal monitor 12, and a shutter button 13.

  The photographic lens unit 11 includes a lens that collects light and the like, and collects light from characters such as the white plate 2, drawings, and photographs.

The liquid crystal monitor 12 is for projecting an image taken in through the photographing lens unit 11.
The shutter button 13 is pressed when shooting a shooting target.

  As shown in FIG. 2, the digital camera 1 includes an optical lens device 21, an image sensor 22, a memory 23, a display device 24, an image processing device 25, an operation unit 26, a computer interface unit 27, An external storage IO device 28 and a program code storage device 29 are provided.

  The optical lens device 21 includes a photographic lens unit 11 and a driving unit thereof, and forms an image on the image sensor 22 by condensing light from characters, drawings, photographs, and the like on the white plate 2. .

  The image sensor 22 is for capturing a formed image as digitized image data, and is constituted by a CCD or the like. If the image sensor 22 is controlled by the CPU 30 and the shutter button 13 is not pressed, digital image data having a low preview resolution is generated, and the image data is periodically stored in the memory 23 at intervals of about 30 sheets per second. To send. Further, when the shutter button 13 is pressed, the image sensor 22 generates image data with high resolution and sends the generated image data to the memory 23.

  The memory 23 temporarily stores low-resolution preview images from the image sensor 22, high-resolution image data, original image data processed by the image processing device 25, and processed image data. The memory 23 sends the temporarily stored image data to the display device 24 or the image processing device 25.

  The display device 24 includes the liquid crystal monitor 12 and displays the image on the liquid crystal monitor 12. The display device 24 displays a low-resolution preview image or a high-resolution image temporarily stored in the memory 23 on the liquid crystal monitor 12.

  The image processing device 25 is for performing image processing such as image data compression, image distortion correction, and image effect processing on the image data temporarily stored in the memory 23.

  As shown in FIG. 3A, when the digital camera 1 shoots characters on the white plate 2 from the left direction and the right direction, the liquid crystal monitor 12 displays the characters shown in FIGS. 3B and 3C. As shown, the white plate 2 and images such as letters, diagrams, and photographs are distorted and displayed. The image processing device 25 performs image processing on the images as shown in FIGS. 3B and 3C to generate an image taken from the front as shown in FIG. 3D. To do.

  In order to correct image distortion, the image processing device 25 cuts a square from the distorted image, and performs projective transformation of the captured image into the cut quadrangle.

More specifically, the image processing device 25 is controlled by the CPU 30 and mainly performs the following processing.
(1) Extraction of affine parameters from photographed image (2) Image conversion using extracted affine parameters (3) Adjustment processing of image conversion (4) Extraction of image effect correction parameters relating to luminance or color difference and image effect processing The processing contents of will be described later.

  The operation unit 26 includes switches and keys for controlling the document projection function. When the user presses these keys and switches, the operation unit 26 responds and transmits the operation information at this time to the CPU 30.

  As shown in FIG. 4, the operation unit 26 includes an upper reduction key 111, a lower reduction key 112, a right reduction key 113, a left reduction key 114, a right reduction key 115, and a left reduction key 116. .

  The upper reduction key 111, the lower reduction key 112, the right reduction key 113, and the left reduction key 114 are projective conversion keys for performing projective conversion. The upper reduction key 111 is a key that is pressed when the upper part of the image is compared with the lower part around the X axis and the upper part is large and the upper part is rotated downward toward the paper surface.

  The lower reduction key 112 is a key that is pressed when the upper part and the lower part of the image are compared around the X axis, and the lower part is rotated downward toward the paper surface when the lower part is large.

  The right reduction key 113 and the left reduction key 114 are keys that are pressed when adjusting left and right distortion around the Y-axis side, and the right reduction key 113 is a key that is pressed when the right is large, and left The reduction key 114 is a key that is pressed when the left side is large.

  The right rotation key 115 and the left rotation key 116 are rotation correction keys for adjusting the rotation of the image. The right rotation key 115 is a key that is pressed when the image is rotated to the right, and the left rotation key 116 is a key that is pressed when the image is rotated to the left.

  In addition, the operation unit 26 includes a photographing key, a reproduction key, a cursor key, a control key, and the like (not shown). The shooting key is a key for selecting a shooting mode when shooting a shooting object. The reproduction key is a key for selecting a reproduction mode for reproducing an image to be photographed obtained by photographing. The control key is a key having functions such as a YES key for confirming the operation, a NO key for canceling the operation, and an editing key for performing editing.

  The computer interface unit 27 operates as a USB store registration class driver when the digital camera 1 is connected to a computer (not shown). Thus, when the computer is connected to the digital camera 1, the computer handles the memory card 31 as an external storage device of the computer.

  The external storage IO device 28 inputs and outputs image data and the like with the memory card 31. The memory card 31 stores image data and the like supplied from the external storage IO device 28.

  The program code storage device 29 is for storing a program executed by the CPU 30, and is configured by a ROM or the like.

  The CPU 30 controls the entire system according to a program stored in the program code storage device 29. The memory 23 is also used as a work memory for the CPU 30.

  When operation information is transmitted from the operation unit 26 by pressing a switch or key of the operation unit 26, the CPU 30 performs image sensor 22, memory 23, display device 24, image processing device based on the operation information. 25 etc. are controlled.

  Specifically, when the operation information indicating that the photographing key is pressed is transmitted from the operation unit 26, the CPU 30 sets each unit to the photographing mode. The CPU 30 sets the image sensor 22 to the preview mode if the shutter button 13 is not pressed in the state set to the shooting mode, and reads the high-resolution shooting target image if the shutter button 13 is pressed. Set to. Further, when the operation information indicating that the reproduction key is pressed is transmitted, the CPU 30 sets each unit to the reproduction mode.

  Further, when the operation information indicating that the projection conversion key and the rotation correction key are pressed is transmitted from the operation unit 26, the CPU 30 transmits the operation information to the image processing device 25, and causes the image processing device 25 to operate. Control.

  Further, the CPU 30 records preview image and high-resolution image data on the memory card 31 via the external storage IO device 28, and reads the recorded image data from the memory card 31. For example, the CPU 30 records image data compressed in the JPEG format in the memory card 31.

  When the image data is temporarily stored in the memory 23, the CPU 30 records the preview image and the high-resolution image data in different storage areas. Further, the CPU 30 records the image data separately in the image file in the memory card 31, and also records the header information regarding the image data in the header information storage area of the image file.

Next, the operation of the digital camera 1 according to this embodiment will be described.
When the user turns on (turns on) the digital camera 1, the CPU 30 acquires program data stored in the program code storage device 29. When the user presses the shooting button, the operation unit 26 transmits this operation information to the CPU 30. The CPU 30 receives this operation information, and the CPU 30, the image processing device 25, etc. execute the photographing process according to the flowchart shown in FIG.

The CPU 30 sets the image sensor 22 to the preview mode (step S11).
The CPU 30 determines whether or not the shutter button 13 has been pressed based on the operation information transmitted from the operation unit 26 (step S12).

  If it is determined that the shutter button 13 has been pressed (Yes in step S12), the CPU 30 controls the image sensor 22 by switching the image sensor 22 from the preview mode to the high resolution mode (step S13).

  The CPU 30 records the high-resolution photographic target image data generated by the image sensor 22 in a storage area different from the preview image on the memory 23 (step S14).

The CPU 30 determines whether the reading of the image data has been completed (step S15).
If it is determined that the reading has not been completed (No in step S15), the CPU 30 continues to control the image sensor 22 to read the image data.

  If it is determined that all the image data has been read and the image transfer has been completed (Yes in step S15), the CPU 30 generates a low-resolution preview image from the captured image (high-resolution image), and uses the preview image for the preview image in the memory 23. The preview image data is written in the storage area (step S16).

The CPU 30 controls the image processing device 25 so as to create compressed data, and the image processing device 25 creates compressed data (step S17).
The CPU 30 records this compressed data on the memory card 31 via the external storage IO device 28, and stores the compressed data created by the image processing device 25 (step S18).

  Next, the image processing device 25 extracts projection parameters for creating a front image from the captured image under the control of the CPU 30 (step S19).

The CPU 30 and the image processing device 25 determine whether or not the projection parameters have been extracted (step S20).
If it is determined that the extraction has been completed (Yes in step S20), the image processing device 25 creates a projected transformation image based on the extracted projection parameters (step S21).

  When the projection conversion key and the rotation correction key of the operation unit 26 are pressed, the operation unit 26 transmits this operation information to the CPU 30. The CPU 30 transmits operation information from the operation unit 26 to the image processing device 25, and the image processing device 25 performs image conversion adjustment processing in accordance with the transmitted operation information (step S22).

  The image processing device 25 extracts image effect correction parameters (step S23) and performs image effect processing (step S24).

  The image processing device 25 performs compression processing on the image data that has been subjected to the image effect processing, and creates compressed data (step S25).

  The image processing device 25 records the created compressed data on the memory card 31 (step S26).

  On the other hand, when it is determined that the projection parameters could not be extracted (No in step S20), the CPU 30 performs a warning process (step S27).

  The CPU 30 and the like end the photographing process in this way. Note that the CPU 30 or the like repeatedly executes this photographing process unless the user operates the key.

Next, image processing performed by the image processing device 25 will be described.
First, the basic concept (implementation method) of affine transformation used by the image processing apparatus 25 for image processing will be described.

Affine transformation is widely applied to image spatial transformation. In this embodiment, projective transformation is performed using two-dimensional affine transformation without using three-dimensional camera parameters. This is because the coordinates (u, v) before conversion are related to each other by the following equation (1) by conversion such as movement, enlargement / reduction, and rotation. Projective transformation can also be performed by this affine transformation.
The final coordinates (x, y) are calculated by the following formula 2.
Expression 2 is an expression for projective transformation, and the coordinates x and y are reduced toward 0 according to the value of z ′. That is, the parameter included in z ′ affects the projection. This parameter is a13, a23, a33. Since other parameters can be normalized by the parameter a33, a33 may be 1.

  FIG. 6 shows the coordinates of each vertex of a rectangular captured image. The relationship between the quadrangle photographed with the digital camera 1 and the actual photographing object (white plate 2) will be described with reference to FIG.

In FIG. 7, the UVW coordinate system is a three-dimensional coordinate system of an image obtained by photographing with the digital camera 1. The A (Au, Av, Aw) vector and the B (Bu, Bv, Bw) vector represent the object to be imaged as vectors in the three-dimensional coordinate system UV-W.
The S (Su, Sv, Sw) vector indicates the distance between the origin of the three-dimensional coordinate system UV-W and the object to be imaged.

The projection screen shown in FIG. 7 is for projecting an image of the object to be photographed.
If the coordinate system on the projection screen is x, y, the image projected on the projection screen may be considered as the image photographed by the digital camera 1. It is assumed that the projection screen is positioned vertically at a distance f from the W axis. An arbitrary point P (u, v, w) of the object to be photographed is connected to the origin by a straight line, and there is a point that intersects the straight line and the projection screen, and the XY coordinate of the intersection is p (x, y ). At this time, the coordinate p is expressed by the following equation 3 by projective transformation.
From Equation 3, the relationship shown in the following Equation 4 is obtained from the relationship between the points P0, P1, P2, P3 and the projection points p0, p1, p2, p3 on the projection screen as shown in FIG.
At this time, the projection coefficients α and β are expressed by the following equation (5).

Next, projective transformation will be described.
An arbitrary point P on the object to be imaged is expressed by the following equation 6 using S, A, and B vectors.
When the relational expression of Expression 4 is substituted into Expression 6, the coordinates x and y are expressed by the following Expression 7.

When this relationship is applied to the affine transformation formula, the coordinates (x ′, y ′, z ′) are expressed by the following equation (8).

  By substituting m and n into Equation 8, the corresponding point (x, y) of the captured image is obtained. Since the corresponding point (x, y) is not necessarily an integer value, the pixel value may be obtained using an image interpolation method or the like.

For m and n, an image size (0 ≦ u <umax, 0 ≦ v <vmax) for outputting the corrected image p (u, v) is given in advance, and a method of adjusting the image according to the image size is considered. It is done. According to this method, m and n are expressed by the following equation (9).
However, the aspect ratio of the corrected image to be created does not match the aspect ratio of the shooting target. Here, the relationship between the corrected image p (u, v) and the values of m and n is expressed by the following Expression 10 from Expression 4.

If the focal length f of the lens, which is a camera parameter, is known, the aspect ratio k can be obtained according to Equation 10. Therefore, if the image size of the corrected image p (u, v) is (0 ≦ u <umax, 0 ≦ v <vmax), m and n are obtained according to the following equation 11 to obtain the same as the object to be imaged. An aspect ratio k can be obtained.
If the camera has a fixed focus, the value of the focal length f of the lens can be obtained in advance. When a zoom lens or the like is present, the value of the focal length f of the lens changes depending on the zoom magnification of the lens. Therefore, a table indicating the relationship between the zoom magnification and the focal length f of the lens is created and stored in advance. Then, the focal length f can be read based on the zoom magnification, and projective transformation can be performed according to Equations 10 and 11.

In order to perform such affine transformation, the image processing device 25 first extracts projection parameters from the image of the photographed subject (step S19 in FIG. 5).
Projection parameter extraction processing executed by the image processing device 25 will be described with reference to the flowchart shown in FIG.

  The image processing device 25 extracts four corner coordinate points (rectangular contours) of the image of the imaged object from the imaged object (step S31). The image processing device 25 performs the process shown in the flowchart of FIG. 9 and extracts a quadrilateral outline.

That is, the image processing device 25 generates a reduced luminance image from the input image in order to reduce the number of operations for image processing (step S41).
The image processing device 25 generates an image of only a straight line from the generated reduced luminance image, and sets this as an edge image of the photographing object (step S42).
The image processing device 25 adds distortion correction to the edge image (step S43).

The image processing device 25 detects a straight line parameter included in the edge image from the edge image of the photographing object (step S44).
The image processing device 25 creates a quadrangle that is a candidate for forming the contour of the object to be photographed from the detected straight line parameters (step S45).

The image processing device 25 generates the candidate rectangles in this way, and assigns priorities to the generated rectangles (step S32 in FIG. 8).
The image processing device 25 selects a rectangle according to the priority order, and determines whether or not the selected rectangle has been extracted (step S33).

  If it is determined that the quadrangle could not be extracted (No in step S33), the CPU 30 ends the projection parameter extraction process.

  On the other hand, if it is determined that the quadrangle has been extracted (Yes in step S33), the CPU 30 acquires the extracted quadrangle from the image processing device 25, sends it to the display device 24, and displays the quadrangle preview image on the liquid crystal monitor 12. It is displayed (step S34).

  The CPU 30 determines whether the YES key or the NO key has been pressed based on the operation information transmitted from the operation unit 26 (step S35).

  If the CPU 30 determines that the NO key has been pressed (No in step S35), the image processing device 25 designates the next candidate rectangle (step S36).

  On the other hand, if it is determined that the YES key has been pressed (Yes in step S35), the CPU 30 calculates affine parameters from the extracted quadrangular vertices (step S37).

Next, the projection parameter extraction process will be described more specifically.
An example of the reduced luminance image generated by the image processing device 25 in step S41 is shown in FIG. The image processing device 25 generates an edge image as shown in FIG. 10B from such a reduced luminance image using an edge detection filter called a Roberts filter (step S42). The Roberts filter is a filter that detects an edge of an image by weighting two 4-neighboring pixels, obtaining two filters Δ1 and Δ2, and averaging them.

11A shows the coefficient of the filter Δ1, and FIG. 11B shows the coefficient of the filter Δ2. When the coefficients of the two filters Δ1 and Δ2 are applied to the pixel value f (x, y) of a certain coordinate (x, y), the converted pixel value g (x, y) Represented by

  In the edge image generated in step S42, a straight line may not be extracted due to distortion. Therefore, the image processing device 25 performs distortion correction on the edge image (step S43). The reason for performing distortion correction not on the original image but on the edge image is to increase the processing speed. The distortion correction calculation is to calculate for each pixel which pixel in the original image becomes the pixel to be examined after distortion correction. At that time, in the case of an edge image, only the coordinates skipped at a constant interval are investigated, and processing can be performed at a high speed because there are few pixels.

  The edge image shown in FIG. 10B includes straight line parameters. The image processing device 25 performs a Hough transform from the edge image to detect a straight line parameter (processing in step S44).

The Hough transform is a transform that votes points constituting a straight line on the XY plane to the ρ-θ plane defined by the following equation (13).

  When θ is changed from 0 to 360 ° in the coordinates (x, y) of each point, the same straight line is represented by one point on the ρ-θ plane. For this reason, it is possible to determine a ρ-θ coordinate having a large number of votes as a straight line. At that time, the number of votes is the number of pixels on a straight line, which corresponds to the length of the straight line. Therefore, ρ−θ having an extremely small number of votes, that is, a short straight line can be excluded from the straight line candidates.

  Thus, in the method using the Hough transform, the processing speed decreases as the number of points or angles to be investigated increases. In order to avoid this decrease in processing speed, the edge image is reduced by processing the coordinates of the investigation target at a constant interval in both the X and Y axis directions at the time of edge detection. As a result, the number of survey targets can be reduced.

Furthermore, the investigation angle can be reduced in another way.
That is, when considering the coordinate system with the center of the image as the origin in the edge image to be examined, ρ also takes a negative value, so it is sufficient to measure the angle 0 ≦ θ <180 ° and the remaining 180 °. ≦ θ <360 ° can be expressed by taking a negative value of ρ.

However, when a square is actually captured, each side exists vertically and horizontally when the square is at the center of the image. Therefore, by investigating in the range of 0 ° ≦ θ <180 °, as shown in FIGS. 12A and 12B, the upper and lower sides and the left and right sides are within the range represented by the following formula 14, respectively. The measurement is more effective.

  As is clear from Equation 14, cos θ of 45 ° ≦ θ <135 °, sin θ of 135 ° ≦ θ <225 °, sin θ of 45 ° ≦ θ <135 °, and cos θ of 135 ° ≦ θ <225 ° Since the minus numbers correspond to each other, the sin table and the cos table can be shared.

  In addition, a straight line detected at 45 ° ≦ θ <135 ° in FIG. 13A is indicated by a thick line, and a straight line detected at 135 ° ≦ θ <225 ° is indicated by a thick line in FIG. Depending on whether is a positive value or a negative value, it is possible to specify the upper and lower sides or the left and right sides. Therefore, it is possible to select the sides constituting the central square more efficiently.

  Based on such a concept, the image processing device 25 detects a straight line and identifies a quadrangle. Note that this quadrangle specifying process is different between when shooting an object to be shot from directly above as in document shooting and when shooting with the digital camera 1 tilted with respect to the object to be shot. First, processing in the case of photographing an object to be photographed from directly above will be described according to the flowchart shown in FIG.

  The image processing device 25 uses the Hough transform to vote from the XY plane to the ρ-θ plane (step S51). Then, the image processing apparatus 25 acquires a plurality of coordinates having a large number of votes in 45 ° ≦ θ <135 ° as candidates for straight lines forming the upper and lower sides (step S52). Similarly, the image processing apparatus 25 acquires a plurality of coordinates having a large number of votes when 135 ° ≦ θ <225 ° as candidates for straight lines forming the left and right sides (step S53).

  The image processing apparatus 25 selects an X-axis direction candidate and a Y-axis direction candidate that have different values of ρ and the same θ (step S54). This is because, when the object to be imaged is photographed from directly above, the object to be imaged does not have to be greatly distorted, and the opposite sides are parallel and can be regarded as perpendicular. Furthermore, if there are two parallel straight lines in different directions perpendicular to each other, a quadrangle (rectangle) is specified as shown in FIG.

  The image processing apparatus 25 selects a combination having a ratio close to the aspect ratio of the existing paper from the X-axis direction candidates and the Y-axis direction candidates (step S55). Specifically, first, the image processing device 25 replaces the straight line expressed by Equation 14 with y = ax + b and calculates orthogonal xy coordinates. This reveals the distance between the two points. Further, since which side is already specified in the x-axis direction and which side is in the y-axis direction, the aspect ratio of the subject to be imaged can be easily determined. By executing the above processing, the image processing device 25 identifies a quadrangle that matches the paper size (vertical / horizontal ratio) as a photographing quadrangle.

  Next, processing when the digital camera 1 is tilted with respect to the object to be imaged will be described with reference to FIG. Note that the processing from step S61 to S63 is the same as the processing from step S51 to S53 in the case where the object to be photographed is photographed from directly above (FIG. 14), and therefore description thereof is omitted here.

  The image processing device 25 identifies candidates having the same absolute value | ρ | of ρ among candidates in the X-axis direction and candidates in the Y-axis direction, and sets priorities in descending order of the absolute values (steps). S64). This is because the opposite sides are not always parallel when the object to be photographed is photographed from an oblique direction. By specifying the top, bottom, left, and right sides to have a large absolute value of | ρ |, a straight line priority order for specifying a region having the maximum area as shown by a thick line in FIG. 15B is obtained. be able to.

  The image processing device 25 identifies the quadrangle that constitutes the above-described maximum area as a photographing quadrangle (step S65).

  In this way, using the coordinates (x0, y0), (x1, y1), (x2, y2), (x3, y3) of the four points of the quadrangle of the selected photographing object, According to Equation 8, affine parameters that are each element in the matrix shown in Equation 8 can be obtained.

(1) Extraction of Affine Parameter from Image to be Captured Based on such a concept, the image processing device 25 acquires an affine parameter from a rectangular vertex. This process will be described based on the flowchart shown in FIG.

  The image processing device 25 calculates projection coefficients α and β according to Equation 5 from the vertex coordinates (x0, y0), (x1, y1), (x2, y2), and (x3, y3) of the four points of the rectangle. (Step S71).

The image processing device 25 calculates the aspect ratio k of the object to be photographed according to Equation 10 (step S72).
The image processing device 25 designates the center point (uc, vc) of the image (step S73).
The image processing device 25 compares the maximum image size vmax / umax with the aspect ratio k expressed by Equation 10 (step S74).

  When vmax / umax ≦ k (No in step S74), the image processing apparatus 25 assumes that the maximum image size umax on the U-axis side (lateral) is larger than the image size of the object to be photographed, assuming that the aspect ratio k is not changed. Is also determined to be large. Then, the image processing device 25 obtains the values of m and n according to the condition (1) of Equation 11 so that the maximum image size on the V-axis side and the image size of the object to be photographed coincide (step S75).

  When vmax / umax> k (Yes in step S74), assuming that the aspect ratio k is not changed, the image processing apparatus 25 determines that the maximum image size vmax on the V-axis side (vertical) is larger than the image size of the object to be photographed. Is also determined to be large. Then, the image processing device 25 obtains the values of m and n according to the condition (2) of Equation 11 so that the maximum image size on the U-axis side matches the image size of the object to be photographed (step S76).

The image processing device 25 calculates the affine transformation from the calculated vertex coordinates (x0, y0), (x1, y1), (x2, y2), (x3, y3) of the four points of the quadrangle according to Equation 8 A matrix Af is obtained (step S77).
The image processing apparatus 25 acquires the affine parameter A using each element of the affine transformation matrix Af as the affine parameter A (step S78).

  Note that, as in the case where the quadrangle cannot be recognized (No in step S33 in FIG. 8), the image capturing condition or the like is inappropriate, and the projection parameter may not be obtained. In such a case, a warning message as shown in FIG. 18A is displayed on the liquid crystal monitor 12 to appropriately warn the photographer that the area could not be detected, and to the photographer, FIG. It is preferable to change the shooting condition to the camera shooting setting mode as shown in FIG. Furthermore, it may be preferable to warn again to change the shooting conditions.

  When the proper projection parameter cannot be obtained as described above, the CPU 30 performs a warning process according to the flowchart shown in FIG. 19 (step S27 in FIG. 5).

  The CPU 30 controls the display device 24 to display a warning text as shown in FIG. 18A on the liquid crystal monitor 12 (step S81).

  The CPU 30 determines which of the YES key and the NO key is pressed based on the operation information transmitted from the operation unit 26 (step S82).

  When it is determined that the NO key has been pressed (No in step S82), the CPU 30 ends the warning process.

  On the other hand, if it is determined that the YES key has been pressed (Yes in step S82), the CPU 30 switches to the shooting mode (step S83) and ends this warning process.

(2) Image Conversion Using Extracted Affine Parameters Next, an image processing method for creating a corrected image using the obtained affine parameters will be described.
First, when projective transformation or other affine transformation is performed using affine parameters, as shown in FIG. 20, the point p (x, y) of the original image is transformed by projective transformation using the transformation matrix Ap (after). Assume that it corresponds to the point P (u, v) of the image. In this case, it is better to obtain the point p (x, y) of the original image corresponding to the point P (u, v) of the converted image than to obtain the point P of the converted image corresponding to the point p of the original image. preferable.

Note that when obtaining the coordinates of the point P of the converted image, an interpolation method based on the bilinear method is used. In the bilinear interpolation method, the coordinate point of the other image (transformed image) corresponding to the coordinate point of one image (original image) is searched, and the (pixel) values of four points around the coordinate point of the one image are found. Is a method for obtaining a (pixel) value of a point P (u, v) of the converted image. According to this method, the pixel value P of the point P of the converted image is calculated according to the following Expression 15.

In order to obtain the point p (x, y) of the original image corresponding to the point P (u, v) of the converted image, the image processing device 25 executes the process of the flowchart shown in FIG.
The image processing device 25 initializes the pixel position u of the converted image to 0 (step S91).

The image processing device 25 initializes the pixel position v of the converted image to 0 (step S92).
The image processing device 25 substitutes the pixel position (u, v) of the converted image into the affine parameter A obtained by Equations 5 and 8, and obtains the pixel position (x, y) of the original image according to Equation 2. (Step S93).

The image processing device 25 obtains the pixel value P (u, v) from the obtained pixel position (x, y) according to the formula 15 by the bilinear method (step S94).
The image processing device 25 increments the corrected image coordinate v by one (step S95).

  The image processing device 25 compares the coordinate v of the corrected image with the maximum value vmax of the coordinate v, and determines whether or not the coordinate v of the corrected image is greater than or equal to the maximum value vmax (step S96).

  When it is determined that the coordinate v is less than the maximum value vmax (No in step S96), the image processing device 25 executes steps S93 to S95 again.

  When it is determined that the coordinate v has reached the maximum value vmax by repeating the processes in steps S93 to S95 (Yes in step S96), the image processing device 25 increments the coordinate u of the corrected image by one ( Step S97).

The image processing device 25 compares the coordinate u with the maximum value umax of the coordinate u, and determines whether or not the coordinate u is equal to or greater than the maximum value umax (step S98).
When it is determined that the coordinate u is less than the maximum value umax (No in step S98), the image processing device 25 executes the processes of steps S92 to S97 again.

  When it is determined that the coordinate u has reached the maximum value umax by repeating the processes of steps 92 to S97 (Yes in step 98), the image processing apparatus 25 ends the image conversion process.

(3) Image Conversion Adjustment Next, adjustment (step S22 in FIG. 5) performed on an image that has been once converted will be described.
In the case where some errors are included in the coordinates of the extracted quadrangular vertices, as shown in FIG. 22A, the result of projection with the obtained affine parameters may not be preferable. For this reason, the digital camera 1 of the present embodiment is configured so that the user can adjust the projective transformation in order to adjust the image once converted and obtain an image as shown in FIG. Has been.

  When the user operates the projection conversion key and the rotation correction key of the operation unit 26, the operation unit 26 transmits this operation information to the CPU 30 in response to the user's operation. The CPU 30 determines this operation information and controls the image processing device 25 according to the determination result.

  As shown in FIG. 23, when the interpolation pixel Q (u ′, v ′) of the corrected image is obtained, an inverse transformation Ai is performed on the interpolation pixel Q (u ′, v ′) to obtain the interpolation pixel Q. A corrected image P (u, v) corresponding to (u ′, v ′) is obtained, and further the inverse transformation is performed on the corrected image P (u, v) to obtain p (x, y) of the original image. Then, pixel interpolation is performed on the image p (x, y). When performing two-stage image conversion, such as projection conversion and enlargement conversion, a conversion matrix obtained by combining the two conversions is obtained, and the original image is converted at a time. In this case, an image can be obtained at a higher speed than when conversion is performed twice, and image deterioration can be reduced.

The rotation inverse transformation matrix Ar for obtaining the image before conversion from the image after conversion obtained by rotating the image before conversion by the angle θ about the X axis and the Y axis is expressed by the following equation (16). expressed.
An enlargement matrix Asc in the case of obtaining an image before conversion from an image after conversion obtained by enlarging the image before conversion by Sc magnification around the X axis and the Y axis is expressed by the following Expression 17. The

  Note that once an image is enlarged, a rounding error may be processed by adjusting or calculating an affine parameter. For this reason, before enlarging the image, it is necessary to restore the original affine parameters of the same magnification.

The movement matrix As for obtaining the image before conversion from the image after conversion obtained by moving the image before conversion in the X and Y directions by Tx and Ty, respectively, is expressed by the following equation (18). Is done.

The projection effect matrix Ap in the case of acquiring the image before conversion from the image after conversion obtained by inclining the image before correction in the X and Y directions by α and β, respectively, is given by expressed.
When performing two-stage inverse transformation, the inverse transformation matrix A is expressed by the following equation (20).

  In this embodiment, the projection effect parameters α and β are set in increments of 0.1, and the angle correction parameter θ is set in increments of 1 degree. However, the adjustment range of each parameter can be determined while checking the actual correction effect.

Image conversion adjustment processing executed based on this concept will be described based on the flowchart shown in FIG.
The CPU 30 sets the center coordinates Xc and Yc of the image (step S101).
The CPU 30 determines whether or not the projection conversion key has been pressed based on the operation information transmitted from the operation unit 26 (step 102).
If it is determined that the projection conversion key has been pressed (Yes in step S102), the CPU 30 has pressed one of the upper reduction key 111, the lower reduction key 112, the right reduction key 113, and the left reduction key 114 as the projection conversion key. The key type is determined (steps S103 to S106).

  When determining that the pressed projective transformation key is the up-reduction key 111, the CPU 30 assigns α = 0.1 and β = 0 to the projective effect matrix Ap shown in Equation 19 to obtain an inverse transformation matrix Ai = Ap. Is acquired (step S103).

  If it is determined that the pressed projective transformation key is the lower reduction key 112, the CPU 30 assigns α = −0.1 and β = 0 to the projective effect matrix Ap shown in Equation 19 to obtain an inverse transformation matrix Ai = Ap is acquired (step S104).

  When it is determined that the pressed projective transformation key is the right rotation key 113, the CPU 30 substitutes α = 0 and β = 0.1 for the projection effect matrix Ap shown in Equation 19 to obtain an inverse transformation matrix Ai = Ap. Is acquired (step S105).

  When determining that the pressed projective transformation key is the left rotation key 114, the CPU 30 assigns α = 0 and β = −0.1 to the projective effect matrix Ap shown in Equation 19, respectively, and the inverse transformation matrix Ai = Ap is acquired (step S106).

If it is determined that the projection conversion key has not been pressed (No in step 102), the CPU 30 determines whether or not the rotation correction key has been pressed (step S107).
If it is determined that the rotation correction key has been pressed (Yes in step S107), the CPU 30 determines the type of the rotation correction key that has been pressed (steps S108 and S109).

  When determining that the pressed rotation correction key is the right rotation key 115, the CPU 30 substitutes θ = −1 into the rotation inverse transformation matrix Ar shown in Expression 16 to obtain the inverse transformation matrix Ai = Ar. (Step 108).

  If it is determined that the pressed rotation correction key is the left rotation key 116, the CPU 30 substitutes θ = 1 for the rotation inverse transformation matrix Ar shown in Equation 16 to obtain the inverse transformation matrix Ai = Ar ( Step 109).

  When it is determined that the projective transformation key or the rotation correction key has been pressed and the inverse transformation matrix Ai is set (steps S103 to S106, S108, S109), the CPU 30 obtains the inverse transformation matrix A according to Equation 20 ( Step S110).

The CPU 30 supplies the obtained conversion matrix A to the image processing device 25, and controls the image processing device 25 to perform image conversion based on the inverse conversion matrix A. The image processing device 25 performs image conversion by affine transformation based on the supplied inverse transformation matrix A (step S111), and ends this image transformation adjustment processing.
On the other hand, if it is determined that the rotation correction key has not been pressed (No in step S107), the CPU 30 ends the image conversion adjustment process as it is.

  By performing image conversion by the above-described method using the affine parameters obtained in this manner, the image can be further manually adjusted with respect to the corrected image.

  For example, when adjusting an image as shown in FIG. 22, since the image is distorted leftward, first, when the right rotation key 115 is pressed, the image processing device 25 rotates the image to the right. If the right rotation key 115 is pressed as it is and the character string is correctly displayed and the right rotation key 115 stops being pressed, the image processing device 25 stops the right rotation of the image.

  Next, since the left side of the image is larger than the right side, when the left reduction key 114 is pressed, the image processing device 25 adjusts the left and right sides of the image. If the left reduction key 114 is pressed and the left and right balances are aligned and the pressing of the left reduction key 114 stops, the image processing device 25 stops the left reduction process.

  In this method, an image can be obtained by performing only one image conversion on the original image, so that an image with better image quality than rotating and projecting the image once subjected to projection correction can be obtained. Can do.

(4) Extraction of image effect correction parameters relating to luminance or color difference and image effect processing Next, processing for extracting image effect correction parameters from the image thus obtained, and image effects performed using these parameters Processing will be described. The image effect process is a process for obtaining a clearer image.

  The image thus obtained is an image obtained by cutting out the white board 2 or the document as a result. When performing image effects such as histogram correction, more effective parameters are expected to be obtained by obtaining parameters from the cut out image than by obtaining correction parameters from the original image and performing correction.

  In the present embodiment, a histogram is created from luminance (Y) from the corrected image data, and image effect processing is performed according to the histogram.

  The image effect correction parameters are variables necessary for image effect processing such as the maximum value, minimum value, and peak value of the luminance histogram.

  In order to extract the image effect correction parameter, it is necessary to generate a luminance histogram. First, processing necessary for extracting image effect correction parameters will be described.

  The luminance histogram indicates the distribution of luminance values (Y) existing in the image, and is generated by counting the number of pixels for each luminance value. FIG. 25 shows an example of a luminance histogram. In FIG. 25, the horizontal axis indicates the luminance value (Y), and the vertical axis indicates the number of pixels. In order to correct the image effect, it is necessary to obtain a maximum value (Ymax), a minimum value (Ymin), and a peak value (Ypeak) as image effect correction parameters.

  The maximum value counts the number of pixels for each luminance value, and is a value indicating the maximum luminance among luminance values having a predetermined number or more set in advance, and the minimum value is a predetermined number or more This is a value indicating the minimum luminance among the luminance values having the counted value. The peak value is a luminance value that maximizes the count value. The peak value is considered to indicate the luminance value of the background color of the photographing object.

  In order to obtain an image with excellent visibility by correcting the image effect, the correction effect differs depending on the background color of the object to be imaged. Therefore, it is necessary to change the image effect correction method depending on the background color of the object to be imaged. is there. For this reason, it is necessary to determine the background color of the object to be photographed. The background color of the object to be photographed is determined from the peak values of the luminance histogram and the color difference histogram.

  Here, the background color of the photographing object is classified into three. The first is a case where the background color is white, such as a whiteboard or notebook. The second is a case where the background color is black, such as a blackboard. The third is a case where the background color is other than white or black, such as a magazine or a pamphlet.

Specifically, the background color of the shooting target is determined according to the following discriminant.
(2-a) White Determination Condition The white determination condition is expressed by the following Expression 21, and when the condition shown in Expression 21 is satisfied, the background color of the photographing target is determined to be white (W).
(2-b) Black Determination Condition The black determination condition is represented by the following Expression 22, and when the condition shown in Expression 22 is satisfied, the background color of the shooting target is determined to be black (b).

  If the conditions shown in Equations 21 and 22 are not satisfied, the background color of the object to be photographed is determined to be color (C). For example, the color threshold is set to 50, the white determination threshold is set to 128, and the black determination threshold is set to 50, for example.

Based on such a concept, the image processing device 25 executes image effect correction parameter extraction processing according to the flowchart shown in FIG.
The image processing device 25 counts the number of pixels having each luminance (Y) value, and generates a luminance histogram as shown in FIG. 25 (step S121).

  The image processing device 25 acquires the maximum value (Ymax), the minimum value (Ymin), and the peak value (Ypeak) of the luminance from the generated luminance histogram (step S122).

The image processing device 25 discriminates the background color of the object to be photographed from the peak value (Ypeak) of the luminance histogram according to the judgment condition formulas shown in Equations 21 and 22 (step S123).
The image processing device 25 stores the image effect correction parameter and the background color data of the shooting target in the memory 23 (step S124).

Next, the image processing device 25 performs image effect processing (step S24 in FIG. 5) using the image effect correction parameters extracted in this way.
As described above, in order to effectively perform the image effect processing, it is necessary to change the processing contents depending on the background color.

  When the background color is white, such as a whiteboard or notebook, luminance conversion as shown in FIG. When the background color is black, such as a blackboard, luminance conversion as shown in FIG. When the background color is other than white or black, such as a magazine or pamphlet, the conversion shown in FIG. 27C is performed. In FIGS. 27A, 27B, and 27C, the horizontal axis indicates the input value of the pixel value, and the vertical axis indicates the output value of the pixel value.

  When the background color is white, as shown in FIG. 27A, the inclination angle of the luminance conversion line is changed with the peak value as a boundary. The predetermined luminance value is set to 230, for example, and the input luminance peak value is raised to the luminance value 230. Then, the maximum value is raised to the maximum brightness. Therefore, the luminance conversion line is represented by two line segments as shown in FIG.

  When the background color is black, luminance conversion is performed so that the peak value becomes a certain luminance value (20) as shown in FIG. Also in this case, as shown in FIG. 27B, the luminance conversion line is represented by two line segments.

  When the background color is a color other than white or black, as shown in FIG. 27C, the minimum value or less and the maximum value or more are cut and expressed as one line segment as in the normal enlargement process. The luminance conversion line is set as follows.

  It should be noted that a conversion table of background luminance (Y) and output (Y ′) may be set in advance and stored in the memory card 31 as described above. According to the created conversion table, each output value is obtained from the input pixel value, and image effect processing is performed. In the image thus converted, bright pixels are brighter and dark pixels are darker, so that the luminance distribution is widened and the image is excellent in visibility.

Based on such a concept, the image processing device 25 executes image effect processing according to the flowchart shown in FIG.
The image processing device 25 reads the stored image effect correction parameters from the memory 23 (step S131).

The image processing device 25 determines whether or not the background is white (step S132).
When the background is determined to be white (Yes in step S132), the image processing device 25 performs luminance conversion as illustrated in FIG. 27A so that the background is whiter and the visibility is improved. The brightness histogram is adjusted (step S133).

  If it is determined that the background is not white (No in step S132), the image processing apparatus 25 determines whether the background is black (step S134).

  When it is determined that the background is black (Yes in step S134), when the background is black, the image processing apparatus 25 performs luminance conversion as illustrated in FIG. 27B and adjusts the luminance histogram (step). S135).

  If it is determined that the background is not black (No in step S134), the image processing apparatus 25 performs luminance conversion as shown in FIG. 27C and performs histogram adjustment according to the background color of the shooting target. (Step S136).

In the digital camera 1 of the present embodiment, the mode can be set not only to the shooting mode but also to the playback mode.
When the playback key of the operation unit 26 is pressed, the operation unit 26 transmits this operation information to the CPU 30.

When the CPU 30 determines that the mode is the playback mode based on the operation information, the CPU 30 executes the playback process according to the flowchart shown in FIG.
The CPU 30 selects one image selected by the user from the image files recorded in the memory card 31 (step S141).

The CPU 30 reads the selected image file from the memory card 31 and writes it in the memory 23 (step S142).
The CPU 30 creates a reduced image from the read image (step S143).

  The CPU 30 writes in the preview image storage area on the memory 23 (step S144). Thus, the reduced image is displayed from the display device 24 as in the photographing mode. The user can confirm the reproduced image by viewing this display.

If the user visually recognizes this display and presses the NO key to view another image, the operation unit 26 transmits this operation information to the CPU 30.
The CPU 30 determines whether or not the next image has been designated according to the operation information (step S145).

  When it is determined that the next image has been designated (Yes in step S145), the CPU 30 selects the next image (step S146) and executes steps S142 to S144 again.

If it is determined that the next image has not been designated (No in step S145), the CPU 30 determines whether or not the reproduction mode has ended (step S147).
If the shooting key is not pressed, the CPU 30 determines that the playback mode has not ended (No in step S147), and returns to step 145 (determination of whether or not the next image has been designated).

  On the other hand, when the photographing key is pressed, the CPU 30 receives this operation information from the operation unit 26 and determines that the reproduction mode has ended (Yes in step S147), and ends the reproduction process.

Next, writing parameters to the file header will be described.
When both the original image and the corrected image are recorded as JPEG format images, the file name of the original image is recorded in the header (option data) area of the image file. When it is desired to correct the corrected image again, if the corrected image file is corrected, an image with little deterioration can be generated by generating the corrected image file from the original image. This image editing may be performed in the digital camera 1. However, image editing can also be performed on a computer. If image editing is performed on a computer, it is expected that more advanced image correction can be performed.

  Further, by recording the parameters when image processing is performed in the header, the computer can easily create a corrected image again using the parameters for the original image. For this reason, it becomes easy for the user to make a change immediately from the previous corrected image.

As shown in FIG. 30, the header information includes the type of data (data name), the number of bytes of each data, and the contents thereof.
The CPU 30 sequentially stores the header information in the header information storage area, the image file name of the original image data, the image size of the corrected image, the affine parameters, and an input / output data set for creating a histogram table.

  The input / output data set is a set of data indicating the relationship between the input data and the output data, and is arranged in order from the smallest of the input data for each change point where the slope of the straight line indicating the input / output relationship changes. This is a set of output data.

If the input data at the change point is x1, x2, ..., xm in order from the smallest, and the corresponding output data is y1, y2, ..., ym, the data set is (x1, y1), ( x2, y2),..., (xm, ym). In this case, if the input data x is xi <x <xi + 1, the output data y is expressed by the following equation (23).

  For example, when the relationship between the input data and the output data is as shown in FIG. 27A, the CPU 30 determines (0, 0), (minimum value, 0), (peak value, constant luminance value ( = 230)), (maximum value, 255), and (255, 255) are stored in the memory card 31.

  Similarly, when the relationship between the input and output data is as shown in FIG. 27B, the CPU 30 determines (0, 0), (minimum value, 0), (peak value, constant luminance value (= 20). )), (Maximum value, 255), and (255, 255) are stored in the memory card 31.

  When the relationship between the input and output data is as shown in FIG. 27C, the CPU 30 determines (0, 0), (minimum value, 0), (maximum value, 255), (255, 255). Four data sets are stored in the memory card 31.

  By recording such a brightness value data set, various histograms can be described. The CPU 30 records such data on the memory card 31 simultaneously with the recording of the corrected image data (step S26 in FIG. 5).

Next, correction image editing processing will be described.
In some cases, after image data is stored once, it is desired to correct the image data again. In this case, as described above, it is preferable to collectively correct the original image data in that image deterioration is small.

When the user operates the control key to designate an image editing mode, the CPU 30 executes an image editing process according to the flowchart shown in FIG.
The CPU 30 selects a corrected image according to the operation information transmitted from the operation unit 26 (step S151).

The CPU 30 reads header information from the header information storage area of the memory card 31 (step S152).
The CPU 30 reads the original image (step S153).
The CPU 30 sends the read corrected image, header information, and original image to the image processing device 25, and controls the image processing device 25 to correct again (step S154).

  The image processing device 25 acquires a corrected image, header information, and an original image from the CPU 30, and creates a projective conversion image (step S155). The image processing device 25 sends the data of the created projective transformation image to the display device 24, and the display device 24 displays this image on the liquid crystal monitor 12.

  When the user operates the projection conversion key and the rotation correction key, the operation unit 26 transmits this operation information to the CPU 30. The CPU 30 acquires this operation information and controls the image processing apparatus 25 to perform the projection manual correction process.

The image processing device 25 performs a projection manual correction process according to the flowchart shown in FIG. 24 (step S157).
The image processing device 25 records the corrected image subjected to the projection manual correction process together with the header information on the memory card 31 (step S158), and ends the editing process.

  As described above, according to the present embodiment, the image processing device 25 generates a reduced image by selecting pixels of the image of the shooting target (white plate 2), and distorts the selected pixels as a survey target. Correction was made to create an edge image.

  Accordingly, the number of data to be processed can be reduced while maintaining accuracy, so that the correction of the distortion of the image of the photographic subject can be speeded up. Even when a straight line is extracted, the Hough transform is executed with the investigation range reduced, so that the processing can be speeded up.

  Also, because the contour is obtained from the created edge image, the shape of the object to be photographed is obtained, and the projection parameter is obtained from the vertex position of the quadrangle of the object to be photographed, and the image of the object to be photographed is projectively transformed. If the object to be photographed is a quadrangle, the image distortion can be automatically corrected easily. Further, even when the image is taken from an oblique direction, an image having high legibility similar to the image of the object to be imaged taken from the front can be obtained.

In carrying out the present invention, various forms are conceivable and the present invention is not limited to the above embodiment.
For example, in the above embodiment, the parameters used to create the corrected image are stored in the corrected image file. However, this parameter can also be stored in the original image file. If the parameters are saved in the original image file, the corrected image can be easily created from the original image without saving the corrected image.

  As shown in FIG. 32, the header information in this case includes only the image size, projection parameters, and image effect correction parameters.

Further, when performing the photographing process, the digital camera 1 immediately performs the projection parameter extraction process without storing the converted image. The contents of this processing are shown in the flowchart of FIG.
The image processing device 25 creates a projection correction image (steps S161 to S169), performs image conversion adjustment, extraction of image effect correction parameters, and image effect processing (steps S170 to 172).

The CPU 30 creates header information shown in FIG. 32 (step S173).
The image processing device 25 compresses the original image data, and the CPU 30 records the created header information together with the compressed original image data in the memory card 31 (steps S174 and 175).

Also, the re-editing process is different from the processing content shown in FIG. That is, the re-editing process is performed not on the corrected image but on the original image. This processing content is shown in the flowchart of FIG.
The CPU 30 selects the original image and reads header information from the original image file (steps S181 and S182).

The image processing device 25 immediately creates a projection corrected image and performs image effect processing (step S183).
The CPU 30 and the image processing device 25 perform a projection manual correction process. After the correction is completed, the CPU 30 creates header information based on the changed projection correction parameter, and records the created header information in the original image file again. (Steps S184 to S187).

  In the above embodiment, the image correction is performed by the digital camera 1. However, image correction can also be performed by a computer. In this case, a computer is connected to the computer interface unit 27, and the computer executes image editing processing according to the flowchart shown in FIG. If the computer performs such image editing processing, the computer includes a mouse and the like, and it becomes easier to input operation information than the digital camera 1, so that the operability is improved. Further, since the display device of the computer is generally larger than the liquid crystal monitor 12 of the digital camera 1, the image can be visually confirmed in detail and image correction can be performed with high accuracy.

  In the above embodiment, a warning is given when a quadrangle cannot be acquired. However, instead of issuing a warning, the photographed image can be displayed and the user can designate four rectangular points using a control key or the like. Then, the affine parameters can be obtained using the designated four points.

  In the above-described embodiment, the program is described as being stored in advance in a memory or the like. However, a program for causing a computer to operate as the whole or a part of the apparatus or to execute the above-described processing is a flexible disk, a CD-ROM (Compact Disk Read-Only Memory), a DVD (Digital Versatile Disk), The information may be stored in a computer-readable recording medium such as an MO (Magneto Optical disk) and distributed, installed in another computer, operated as the above-described means, or the above-described steps may be executed.

  Furthermore, the program may be stored in a disk device or the like of a server device on the Internet, and may be downloaded onto a computer by being superimposed on a carrier wave, for example.

It is explanatory drawing which shows a state when image | photographing a white board with the digital camera which concerns on embodiment of this invention. It is a block diagram which shows the structure of the digital camera shown in FIG. It is explanatory drawing for demonstrating the function of the image processing apparatus shown in FIG. It is explanatory drawing for demonstrating each key with which the operation part shown in FIG. 2 is provided. It is a flowchart which shows the content of the imaging | photography process which a digital camera performs. It is explanatory drawing which shows the square of the clipping object of the image processing apparatus shown in FIG. It is explanatory drawing for demonstrating the basic idea of extraction of a projection parameter, and an affine transformation. It is a flowchart which shows the content of the projection parameter extraction process which the image processing apparatus shown in FIG. 2 performs. It is a flowchart which shows the content of the square outline extraction process which the image processing apparatus shown in FIG. 2 performs. It is explanatory drawing for demonstrating a reduced luminance image and an edge image. It is explanatory drawing for demonstrating the function of a Roberts filter. It is a figure for demonstrating the straight line detection using Hough transform, (a) shows the process which detects an upper and lower side, (b) has shown the process which detects a right and left side. It is an image figure which shows that upper and lower sides or right and left can be determined depending on whether ρ is a positive value or a negative value. FIG. 3 is a flowchart showing the contents of a quadrangle detection process in the case of document shooting performed by the image processing apparatus shown in FIG. 2. It is an image figure which shows a mode that the square after a straight line detection is specified typically, (a) shows the case of document photography, (b) has shown the case other than document photography. FIG. 3 is a flowchart showing the contents of a quadrangle detection process in a case other than document shooting performed by the image processing apparatus shown in FIG. 2. It is a flowchart which shows the content of the process which calculates | requires an affine parameter from the square vertex performed by the image processing apparatus shown in FIG. It is explanatory drawing which shows the content of the warning when a square cannot be extracted. It is a flowchart which shows the content of the warning process which CPU shown in FIG. 2 performs. It is explanatory drawing for demonstrating the inverse transformation for obtaining the original image from the image after projective transformation. It is a flowchart which shows the content of the image conversion process by the affine transformation which the image processing apparatus shown in FIG. 2 performs. It is explanatory drawing which shows the example which could not correct | amend the distortion of an image by projective transformation. It is explanatory drawing for demonstrating the correspondence with the original image, a projection conversion image, and the enlarged projection conversion image. 3 is a flowchart showing details of correction adjustment and image conversion processing executed by the image processing apparatus shown in FIG. 2. It is explanatory drawing which shows an example of a brightness | luminance histogram. 3 is a flowchart showing the contents of image effect correction parameter extraction processing executed by the image processing apparatus shown in FIG. 2. It is explanatory drawing which shows an image effect process, (a) shows an image effect process in case a background color is white, (b) shows an image effect process in case a background is black, (c), The image effect processing when the background is other than white or black is shown. It is a flowchart which shows the content of the image effect process which the image processing apparatus shown in FIG. 2 performs. It is a flowchart which shows the content of the reproduction | regeneration processing which CPU shown in FIG. 2 and an image processing apparatus perform. It is explanatory drawing which shows the content of header information. 3 is a flowchart showing the contents of a re-editing process of a corrected image executed by the CPU and image processing apparatus shown in FIG. It is explanatory drawing which shows the content of the header information in the case of preserve | saving only an original image. FIG. 3 is a flowchart showing the contents of a photographing process executed by the CPU and image processing apparatus shown in FIG. 2 when only an original image is stored. FIG. 3 is a flowchart showing details of a correction image re-editing process executed by the CPU and the image processing apparatus shown in FIG. 2 when only the original image is stored.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Digital camera, 2 ... White board, 11 ... Shooting lens part, 12 ... Liquid crystal monitor, 13 ... Shutter button, 21 ... Optical lens apparatus, 22 ... Image sensor, 23 ... Memory, 24 ... Display device, 25 ... Image processing device, 30 ... CPU, 31 ... Memory card

Claims (8)

In a photographing device that photographs a photographing object by a photographing unit,
An image processing unit that performs image processing on the photographed image so as to correct distortion of the image of the photographing object obtained by photographing;
The image processing unit
A reduced image with a reduced amount of data is generated by selecting pixels of the image of the photographing object, distortion correction of the image of the photographing object is performed on the selected pixels, and the reduced image is generated from the generated reduced image. A shape acquisition unit that generates an edge image of the image of the shooting target and acquires the shape of the image of the shooting target from the generated edge image;
The shape of the image acquired by the shape acquisition unit and the actual shape of the object to be photographed are associated with each other by setting a three-dimensional coordinate system. A projection parameter obtaining unit for obtaining a projection parameter indicating a relationship with the object to be imaged,
An aspect ratio acquisition unit for obtaining an aspect ratio of the photographing object based on a focal length of a lens included in the photographing unit and a vertex position of an image of the photographing object;
An image conversion unit that performs image conversion of the image of the photographing object using the projection parameter obtained by the projection parameter obtaining unit and the aspect ratio obtained by the aspect ratio obtaining unit;
An imaging apparatus characterized by that. The shape acquisition unit
An edge that selects a pixel that generates a reduced image of the image of the shooting object, and performs an edge correction of the image of the shooting object by performing distortion correction on the image of the shooting object using the selected pixel as an investigation target. An image generator;
A straight line acquisition unit that acquires a straight line forming the edge image from the edge image generated by the edge image generation unit;
A shape candidate generation unit that generates a shape candidate of the image of the object to be photographed by the straight line acquired by the straight line acquisition unit;
A shape selection unit that selects the shape of the image of the shooting target from the shape candidates of the image of the shooting target generated by the shape candidate generation unit;
The imaging apparatus according to claim 1, wherein: The straight line acquisition unit obtains the length of the straight line obtained from the edge image generated by the edge image generation unit, and based on the obtained straight line length, forms a straight line that forms the shape of the image of the photographing object. Select to get,
The imaging apparatus according to claim 2, wherein: The straight line acquisition unit obtains an angle of the straight line obtained from the edge image generated by the edge image generation unit, and selects and obtains a straight line that forms the shape of the image of the photographing object based on the obtained angle. ,
The imaging apparatus according to claim 2, wherein: The shape selection unit selects and selects the image of the object to be imaged generated by the shape candidate generation unit that has the largest image area size surrounded by the shape. As the shape of the image of the object to be photographed,
The imaging apparatus according to claim 2, wherein: A storage unit for storing image data of the photographing object;
The photographing apparatus according to claim 1, wherein the photographing apparatus is characterized in that: An image processing method of a photographing apparatus that performs image processing on an image of the photographing target so as to correct distortion of the image of the photographing target obtained by photographing the photographing target by a photographing unit. ,
Generating a reduced image with a reduced amount of data by selecting pixels of the image of the shooting object, and correcting distortion of the image of the shooting object for the selected pixels;
Generating an edge image of the image of the object to be photographed from the generated reduced image;
Obtaining the shape of the image of the object to be photographed from the generated edge image;
The shape of the object to be photographed and the shape of the image of the object to be photographed acquired by the shape acquisition unit are associated with each other by setting a three-dimensional coordinate system, and from the vertex position of the image of the object to be photographed, Obtaining a projection parameter indicating a relationship between an image and the object to be photographed;
Obtaining an aspect ratio of the photographing object based on a focal length of a lens included in the photographing unit and a vertex position of an image of the photographing object;
Performing image conversion of the image of the object to be photographed using the obtained projection parameters and aspect ratio,
An image processing method for a photographing apparatus. In the computer of the photographing device that photographs the object to be photographed by the photographing unit,
A procedure for generating a reduced image with a reduced amount of data by selecting pixels of the image of the shooting object, and correcting distortion of the image of the shooting object for the selected pixel;
A procedure for generating an edge image of the image of the object to be photographed from the generated reduced image;
A procedure for acquiring the shape of the image of the photographing object from the generated edge image,
The shape of the object to be photographed and the shape of the image of the object to be photographed acquired by the shape acquisition unit are associated with each other by setting a three-dimensional coordinate system, and from the vertex position of the image of the object to be photographed, A procedure for obtaining a projection parameter indicating a relationship between an image and the object to be imaged;
A procedure for obtaining an aspect ratio of the photographing object based on a focal length of a lens of the photographing unit and a vertex position of an image of the photographing object;
A procedure for performing image conversion of the image of the object to be photographed using the obtained projection parameters and aspect ratio;
A program for running