JP4848230B2 - Image processing method, imaging apparatus, image processing apparatus, and program - Google Patents

Description

  The present invention relates to an image processing method, an imaging apparatus, an image processing apparatus, and a program, and more particularly, to an image processing method, an imaging apparatus, an image processing apparatus, and a program that perform correction of tilt on an input image.

  The digital camera has the feature that the photographed image can be viewed and printed immediately. For this reason, photographing with a digital camera is being performed as a substitute for notes such as timetables and posters. However, timetables and posters shot with a digital camera cause tilt (trapezoidal distortion) depending on the shooting position. As a result, the timetables and posters taken were difficult to read and were not optimal for reusing images.

  Therefore, for example, by using the techniques disclosed in Patent Documents 1 to 4 below, a quadrilateral of a photographing target (subject) is extracted from the photographed image, a tilt angle is estimated from the quadrilateral image, and the position from the directly facing position is estimated. Image processing (tilt correction) is performed to convert the captured image. However, in the image processing methods disclosed in Patent Documents 1 to 3 below, the contrast of the image is used to determine the background and the subject. Therefore, for example, when white printing paper is placed on a white desk, there is a problem that the background and the subject cannot be separated because the contrast is low, and the quadrilateral that is the subject cannot be extracted from the photographed image.

  In order to solve this problem, a method of placing a quadrangular marker in the vicinity of a plane on which a document or the like as a subject is placed is conceivable. However, when a marker is placed, the saved image includes the marker, which is inconvenient when secondary use of the saved image is performed.

  Furthermore, in the image processing methods disclosed in the following Patent Documents 1 to 3, the tilt correction target area is limited to a quadrilateral. On the other hand, in the image processing method disclosed in Patent Document 4 below, a photographer instructs an image element and a relationship between image elements for estimating a relative angle of a subject plane from a photographed image. If this technique is used, correction can be made for subjects other than the quadrilateral, but there is a problem that the operation of the photographer becomes complicated.

JP 2005-122328 A JP 2005-143092 A Japanese Patent Application Laid-Open No. 2004-363636 JP-A-11-238123

  The present invention has been made in view of the above, and it is possible to perform correction for a subject having a low contrast with the background without complicated operations by the photographer, and to remove unnecessary areas such as markers. It is an object to obtain an image processing method, an imaging device, an image processing device, and a program that can be stored without being included.

In order to solve the above-described problems and achieve the object, the invention according to claim 1 is an image processing method which is executed by an image processing apparatus and performs tilt correction, wherein the reference graphic extraction means of the image processing apparatus includes: A reference graphic extraction step for extracting the reference graphic from image data including a subject to be corrected for tilt and a reference graphic used for tilt correction; and a trimming means of the image processing device, wherein the image data is converted into the reference data. A trimming step for trimming an area not including a graphic; a conversion matrix calculating unit for calculating a conversion matrix based on the reference graphic; and a projecting converting unit for the image processing apparatus. image data, seen including a projective conversion step of projective transformation, a using the transformation matrix, the reference graphic extracting means, the image data To evaluate the strength of the size or contrast for graphic contained, and wherein the selecting the higher figure most evaluation value to the reference graphic.

  The invention according to claim 2 is characterized in that in the projective transformation step, the trimmed image is subjected to the projective transformation.

  The invention according to claim 3 is characterized in that, in the trimming step, the image data subjected to the projective transformation is trimmed into an area not including the reference graphic.

  The invention according to claim 4 is characterized in that the region to be trimmed is a region obtained by removing a rectangular region including the reference graphic from the entire image region.

The invention according to claim 5 is an image processing method that is executed by an image processing apparatus and performs tilt correction, wherein the reference graphic extraction means of the image processing apparatus is used to correct the tilt and the subject to be corrected. A reference graphic extraction step for extracting the reference graphic from image data including a reference graphic to be used, and a trimming step in which the trimming means of the image processing apparatus trims the image data into an area not including the reference graphic; A transformation matrix calculating step in which a transformation matrix calculating means of the image processing device calculates a transformation matrix based on the reference graphic; and a projective transformation means of the image processing device uses the transformation matrix to perform a projective transformation. A projective transformation step, wherein the trimming means is adapted to operate the image data in accordance with a digital zoom operation by a user. Trimmed, the reference graphic extracting means may be selected in the reference graphic by extracting a shape from the region is removed from the trimming region by said digital zoom.

According to a sixth aspect of the present invention, there is provided an imaging apparatus that performs tilt correction, an imaging unit that captures an imaging target region including a subject to be corrected for tilt and a reference graphic used for tilt correction, and the imaging A reference graphic extraction means for extracting the reference graphic from the image data captured by the means; a trimming means for trimming the image data into a region not including the reference graphic; and a conversion for calculating a conversion matrix based on the reference graphic Matrix calculation means, and projective transformation means for projective transformation of the image data using the transformation matrix , wherein the reference graphic extraction means determines the size or contrast strength of the graphic included in the image data. The figure having the highest evaluation value is selected as the reference figure .

  The invention according to claim 7 is characterized in that the projective transformation means performs the projective transformation on a trimmed image.

  The invention according to claim 8 is characterized in that the trimming means trims the image data subjected to the projective transformation into an area not including the reference graphic.

According to a ninth aspect of the present invention , there is provided an imaging apparatus that performs tilt correction, an imaging unit that captures a shooting target area including a subject to be corrected for tilt and a reference graphic used for tilt correction, and the imaging A reference graphic extraction means for extracting the reference graphic from the image data captured by the means; a trimming means for trimming the image data into a region not including the reference graphic; and a conversion for calculating a conversion matrix based on the reference graphic Matrix calculation means, and projective transformation means for projective transformation of the image data using the transformation matrix, the trimming means trimming the image data according to a digital zoom operation by a user, and the reference graphic extraction means chooses the reference graphic by extracting a shape from the region is removed from the trimming region by the digital zoom And wherein the door.

Further, the invention according to claim 10 is characterized by comprising display means for simultaneously displaying the imaging target area and the trimming area.

  The invention according to an eleventh aspect includes an optical zoom unit that can change a focal length by a user's operation, and a photographing target region temporary setting unit that temporarily sets a photographing target region. A reference graphic is extracted from the image data of the set shooting target area, and the optical zoom means automatically automatically extracts the reference graphic in the temporarily set shooting target area when the reference graphic extraction means is not extracted. It is characterized by zooming out.

According to a twelfth aspect of the present invention, there is provided an image processing apparatus for performing tilt correction, wherein the reference graphic is extracted from image data including a subject to be corrected for tilt and a reference graphic used for tilt correction. Extraction means, trimming means for trimming the image data into an area not including the reference graphic, conversion matrix calculation means for calculating a conversion matrix based on the reference graphic, and the image data using the conversion matrix Projection conversion means for projective conversion , wherein the reference graphic extraction means evaluates the size or contrast strength of the graphic included in the image data, and selects the graphic with the highest evaluation value as the reference graphic It is characterized by.

According to a thirteenth aspect of the present invention, there is provided a function as reference graphic extracting means for extracting the reference graphic from image data including a subject to be corrected for tilt and a reference graphic used for tilt correction, and the image data. A function as a trimming means for trimming an area not including the reference graphic, a function as a conversion matrix calculating means for calculating a conversion matrix based on the reference graphic, and projective conversion of the image data using the conversion matrix. The reference graphic extraction means evaluates the size or contrast strength of a graphic included in the image data, and has the highest evaluation value. A figure is selected as the reference figure .

  According to the first aspect of the present invention, since the quadrilateral other than the subject is automatically extracted as a marker to correct the subject's tilt and the region excluding the portion including the marker is saved, the subject and the background are saved. Even when the contrast is low, it is possible to perform correction for a subject having an arbitrary shape without requiring a complicated operation of the photographer, and an unnecessary marker is not included in the corrected stored image. .

  According to the invention of claim 2, since the quadrilateral other than the subject is automatically extracted as a marker, the subject is corrected for tilting, and the region excluding the portion including the marker is saved. Even when the contrast between the background and the background is low, the subject can be corrected without requiring a complicated operation of the photographer for an object of arbitrary shape, and an unnecessary marker is not included in the corrected stored image Play. Furthermore, since the projective transformation is performed after trimming, the processing time is shortened.

  According to the invention of claim 3, the quadrilateral other than the subject is automatically extracted as a marker, the subject is corrected for tilting, and the area excluding the portion including the marker or the like is saved. Even when the contrast between the background and the background is low, the subject can be corrected without requiring a complicated operation of the photographer for an object of arbitrary shape, and an unnecessary marker is not included in the corrected stored image Play.

  According to the invention of claim 4, since the area excluding the rectangular part including the marker or the like is stored, the trimming area can be easily set.

  According to the fifth aspect of the invention, since the user designates the trimming area, the trimming area can be set according to the user request.

  According to the invention of claim 6, the quadrilateral other than the subject is automatically extracted as a marker, the subject is corrected for tilting, and the region excluding the portion including the marker or the like is stored. Even when the contrast between the background and the background is low, the subject can be corrected without requiring a complicated operation of the photographer for an object of arbitrary shape, and an unnecessary marker is not included in the corrected stored image Play.

  According to the seventh aspect of the present invention, since the quadrilateral other than the subject is automatically extracted as a marker, the subject is corrected for tilting, and the region excluding the portion including the marker is stored. Even when the contrast between the background and the background is low, the subject can be corrected without requiring a complicated operation of the photographer for an object of arbitrary shape, and an unnecessary marker is not included in the corrected stored image Play. Furthermore, since the projective transformation is performed after trimming, the processing time is shortened.

  According to the invention according to claim 8, since the quadrilateral other than the subject is automatically extracted as a marker, the subject is corrected for tilting, and the area excluding the portion including the marker is saved. Even when the contrast between the background and the background is low, the subject can be corrected without requiring a complicated operation of the photographer for an object of arbitrary shape, and an unnecessary marker is not included in the corrected stored image Play.

  According to the ninth aspect of the invention, since the user designates the trimming area by digital zoom, the trimming area can be set according to the user request.

  According to the tenth aspect of the present invention, since the photographing area and the digital zoom area are displayed at the same time, there is an effect that the user can easily set the trimming area.

  According to the eleventh aspect of the present invention, the shooting area is temporarily set using optical zoom, and when the reference figure is not included in the temporary shooting area, the zoom-out is automatically performed. There is an effect that the photographing area including the reference graphic can be easily set.

  According to the invention of claim 12, the quadrilateral other than the subject is automatically extracted as a marker, the subject is corrected for tilting, and the region excluding the portion including the marker is saved. Even when the contrast between the background and the background is low, the subject can be corrected without requiring a complicated operation of the photographer for an object of arbitrary shape, and an unnecessary marker is not included in the corrected stored image Play.

  According to the invention of claim 13, the quadrilateral other than the subject is automatically extracted as a marker, the subject is corrected for tilting, and the region excluding the portion including the marker is saved. Even when the contrast between the background and the background is low, the subject can be corrected without requiring a complicated operation of the photographer for an object of arbitrary shape, and an unnecessary marker is not included in the corrected stored image Play.

  Exemplary embodiments of an image processing method, an imaging apparatus, an image processing apparatus, and a program according to the present invention are explained in detail below with reference to the accompanying drawings.

(First embodiment)
FIG. 1 is a block diagram illustrating a configuration example of an imaging apparatus 1 according to the first embodiment of the present invention. The imaging device 1 includes an imaging unit 11, a camera signal processing unit 12, a CPU (Central Processing Unit) 13, a frame memory 14, an I / F (interface) 15, a display unit 16, an external storage device 17, and a ROM (Read Only Memory). 18.

  The imaging unit 11 includes a lens 20, a diaphragm 21, a shutter 22, a photoelectric conversion element 23, and a preprocessing unit 24. For the imaging unit 11, for example, a commercially available “digital camera” or a part thereof may be used. For the photoelectric conversion element 23, for example, a CCD (charge coupled device) is used. The preprocessing unit 24 includes an analog signal processor such as a preamplifier and AGC (automatic gain control), and an A / D converter. The preprocessing unit 24 amplifies the analog video signal output from the photoelectric conversion element 23. After preprocessing such as clamping, the analog video signal is converted into a digital video signal.

  The camera signal processing unit 12 includes a digital signal processor (DSP) and the like, and performs image processing such as image compression on the digital video signal output from the preprocessing unit 24. The CPU 13 includes a microcomputer and controls the camera signal processing unit 12 and the imaging unit 11. The frame memory 14 is a memory used for processing of the camera signal processing unit 12, and generally a semiconductor memory such as a VRAM, SRAM, DRAM or the like is used. The I / F 15 mediates between the camera signal processing unit 12, the display unit 16, and the external storage device 17.

  The display unit 16 is a liquid crystal display device, for example, and displays an image signal. The external storage device 17 is a storage device that stores an image signal that has been subjected to signal processing such as image compression in the camera signal processing unit 12, and an IC memory card, a magneto-optical disk, or the like is used. The ROM 18 is a memory in which fixed information such as a program is stored.

  An image signal subjected to signal processing such as image compression in the camera signal processing unit 12 is stored in the external storage device 17 via the I / F 15. In the present embodiment, the image signal is stored in an IC memory card or magneto-optical disk as the external storage unit 17, but the image signal is transmitted to a remote location via a network using a modem card or ISDN card. It may be transmitted directly to the recording medium. In addition, when the compressed image signal recorded in the external storage device 17 is read, the camera signal processing unit 12 acquires the image signal from the external storage device 17 via the I / F 15 and performs image expansion on the image signal. Is done.

  The image signal read from the external storage device 17 and the frame memory 14 is displayed on the I / F 15 after the camera signal processing unit 12 performs signal processing such as D / A conversion and amplification on the image signal. Is transmitted to the display unit 16 via

  Next, the operation of the tilt correction program according to the present embodiment will be described with reference to the drawings. FIG. 2 is a functional block diagram according to the tilt correction program of the present embodiment. The tilt correction program is stored in the ROM 18 and is executed by the camera signal processing unit 12. As shown in FIG. 2, the tilt correction program includes a quadrilateral extraction unit 25 that extracts a quadrilateral from an input image as an input, and trimming that performs image trimming based on the quadrilateral extracted by the quadrilateral extraction unit 25 And a projective transformation processing unit 27 that calculates a projective transformation matrix from the quadrilateral shape extracted by the quadrilateral extraction unit 25 and performs projective transformation on the image.

  In this embodiment, a quadrilateral is extracted from an image as a representative example of a shape used as a marker, but the shape used as a marker is not limited to this. It is only necessary that the shape of the marker itself is known and the tilt correction angle can be calculated based on the actual shape of the marker and the shape of the marker in the captured image. From the viewpoint of simplifying the calculation of the tilt correction angle, a simple shape such as a quadrangle as in the embodiment or a circle is preferable. The quadrilateral extraction unit 25 is a shape extraction unit using a technique disclosed in, for example, “JP-A-3-94383”, and the shape of the marker is registered in advance in the shape extraction unit in the imaging apparatus. Thus, it is possible to cope with any shape other than the quadrilateral.

  FIG. 3 shows a flowchart of the tilt correction process according to the present embodiment. The tilt correction process will be described with reference to FIG. First, an image taken by the imaging unit 11 is input to the external storage device 17 (step S11). Next, the quadrilateral extraction unit 25 extracts a quadrilateral for use as a marker from the photographed image (step S12). The quadrilateral extraction process of the quadrilateral extraction unit 25 is preferably a process in which high speed is devised, but an extraction method using a general Hough transform may be used. There is a possibility that a plurality of quadrilaterals are extracted in the quadrilateral extraction process of the quadrilateral extraction unit 25. In that case, select one of them. As a selection method, for example, an evaluation value is set for a quadrilateral and the one having the highest evaluation value is selected. A plurality of extracted quadrilaterals are displayed and the photographer selects one quadrilateral. There are methods that can be used. The evaluation value is determined in consideration of, for example, the size of the quadrilateral and the strength of contrast.

  FIG. 4 is an example of an image captured by placing a quadrangular marker 102 that can be easily extracted by the quadrilateral extraction unit 25 in the vicinity of the subject 101. When the image 100 shown in this figure is taken, the marker 102 is extracted in step S12.

  Next, the trimming unit 26 performs trimming so as not to include the quadrilateral (marker) extracted in step S12 (step S13). FIG. 5 is a diagram for explaining a trimming region of the image 100 shown in FIG. The trimming unit 26 first obtains a trimming area excluding the rectangular area (the non-trimming area 104 in the example of FIG. 5) including the quadrilateral (the marker 102 in the example of FIG. 4) obtained in step S12, and then the image An area excluding the non-trimming area from 100 is specified as a trimming area (trimming area 103 in the example of FIG. 5). Then, trimming is performed on this trimming region.

  Next, the projective transformation processing unit 27 calculates a projective transformation matrix for tilt correction based on the quadrilateral (marker) extracted in step S12 (step S14). The trimming area 27 obtained by the trimming unit 26 is subjected to projective transformation using this projective transformation matrix, and the transformed image is stored in the external storage device 17.

  In this embodiment, after the trimming unit 26 calculates the trimming area, the projective transformation processing unit 27 performs the projective transformation. However, the projective transformation image obtained by performing the projective transformation by the projective transformation processing unit 27 is used. On the other hand, a portion other than the rectangular area including the quadrilateral that is the marker may be trimmed and stored in the external storage device 17.

  In the present embodiment, the marker is selected from any object in the imaging area of the imaging unit 11. However, in addition to the imaging unit 11, the imaging unit holding a polarization filter that performs imaging simultaneously with the imaging unit 11. And an object made of a transparent polarizing plate can be used as a marker. In this case, since the marker is included only in the captured image of the imaging unit holding the polarization filter and is not included in the captured image of the imaging unit 11 without the polarization filter, trimming is not necessary.

  As described above, according to the present embodiment, since the subject is corrected by automatically extracting a quadrilateral other than the subject as a marker, the subject having an arbitrary shape can be used even when the contrast between the subject and the background is low. As a result, the tilt correction can be performed without requiring a complicated operation of the photographer. In addition, since the region excluding the portion including the marker or the like is stored, there is an effect that unnecessary markers are not included in the corrected stored image.

(Second Embodiment)
FIG. 6 is a block diagram illustrating a configuration example of an imaging apparatus 1a according to the second embodiment of the present invention. The same parts as those in the first embodiment are denoted by the same reference numerals as those in FIG. In this embodiment, instead of the frame memory 14, the external storage device 17, and the ROM 18, a frame memory 14a, an external storage device 17a, and a ROM 18a are provided.

  FIG. 7 is a rear view of a configuration example of the imaging apparatus 1a according to the present embodiment. On the rear surface of the imaging apparatus 1, a display unit 30, a zoom-in / out button 31, a shutter button 32, a viewfinder 33, an upper button 34, a left button 35, a lower button 36, a right button 37, and a decision button 38 are provided. The display unit 30 can be used as an electronic viewfinder for checking the angle of view at the time of shooting, and can display a preview image of a shot image, a reproduced image read from a memory card, and the like. The photographer operates the up button 34, the left button 35, the down button 36, the right button 37, and the enter button 38 while viewing the menu and setting items displayed on the display unit 30 to select a menu. You can also set various items in each menu. The zoom-in / out button 31 can be operated, for example, in the left / right direction, and the optical zoom function is driven in the zoom-out direction by operating the optical zoom function in the zoom-in direction by operating the left side.

  The photographer determines the angle of view by operating the zoom-in / out button 31 while confirming the real-time image (through image) displayed on the finder 33 or the display unit 30 and depresses the shutter button 32 to shoot.

  Next, the operation of the tilt correction program according to the present embodiment will be described with reference to the drawings. FIG. 8 is a functional block diagram according to the tilt correction program of the present embodiment. The tilt correction program is stored in the ROM 18a and executed by the camera signal processing unit 12, and includes a quadrilateral extraction unit 25a, a trimming unit 26a, and a projective transformation processing unit 27a. Further, the tilt correction program refers to the trimming area storage unit 28 of the frame memory 14a and the processed image storage unit 29 of the external storage device 17a.

  The quadrilateral extraction unit 25a extracts the quadrilateral from the input image, the trimming unit 26a stores the trimming region selected by the photographer by digital zoom in the trimming region storage unit 28 of the frame memory 14a, and the projective transformation processing unit 27a is the quadrilateral. A projective transformation matrix is calculated from the quadrilateral shape extracted by the extraction unit 25a. The trimming area storage unit 28 is a memory for storing the trimming area selected by the photographer as described above. The processed image storage unit 29 stores the image processed by the projective transformation processing unit 27a.

  FIG. 9 is an example of an image captured by placing a quadrangular marker 106 that can be easily extracted by the quadrilateral extraction unit 25 a in the vicinity of the subject 105.

  FIG. 10 is a flowchart of the tilt correction process according to the present embodiment. The tilt correction processing procedure of the present embodiment will be described with reference to FIGS. 8 and 10 taking the image of FIG. 9 as an example.

  First, the photographer needs to place the subject 105 for which tilt correction is desired and the marker 106 on the desk and view the real-time image displayed on the display unit 30 so that the subject 105 and the marker 106 enter the photographing region. Accordingly, the zoom-in / out button 31 is operated to determine the photographing range (step S21). The marker 106 indicates an appropriate quadrilateral that can be extracted by the quadrilateral extraction unit 25a. However, if the quadrilateral is extracted by the quadrilateral extraction unit 25a, there is no need to prepare the marker 106. An appropriate quadrilateral existing around the subject may be used.

  Next, the photographer sets digital zoom (step S22). The digital zoom operation is performed, for example, by pressing a left button 35 to which a digital zoom-in operation is assigned and a right button 37 to which a digital zoom-out operation is assigned. At this time, for example, as shown in FIG. 11, the display unit 30 displays the brightness value outside the digital zoom area 107 in the entire area to be imaged by lowering the brightness value of the captured image by a certain value. The photographer can recognize the area 107. In this embodiment, the digital zoom area 107 is identified by lowering the luminance value outside the digital zoom area 107, but other display methods may be used as long as the digital zoom area 107 can be recognized by the photographer. The photographer determines the digital zoom area by operating the determination button or the like while viewing the displayed image.

  Next, the photographer takes a picture by pressing the shutter button 32 (step S23). At this time, in response to pressing of the shutter button 32, the trimming unit 26a stores the selected digital zoom region image in the trimming region storage unit 28 as a trimming region image. Subsequently, quadrilateral extraction is performed on the entire captured image by the quadrilateral extraction unit 21a (step S24). The quadrilateral extraction unit 21a has the same function as the quadrilateral extraction unit of the first embodiment, and determines one quadrilateral to be used as a marker. In the present embodiment, the quadrilateral extraction unit 21a performs quadrilateral extraction on the entire captured image. However, the entire image is excluded from the trimming region stored in the trimming region storage unit 28. Quadrilateral extraction may be performed. In the example of the image of FIG. 9, the marker 106 is extracted and determined as a quadrilateral used as a marker by the quadrilateral extraction unit 21a.

  Next, the projective transformation processing unit 27a calculates a projective transformation matrix for tilt correction based on the quadrilateral (marker 106 in the example of FIG. 9) extracted and selected by the quadrilateral extracting unit 21a. Next, using the calculated projective transformation matrix, the image stored in the trimming area storage unit 28 is read and subjected to projective transformation (step S25). Then, the processed image is stored in the processed image storage unit 29. FIG. 12 shows an example of an image stored after the projective transformation process in the photographing example shown in FIG.

  As described above, according to the present embodiment, the photographer selects the region of the subject using the digital zoom, and automatically extracts the quadrilateral other than the subject as a marker to perform the tilt correction of the subject. Even when the contrast between the subject and the background is low, the subject can be corrected for a subject having an arbitrary shape without requiring a complicated operation by the photographer, and only the area saved by the photographer can be saved. Play.

  The other configuration and processing of the imaging device 1a according to the second embodiment are the same as the configuration and processing of the imaging device 1a according to the first embodiment.

(Third embodiment)
FIG. 13 is a block diagram illustrating a configuration example of an imaging apparatus 1b according to the third embodiment of the present invention. The same parts as those in the first embodiment are denoted by the same reference numerals as those in FIG. In this embodiment, an external storage device 17b and a ROM 18b are provided instead of the external storage device 17 and the ROM 18, respectively.

  Next, the operation of the tilt correction program according to the present embodiment will be described with reference to the drawings. FIG. 14 is a functional block diagram according to the tilt correction program of the present embodiment. The tilt correction program is stored in the ROM 18b and executed by the camera signal processing unit 12. The projection conversion processing unit 26b and the trimming region calculation unit 27b are also included.

  The quadrilateral extraction unit 21b extracts a quadrilateral from the input image, and the projective transformation processing unit 26b calculates a projective transformation matrix from the quadrilateral shape extracted by the quadrilateral extraction unit 21b, and performs the projective transformation on the image. The trimming area calculation unit 27b calculates a trimming area from the input image. The processed image storage unit 29a is an area of the external storage device 17b for storing the image processed by the projective transformation processing unit 26b.

  FIG. 15 shows a flowchart of the tilt correction process according to the present embodiment. Further, the device configuration example of the present embodiment is the same as that of the second embodiment shown in FIG. Hereinafter, the tilt correction processing procedure of the present embodiment will be described with reference to FIGS. 7, 14, and 15.

  First, the photographer determines and displays the desired shooting range by using the zoom-in / out button 31 and the like as necessary while viewing the real-time image displayed on the display unit 30 while viewing the shooting range desired to be shot (step S31). S31). An example of an image displayed on the display unit 30 at that time is shown in FIG. In the example of FIG. 16, the subject 108 is an elliptical memo paper, and is an image when the memo paper is photographed from a position having a tilt angle.

  When the photographer presses the shutter button 32 halfway (step S32), the quadrilateral extraction unit 21b performs quadrilateral extraction processing on the image (real-time image) in the photographing range (the range displayed on the display unit 30) ( Step S33).

  If no quadrilateral is obtained by the processing of the quadrilateral extraction unit 21b, the optical zoom is automatically zoomed out by one step (step S34). Then, on the display unit 30, for example, as shown in FIG. 17, the characters “cannot be corrected. Please press the shutter button again” are superimposed on the real-time image for several seconds. Then, returning to step S31, the photographer again presses the shutter button 32 halfway (step S32). Then, again, the quadrilateral extraction unit 21b performs quadrilateral extraction processing on the real-time image in step S33 (step S33). As a result, when the quadrilateral is not obtained by the process of the quadrilateral extraction unit 21b, the processes of step S34, step S31, step S32, and step S33 are repeated again. Thereafter, S34, S31, S32, and S33 are repeated until a quadrilateral is obtained in the process of step S33.

  FIG. 18 shows an example of a real-time image when a quadrilateral is obtained by the processing of the quadrilateral extraction unit 21b. FIG. 18 is an example in which the surrounding quadrilateral is included in the shooting range by the automatic zoom-out (step S34) from the shooting range example of FIG.

  In step S33, when a quadrilateral (a quadrilateral 109 in the example of FIG. 18) is extracted by the quadrilateral extraction unit 21b, the text “Please press the shutter button” is displayed on the display unit 30 as shown in FIG. Superimposed. Then, the photographer further depresses the shutter button 32 from the half-pressed state, and pushes down the shutter button 32 (step S35).

  Next, the quadrilateral extraction unit 21b performs quadrilateral extraction on the captured image (step S36). The quadrilateral extraction unit 21b has the same function as the quadrilateral extraction unit of the first embodiment, and extracts one quadrilateral used as a marker. In the example of FIG. 18, a quadrilateral 109 is extracted.

  Next, using the quadrilateral extracted by the quadrilateral extraction unit 21b (the quadrilateral 109 in the example of FIG. 18), the trimming region calculation unit 27b performs image trimming (step S37). First, the trimming area calculation unit 27b sets, as the initial trimming area 110, an area obtained by removing a rectangular area including a quadrilateral from the entire imaging area (FIG. 20).

  Next, the projective transformation processing unit 26b and the trimming area calculating unit 27b perform the projective transformation as follows based on the quadrilateral extracted by the quadrilateral extracting unit 21b (step S38). In the present embodiment, projective transformation is performed so that the saved image after tilt correction is rectangular.

  First, the projective transformation processing unit 26b calculates a transformation matrix 200 for tilt correction based on the quadrilateral. Next, the trimming area calculation unit 27b calculates a converted initial trimming area 111 obtained by performing projective transformation on the initial trimming area 110 using the projective transformation matrix 200 (FIG. 21). Then, a converted inscribed trimming region 112 that is a rectangle inscribed in the converted initial trimming region 111 is calculated. Next, a projective transformation matrix 201 that is an inverse matrix of the projective transformation matrix 200 is obtained. Next, an area obtained by inversely transforming the converted inscribed trimming area 112 using the projective transformation matrix 201 is calculated, and the calculated area is set as an inscribed trimming area 113 (FIG. 22). Then, the projective transformation processing unit 26b performs projective transformation on the input image corresponding to the calculated inscribed trimming region 113. Finally, the projective transformation processing unit 26b stores the projective transformed image in the processed image storage unit 24a.

  In this embodiment, the trimming area calculation unit 27b creates the inscribed trimming area 113. However, the converted circumscribed trimming area 115 is used instead of the converted inscribed trimming area 112 as shown in FIG. The circumscribing trimming region 115 may be created by the projective transformation matrix 201. Further, the initial trimming area 110 may be directly projective transformed as in the first embodiment.

  In the present embodiment, the quadrilateral extraction is performed on the captured image to obtain the projective transformation matrix, but the projection is performed using the quadrilateral extracted by the quadrilateral extraction (shutter half-pressed state) in step S33. A conversion matrix may be obtained. In the case of this method, it is necessary to maintain the imaging position so that the tilt correction conditions are the same between the half-pressed state and the shutter-released state. For example, it can be realized by fixing the imaging device 1b to a tripod or the like, placing a marker in the imaging range until extracting the quadrilateral, and then removing the marker. According to this method, unnecessary markers can be removed from the image without trimming.

  As described above, according to the present embodiment, a photographer selects a shooting area using digital zoom, and if a quadrangle serving as a marker is not included, it is selected again after zooming out. As a result, the correction can be performed without requiring a complicated operation, and an unnecessary marker is not included in the corrected stored image.

  The other configuration and processing of the imaging device 1b according to the third embodiment are the same as the configuration and processing of the imaging device 1 according to the first embodiment or the imaging device 1a according to the second embodiment. And

(Fourth embodiment)
FIG. 24 is a block diagram illustrating a configuration example of the image processing device 2 according to the fourth embodiment of the present invention. The same parts as those in the first embodiment are denoted by the same reference numerals as those in FIG. The image processing apparatus 2 according to the present embodiment is obtained by removing the imaging unit from the imaging apparatus according to the first embodiment.

  An input image that has been photographed with a digital camera or the like before tilt correction is stored in the external storage device 17 after photographing. The subsequent processing is the same as in the first embodiment.

  In the present embodiment, the image processing apparatus is dedicated hardware, but it can also be realized using hardware resources such as a general-purpose computer.

  As described above, according to the present embodiment, since the image processing apparatus performs correction, there is an effect that the correction can be performed at high speed.

  The present invention is useful for an image processing method, an imaging apparatus, an image processing apparatus, and a program, and is particularly useful for an image processing method, an imaging apparatus, an image processing apparatus, and a program that perform correction of tilt on an input image.

It is a block diagram which shows the function structural example of the imaging device of 1st Embodiment. It is a block diagram which shows the function structural example concerning the tilt correction program of 1st Embodiment. It is a flowchart of a 1st embodiment. It is a figure which shows the example of an image image | photographed by arrange | positioning a marker beside a to-be-photographed object. It is a figure explaining the trimming area | region of 1st Embodiment. It is a block diagram which shows the function structural example of the imaging device of 2nd Embodiment. It is a rear view of the example of apparatus composition of a 2nd embodiment. It is a block diagram which shows the function structural example concerning the tilt correction program of 2nd Embodiment. It is a figure which shows the example of a picked-up image which put the marker in the vicinity of a to-be-photographed object. It is a flowchart of a 2nd embodiment. It is a figure which shows the example of a display of the digital zoom area | region of 2nd implementation. It is a figure which shows the example of the preservation | save image after the projective transformation of 2nd implementation. It is a block diagram which shows the function structural example of the imaging device of 3rd Embodiment. It is a block diagram which shows the function structural example concerning the tilt correction program of 3rd Embodiment. It is a flowchart of a 3rd embodiment. It is a figure which shows the example of a picked-up image which does not contain the quadrilateral used as a marker. It is a figure which shows the example of a warning display of 3rd Embodiment. It is a figure which shows the example of a message image containing the quadrilateral used as a marker. It is a figure which shows the example of a display of 3rd Embodiment. It is a figure explaining the initial trimming area | region of 3rd implementation. It is a figure explaining the projective transformation of an initial trimming area. It is a figure explaining the inscribed trimming area | region and inscribed trimming area | region after conversion. It is a figure explaining the circumscribed trimming area | region and circumscribed trimming area | region after conversion. It is a block diagram which shows the function structural example of the image processing apparatus of 4th Embodiment.

Explanation of symbols

1, 1a, 1b Imaging device 2 Image processing 11 Imaging unit 12 Camera signal processing unit 13 CPU
14 frame memory 15 I / F
16 Display unit 17 External storage device 18 ROM
20 Lens 21 Aperture 22 Shutter 23 Photoelectric conversion element 24 Preprocessing unit 25, 25a, 25b Quadrilateral extraction unit 26, 26a Trimming unit 26b Trimming area calculation unit 27, 27a Projection conversion processing unit 28 Trimming area storage unit 29, 29a Processed image Storage unit 30 Display unit 31 Zoom-in / out button 32 Shutter button 33 Viewfinder 34 Up button 35 Right button 36 Down button 37 Right button 38 Enter button 100 Image 101, 105, 108 Subject 102, 106 Marker 103 Trimming region 104 Non-trimming region 107 Digital zoom Area 109 quadrangle 110 initial trimming area 111 initial trimming area after conversion 112 inscribed trimming area after conversion 113 inscribed trimming area 114 circumscribed trimming area after conversion 115 circumscribing the trimming area 200, 201 transformation matrix

Claims (13)

An image processing method that is executed by an image processing apparatus and performs tilt correction,
A reference graphic extracting step for extracting the reference graphic from image data including a subject to be corrected for tilt and a reference graphic used for tilt correction;
A trimming step in which the trimming means of the image processing apparatus trims the image data into an area not including the reference graphic;
A conversion matrix calculating step in which the conversion matrix calculating means of the image processing apparatus calculates a conversion matrix based on the reference graphic;
Projective transformation means of the image processing apparatus, the image data, seen including a projective conversion step of projective transformation, a using the transformation matrix,
The image processing method according to claim 1, wherein the reference graphic extracting means evaluates the size or contrast strength of a graphic included in the image data and selects a graphic having the highest evaluation value as the reference graphic .   2. The image processing method according to claim 1, wherein in the projective transformation step, the trimmed image is subjected to the projective transformation.   2. The image processing method according to claim 1, wherein in the trimming step, the image data subjected to the projective transformation is trimmed into an area not including the reference graphic.   3. The image processing method according to claim 1, wherein the area to be trimmed is an area obtained by removing a rectangular area including the reference graphic from an entire image area. An image processing method that is executed by an image processing apparatus and performs tilt correction,
A reference graphic extracting step for extracting the reference graphic from image data including a subject to be corrected for tilt and a reference graphic used for tilt correction;
A trimming step in which the trimming means of the image processing apparatus trims the image data into an area not including the reference graphic;
A conversion matrix calculating step in which the conversion matrix calculating means of the image processing apparatus calculates a conversion matrix based on the reference graphic;
A projective transforming step of projecting transforming the image data using the transform matrix, the projective transforming means of the image processing device,
The trimming means trims the image data according to a digital zoom operation by a user,
The image processing method according to claim 1, wherein the reference graphic extraction means extracts a graphic from an area removed from the trimming area by the digital zoom and selects it as the reference graphic . An imaging device that performs tilt correction,
Imaging means for imaging a shooting target region including a subject to be corrected for tilt and a reference graphic used for tilt correction;
Reference graphic extraction means for extracting the reference graphic from the image data captured by the imaging means;
Trimming means for trimming the image data into an area not including the reference graphic;
Transformation matrix calculation means for calculating a transformation matrix based on the reference figure;
A projective transforming means for projectively transforming the image data using the transform matrix ;
The imaging apparatus according to claim 1, wherein the reference graphic extraction means evaluates the size or contrast strength of a graphic included in the image data, and selects a graphic having the highest evaluation value as the reference graphic .   The imaging apparatus according to claim 6, wherein the projective transformation unit performs the projective transformation on a trimmed image.   The imaging apparatus according to claim 6, wherein the trimming unit trims the image data subjected to the projective transformation into an area not including the reference graphic. An imaging device that performs tilt correction,
Imaging means for imaging a shooting target region including a subject to be corrected for tilt and a reference graphic used for tilt correction;
Reference graphic extraction means for extracting the reference graphic from the image data captured by the imaging means;
Trimming means for trimming the image data into an area not including the reference graphic;
Transformation matrix calculation means for calculating a transformation matrix based on the reference figure;
A projective transforming means for projectively transforming the image data using the transform matrix;
The trimming means trims the image data according to a digital zoom operation by a user,
The imaging apparatus according to claim 1, wherein the reference graphic extraction means extracts a graphic from an area removed from the trimming area by the digital zoom and selects it as the reference graphic . Display means for simultaneously displaying the area to be imaged and the trimming area;
The imaging apparatus according to claim 9, further comprising: An optical zoom means capable of changing a focal length by a user operation;
A shooting target area temporary setting means for temporarily setting a shooting target area;
With
The reference graphic extraction means extracts the reference graphic from the image data of the temporarily set shooting target area,
9. The optical zoom unit according to claim 6, wherein when the reference graphic is not extracted by the reference graphic extraction unit in the temporarily set photographing target area, the optical zoom unit automatically zooms out. The imaging device as described in any one. An image processing apparatus that performs tilt correction,
Reference graphic extraction means for extracting the reference graphic from image data including a subject to be corrected for tilt and a reference graphic used for tilt correction;
Trimming means for trimming the image data into an area not including the reference graphic;
Transformation matrix calculation means for calculating a transformation matrix based on the reference figure;
A projective transforming means for projectively transforming the image data using the transform matrix ;
The image processing apparatus, wherein the reference graphic extracting means evaluates the size or contrast strength of a graphic included in the image data, and selects a graphic having the highest evaluation value as the reference graphic . A function as a reference graphic extraction means for extracting the reference graphic from image data including a subject to be corrected and a reference graphic used for the correction;
A function as a trimming means for trimming the image data into an area not including the reference graphic;
A function as a conversion matrix calculation means for calculating a conversion matrix based on the reference figure;
A function as a projective transformation means for projective transformation of the image data using the transformation matrix;
Is a program for causing a computer to realize
The reference graphic extracting means evaluates the size or contrast strength of a graphic included in the image data, and selects a graphic having the highest evaluation value as the reference graphic .

Images