JP4591343B2 - Image processing apparatus, imaging apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, an image processing method, and a program.

In recent years, digital data for data management is easier to search and manage, and therefore various documents have been converted to data. Postcards that exist in ordinary households are no exception. In general, a flatbed scanner is used to convert the postcard into data (see, for example, Patent Document 1).
JP-A-10-294854 (page 4, FIG. 1)

  However, this flatbed scanner is not yet suitable for general use as compared with digital cameras that have been widely used for photographing in recent years, and its diffusion rate is low.

  Therefore, although it is conceivable to convert the postcard into data using this digital camera, when using a digital camera, the image of the postcard must be distorted or the images on the front and back of the postcard must be managed individually. There are still many problems with using a digital camera to create postcard data.

  The present invention has been made in view of the above-described conventional problems. An image processing apparatus, an imaging apparatus, an image processing method, and an image processing apparatus capable of photographing the front and back surfaces of a document and making the data easy to manage. The purpose is to provide a program.

In order to achieve this object, an image processing apparatus according to the first aspect of the present invention provides:
An image processing apparatus for processing a standard document image in which characters are described on the front surface or the back surface, and a preset identification mark is described in a preset area on the front surface,
The image obtained form letters by the photographing, corrected in a shape corresponding to the shape on the fixed document, the shape on the basis of the corrected image, recognizes characters written on the fixed document, character The orientation of the standard document is determined based on the recognition rate, the orientation of the image of the standard document is adjusted based on the determination result, the region set in advance on the surface is determined, and the identification mark is detected in the determined region The photographed image obtained by the photographing is determined to be the front image of the standard document, and when the identification mark is not detected in the determined area, the photographed image is the back image of the standard document. a captured image correcting unit for judging,
Characterized in that and an image combining unit for combining the said rear surface image and the surface image of the standard document in which the captured image correcting unit determines.

  The captured image correction unit acquires a binary edge image from the front image and the back image of the document, acquires the contour of the acquired binary edge image, and calculates the vertex coordinates of the image from the acquired contour. Using the acquired vertex coordinates, the shape of the image may be subjected to projective transformation to the shape corresponding to the document, thereby correcting the shape of each image to the shape corresponding to the document.

An image pickup apparatus according to a second aspect of the present invention provides:
An imaging device for photographing a standard document image in which characters are described on the front surface or the back surface, and a preset identification mark is described in a preset area on the front surface,
A photographing unit for photographing the standard document ;
An image processing unit that performs image processing on an image of a standard document obtained by photographing of the photographing unit,
The image processing unit
The image of the standard document obtained by the photographing, corrected in a shape corresponding to the shape on the fixed document, the shape on the basis of the corrected image, recognizes characters written on the standard document, The orientation of the standard document is determined based on the character recognition rate, the orientation of the image of the standard document is adjusted based on the determination result, the region set in advance on the surface is determined, and the identification mark is added to the determined region. When detected, the photographed image obtained by the photographing is determined as the front image of the standard document, and when the identification mark is not detected in the determined area, the photographed image is the back image of the standard document. a captured image correcting section that determines that,
The image synthesizing section, characterized in that the captured image correcting section is provided with an image combining unit for combining with the surface image and the back image of the standard document determined.

An image processing method according to a third aspect of the present invention includes:
An image processing method for processing a standard document image in which characters are described on the front surface or the back surface, and a preset identification mark is described in a preset area on the front surface,
The image obtained form letters by the photographing, corrected in a shape corresponding to the shape on the fixed document, the shape on the basis of the corrected image, recognizes characters written on the fixed document, character The orientation of the standard document is determined based on the recognition rate, the orientation of the image of the standard document is adjusted based on the determination result, the region set in advance on the surface is determined, and the identification mark is detected in the determined region The photographed image obtained by the photographing is determined to be the front image of the standard document, and when the identification mark is not detected in the determined area, the photographed image is the back image of the standard document. a captured image correcting step of determining,
An image combining step of combining the corrected front surface image and back surface image;

The program according to the fourth aspect of the present invention is:
In the computer of the image processing apparatus that processes the standard document image in which characters are described on the front surface or the back surface, and a preset identification mark is described in a predetermined area on the front surface ,
The image obtained form letters by the photographing, corrected in a shape corresponding to the shape on the fixed document, the shape on the basis of the corrected image, recognizes characters written on the fixed document, character The orientation of the standard document is determined based on the recognition rate, the orientation of the image of the standard document is adjusted based on the determination result, the region set in advance on the surface is determined, and the identification mark is detected in the determined region The photographed image obtained by the photographing is determined to be the front image of the standard document, and when the identification mark is not detected in the determined area, the photographed image is the back image of the standard document. and procedures to determine,
The photographing the surface image of the standard document image correcting unit determines a step of synthesizing the said back surface image is intended for causing the execution.

  According to the present invention, a front image and a back image of a document can be taken and converted into data so as to be easily managed.

Hereinafter, an imaging device according to an embodiment of the present invention will be described with reference to the drawings. In the present embodiment, the imaging apparatus is described as a digital camera.
A configuration of a digital camera 1 according to the present embodiment is shown in FIG.
As shown in FIG. 1, the digital camera 1 according to the present embodiment photographs the front and back surfaces of a postcard 2. This postcard 2 is a standard document. That is, the postcard 2 has a rectangular shape with a long side and a short side, and a stamp or a charge as an identification mark is described in a preset area on the surface of the postcard 2.
According to this description, a stamp of a circle having a diameter of 20 to 30 mm or a rectangle having a length and width of 20 to 30 mm is stamped at the station.

  This postcard 2 may be used in vertical writing or in horizontal writing. The area for describing stamps and fees is set in advance, and when used in vertical writing, this area exists at the upper left of postcard 2, and when used in horizontal writing, this area is the upper right of postcard 2. Exists.

  The digital camera 1 according to the present embodiment acquires the contours of the front and back surfaces from the photographed image of the postcard 2 and performs oblique photographing correction on both images of the postcard 2 so that the front and back images of the postcard 2 are obtained. Are combined into one image and stored.

  By performing such processing, the digital camera 1 converts the contents of the postcard 2 into data without using a scanner or a computer. As shown in FIG. 2, the digital camera 1 includes an image data generation unit 10, a data processing unit 20, and a user interface unit 30.

  The image data generation unit 10 is composed of an optical lens device 11 and an image sensor 12, and photographs the postcard 2 and converts the contents of the postcard 2 into digital data.

  The digital camera 1 is capable of high-resolution image shooting and low-resolution image shooting (preview shooting). The low-resolution image shooting is a shooting that enables moving image shooting and image reading at a speed of 30 fps (frame / second) although the image resolution is about XGA (1024 × 768 dots) and the resolution is low.

  The high-resolution image capturing is, for example, image capturing in which the maximum number of pixels that can be captured, for example, when the digital camera 1 is a 4 million pixel camera, is captured at a resolution of 4 million pixels. The image data generation unit 10 performs high-resolution image shooting and low-resolution image shooting.

  The optical lens device 11 is composed of a lens that collects light in order to photograph the postcard 2, and includes peripheral circuits for adjusting camera setting parameters such as focus, exposure, and white balance.

  The image sensor 12 is for capturing an image formed by condensing light by the optical lens device 11 as digitized image data, and is configured by a CCD (Charge Coupled Device) or the like. The

  The user interface unit 30 includes a display unit 31, an operation unit 32, a PC interface device 33, and an external storage unit 34.

  The display unit 31 is for displaying an image and includes a liquid crystal monitor or the like. The display unit 31 reads the image data from the memory 21 of the data processing unit 20 and displays the preview image of the postcard 2, the binary edge image, the outline of the postcard 2 acquired from the image of the postcard 2, and the like.

  The operation unit 32 is for acquiring operation information of the operator. As a switch and a key for controlling the function of the digital camera 1, a power switch, a shutter, a correction key, a cursor key, and a determination are determined. And a key (not shown).

  The power switch is for turning on (ON) and turning off (OFF) the power of the digital camera 1.

  The shutter is a switch that is pressed during shooting. The correction key is a key for instructing oblique shooting correction for the image of the postcard 2.

  The cursor key is a key for selecting a photograph, a menu, and the like, and is also used as a key for selecting this direction when instructing correction of the direction of the synthesized postcard 2. The determination key is a key for determining the selected instruction.

  When the operator operates these switches and keys, the operation unit 32 supplies the operation information to the CPU 25 of the data processing unit 20.

The PC interface device 33 converts image data or the like into USB (Universal Serial Bus) standard data, transmits the data to a computer (not shown) via a USB cable or the like, and receives data and signals from the computer. Is for.
The external storage unit 34 is for storing the synthesized image of the postcard 2.

  The data processing unit 20 acquires image data from the image data generation unit 10 and processes an image for output to the projector 4. The data processing unit 20 includes a memory 21, a video output device 22, an image processing device 23, a program code storage device 24, and a CPU 25.

  The memory 21 stores image data, various threshold values, and the like. As shown in FIG. 3, the memory 21 has a sensor image storage area 21a, a processed image storage area 21b, a display image storage area 21c, a work data storage area 21d, and a threshold value storage. And a region 21e.

  The sensor image storage area 21a is an area for temporarily storing image data captured by the image sensor 12 every time it is photographed. The image sensor 12 stores the image data in the sensor image storage area 21a.

  The processed image storage area 21b is an area for storing image data necessary for processing of the image processing apparatus 23. The display image storage area 21c is an area for storing an image of a display output image. The work data storage area 21d is an area for storing coordinate data and the like. The threshold storage area 21e is an area for storing a threshold used for various determinations.

  Returning to FIG. 2, the video output device 22 is controlled by the CPU 25, and generates an RGB signal based on the image data stored in the display image storage area 21 c of the memory 21. When connected to a projector (not shown) via a video video cable, the video output device 22 outputs the generated RGB signal to the projector. Note that the video output device 22 may be configured to send RGB signals to the computer.

  The image processing device 23 is controlled by the CPU 25 to perform image processing on image data. The image processing device 23 performs the following image processing in order to realize the functions of the digital camera 1.

(1) Contour acquisition processing (2) Oblique photographing correction processing (3) Front / back and top / bottom detection processing (4) Image composition processing

Hereinafter, the processes (1) to (4) will be described.
(1) Contour Acquisition Processing The contour acquisition processing is processing for acquiring the contour of the postcard 2 image. In order to acquire the contour of the image of the postcard 2, the image processing device 23 first detects an edge from the preview image of the postcard 2 as shown in FIG. 4A, and as shown in FIG. A binary edge image is created.

  The image processing device 23 uses, for example, a filter for edge detection called a Roberts filter from the binarized image generated in this way. The Roberts filter is a filter that detects the edge of an image by weighting two 4-neighboring pixels to obtain two filters Δ1 and Δ2 and averaging them.

画像処理装置２３は、このようにして得られた画素値ｇ（x,y）を予め設定された閾値を用いて２値化する。閾値は、固定的に設定されてもよいし、可変閾値方等の方法で求められたものであってもよい。この閾値は、メモリ２１の閾値記憶領域２１ｅに予め記憶される。 When the Roberts filter is applied to the pixel value f (x, y) at a certain coordinate (x, y), the converted pixel value g (x, y) is expressed by the following equation (1).

The image processing device 23 binarizes the pixel value g (x, y) obtained in this way using a preset threshold value. The threshold value may be set in a fixed manner or may be obtained by a method such as a variable threshold value method. This threshold value is stored in advance in the threshold value storage area 21e of the memory 21.

画像処理装置２３は、生成したエッジ画像に対して、ハフ変換を行い、葉書２の輪郭を形成する直線を検出する。 The image processing device 23 obtains an edge binary image h (x, y) by performing the following calculation 2 on the edge image g (x, y) obtained according to Equation 1.

The image processing device 23 performs Hough transform on the generated edge image and detects a straight line that forms the outline of the postcard 2.

The Hough transform is a point that forms a straight line on the XY plane as shown in FIG. 5A on the ρ-θ plane as shown in FIG. This is a conversion method for voting and converting the number of votes on the ρ-θ plane.

  When the angle θ is changed from 0 to 360 ° in the coordinates (x, y) of each point, the same straight line is represented by one point on the ρ-θ plane. For this reason, the ρ-θ coordinate having a large number of votes can be determined as a straight line. At this time, since the number of votes is the number of pixels on the straight line, it can be regarded as the length of the straight line. Therefore, the ρ-θ coordinate with an extremely small number of votes represents a short straight line and is excluded from the straight line candidates.

  Considering the coordinate system with the image center as the origin in the edge image to be investigated, ρ also takes a negative value. Therefore, if the angle θ is measured in the range of 0 ° ≦ θ <180 °, ρ is It becomes negative in the remaining range of −180 ° ≦ θ <0 °.

However, when the center of the shooting target is located near the center of the image, each side of the shooting target (rectangle) that is actually shot exists vertically and horizontally. In this case, it is more efficient to measure the number of votes on the ρ-θ plane in the range represented by the following formula 4 than to investigate in the range of 0 ° ≦ θ <180 °.

It is possible to specify the top and bottom of the side or the left and right of the side depending on the limit of the investigation angle and whether the value of ρ is a positive value or a negative value.
In that case, the absolute value of ρ is the distance from the center. The straight line candidate having the largest absolute value of the distance ρ from the center among the straight line candidates having a large number of votes can be said to be the outline of the image of the postcard 2. However, it is assumed that the intersection, that is, the vertex exists only in the image.

(2) Diagonal Shooting Correction Process The oblique shooting correction process is a process that performs image clipping and projection correction. The cut-out process is a process for cutting out the front image and the back image of the postcard 2 from the photographed image based on the extracted quadrilateral outline. Usually, the image of the cut-out postcard 2 is distorted.

  The projection correction process is a process in which the image of the cut postcard 2 is subjected to projective transformation to correct image distortion. In order to correct this distortion, the image processing apparatus 23 uses affine transformation widely applied to image spatial transformation.

  The image processing device 23 calculates an affine parameter A indicating the relationship of the affine transformation with the projection corrected image from the contour (four vertices) extracted from the image such as the postcard 2 and uses this transformation A to obtain the projection correction to be obtained. A pixel point p (x, y) of the original image corresponding to each pixel P (u, v) of the image is obtained.

Next, the basic concept (implementation method) of this affine transformation will be described.
Affine transformation is widely applied to image spatial transformation. In this embodiment, projective transformation is performed using two-dimensional affine transformation without using three-dimensional camera parameters.

最終的な座標(x,y)は、次の数６によって算出される。

The point of the coordinate (u, v) before conversion becomes the coordinate (x, y) after conversion by performing conversion such as movement, enlargement / reduction, and rotation. The coordinates (u, v) before conversion and the coordinates (x, y) after conversion are related by the following equation (5). Projective transformation is also performed by this affine transformation.

The final coordinates (x, y) are calculated by the following equation (6).

  Equation 6 is an expression for projective transformation, and the coordinates (x, y) are degenerated toward 0 according to the value of z ′. That is, the parameter included in z ′ affects the projection. This parameter is a13, a23, a33. Since other parameters can be normalized by the parameter a33, a33 may be 1.

  FIG. 6 is a diagram for explaining the relationship between the quadrangle on the photographed image and the actual shape of the photographed object when the camera unit 11 photographs the photographed object (postcard 2). The coordinates of each vertex are shown in the figure.

  In FIG. 6, the UVW coordinate system is a three-dimensional coordinate system of an image obtained by photographing with the digital camera 1. The A (Au, Av, Aw) vector and the B (Bu, Bv, Bw) vector represent the object to be photographed as vectors in the three-dimensional coordinate system UV-W. The S (Su, Sv, Sw) vector indicates the distance between the origin of the three-dimensional coordinate system UV-W and the object to be imaged.

  The virtual projection screen of the captured image shown in FIG. 6 virtually shows an image projected by the digital camera 1. Assuming that the coordinate system on the screen is (x, y), the image projected on the screen is an image taken by the digital camera 1.

数７より、図６に示すように４つの頂点Ｐ0，Ｐ1，Ｐ2，Ｐ3と投影スクリーンへの投影点ｐ0，ｐ1，ｐ2，ｐ3との関係から、次の数８に示す関係が求められる。

このとき、射影係数α、βは次の数９によって表される。

It is assumed that the projection screen is positioned vertically at a distance f from the W axis. An arbitrary point P (u, v, w) on the object to be photographed is connected to the origin by a straight line, and there is a point intersecting the straight line and the projection screen, and the XY coordinate of the intersection is represented by p (x, y). At this time, the coordinate p is expressed by the following equation 7 from the projective transformation.

From Equation 7, the relationship shown in the following Equation 8 is obtained from the relationship between the four vertices P0, P1, P2, P3 and the projection points p0, p1, p2, p3 on the projection screen as shown in FIG.

At this time, the projection coefficients α and β are expressed by the following equation (9).

この数１０に、数８の関係式を代入すると、座標ｘとｙとは、次の数１１によって表される。

この関係を、アフィン変換の式（数５）に当てはめると、座標（x',y',z'）は、次の数１２によって表される。

この数１２にｍ，ｎを代入することにより、撮影画像の対応点（x,y）が求められる。対応点（x,y）は、整数値とは限らないので、画像補間法などを用いて画素の値を求めればよい。 Next, projective transformation will be described. An arbitrary point P = (x, y) on the object to be imaged is expressed by the following equation 10 using S, A, B vectors.

When the relational expression of Expression 8 is substituted into Expression 10, the coordinates x and y are expressed by Expression 11 below.

When this relationship is applied to the affine transformation formula (formula 5), the coordinates (x ′, y ′, z ′) are expressed by the following formula 12.

By substituting m and n into Equation 12, the corresponding point (x, y) of the captured image is obtained. Since the corresponding point (x, y) is not necessarily an integer value, the pixel value may be obtained using an image interpolation method or the like.

The above m and n are obtained in advance by giving an image size (0 ≦ u <umax, 0 ≦ v <vmax) for outputting the corrected image p (u, v) and adjusting the image according to the image size. It is possible. According to this method, m and n are expressed by the following equation (13).

However, the aspect ratio of the corrected image to be created does not match the aspect ratio of the shooting target. Here, the relationship between the corrected image p (u, v) and the values of m and n is expressed by the following Expression 14 from Expressions 7 and 8.

If the focal length f of the lens, which is a camera parameter, is known, the aspect ratio k can be obtained according to Equation 14. Therefore, if the image size of the corrected image p (u, v) is (0 ≦ u <umax, 0 ≦ v <vmax), m and n are obtained according to the following equation 15 to obtain the same vertical and horizontal dimensions as the subject to be imaged. The ratio k can be obtained.

  If the camera has a fixed focus, the value of the focal length f of the lens can be obtained in advance. When a zoom lens or the like is present, the value of the focal length f of the lens changes depending on the zoom magnification of the lens. In this case, a table indicating the relationship between the zoom magnification and the focal length f of the lens is created and stored in advance, the focal length f is read based on the zoom magnification, and projective transformation is performed according to equations 14 and 15.

  The oblique shooting correction uses (1) four points (x0, y0), (x1, y1), (x2, y2), (x3, y3) calculated by contour detection, and (2) these coordinates. Is substituted into Equations 9 and 12, and the coordinates (x ′, y ′) in the original image are calculated from the corrected coordinates (m, n).

  The image size is calculated by performing calculation using Expression 12 on the assumed image size using Expressions 13, 14, and 15.

(3) Front / Back and Upper / Lower Detection Processing The image processing device 23 detects the front / back and upper limit of the postcard 2. In the present embodiment, the image processing device 23 detects the top and bottom of the postcard 2 by performing character recognition. The image processing device 23 performs character recognition from the top, bottom, left, and right directions on the image after oblique photographing correction, and the direction with the highest recognition rate is the correct orientation.

  As the recognition rate, specifically, there are specific characters such as an address, “˜”, “˜” on the surface, and these specific characters described in the postcard 2 by the operator in advance. The digital camera 1 is stored in the work data storage area of the memory 21. The image processing device 23 obtains the recognition rate by determining how many of these specific characters are among the recognized characters.

  When the image processing device 23 detects the top and bottom of the postcard 2, it recognizes the top and bottom and then detects the back and front of the postcard 2. As described above, there is an area on the surface of the postcard 2 where a description regarding stamps and fees is made. The image processing device 23 detects the front and back of the postcard 2 by determining the presence or absence of this region.

  As described above, the station mark has a diameter of 20 to 30 mm in the case of a circle, and 20 to 30 mm in length and width in the case of a square. The image processing apparatus 23 sets the area where the station mark is pressed, for example, 35 × 70 mm.

  This region exists at the upper left when the short side is upward as shown in FIG. 7A and at the upper right when the long side is upward as shown in FIG. 7B. The image processing device 23 searches whether there is an effective pixel satisfying this condition in this area while shifting the center of the circle or the rectangle.

  However, the image processing device 23 may detect the station mark using the Hough transform method. In this case, the image processing device 23 detects the rectangle of the station mark by using the Hough transform method on the image after the oblique photographing correction of the area of the postcard 2 from the photographed image.

Further, the image processing device 23 detects the circle of the station mark by using Equation 16 instead of Equation 3 of the Hough transform for straight line detection.

Here, as shown in FIG. 8, a point on the coordinates existing on a circle whose radius is r and whose center point is P (x _p , y _p ) is defined as (x _i , y _i ).

The same circle on the orthogonal coordinates shown in FIG. 8 is represented by one point in the three-dimensional space (x _p , y _p , r). Therefore, it can be determined as a point (x _p , y _p , r) on the three-dimensional space with a large number of votes.

  When a rectangle or a circle is detected within a designated area in the image, the image processing device 23 determines that this surface is a surface with an address.

(4) Image Compositing Process The image compositing process is a process of combining two images on the front and back of the postcard 2. As shown in FIG. 8, the image processing device 23 arranges the front image on the left side and the back image on the right side, and when the postcard 2 is used vertically as shown in FIG. As shown in FIG. 9 (b), the long sides of the front surface and the back surface of the postcard 2 are made to correspond to each other.

  Further, when the postcard 2 is used horizontally as shown in FIG. 10A, the image processing device 23 has a short side between the front surface and the back surface of the postcard 2 as shown in FIG. 10B. Are combined.

In this way, the image processing apparatus 23 combines the front and back surfaces of the postcard 2 to generate one piece of image data.
The above is the description of the processes (1) to (4) of the image processing apparatus 23.

  Returning to FIG. 2, the program code storage device 24 is for storing a program executed by the CPU 25, and is configured by a ROM or the like.

  The CPU 25 controls each unit in accordance with a program stored in the program code storage device 24.

  Specifically, the CPU 25 executes camera basic processing described later, and first initializes the shooting conditions of the camera unit 11. The shooting conditions include focus of the camera unit 11, shutter speed (exposure time), exposure such as an aperture, white balance, ISO sensitivity, and the like. The CPU 25 performs autofocus measurement, photometry, and the like to search for conditions under which shooting is optimal, and initializes the image data generation unit 10 to satisfy these conditions.

  The CPU 25 also initializes data used for initialization of the memory 21, communication, image processing, and the like.

  Further, when the operation information is supplied from the operation unit 32 of the user interface unit 30, the CPU 25 controls each unit according to the operation information.

  That is, when operation information indicating that the shutter button has been pressed is supplied from the operation unit 32, the CPU 25 controls the image data generation unit 10 so as to capture a low-resolution image.

  The CPU 25 controls the image sensor 12 and the memory 21 so that the image data captured by the image sensor 12 is stored in the sensor image storage area 21 a of the memory 21.

  The CPU 25 stores, as images stored in each storage area of the memory 21, a low-resolution preview image, an image subjected to oblique photographing correction processing, front / back and top / bottom detection processing, an image in which the orientation of the image of the postcard 2 is corrected, and a postcard The display unit 31 of the user interface unit 30 is controlled so as to display an image obtained by combining the two front and back images and a saved image.

  In addition, when operation information indicating that the correction key has been pressed is supplied from the operation unit 32, the CPU 25 instructs the image processing device 23 to execute oblique photographing correction processing, front / back and top / bottom detection processing.

  When the operation information indicating that the determination key for saving the image is pressed is supplied from the operation unit 32, the CPU 25 stores the image data in the external storage unit 34 of the user interface unit 30.

The CPU 25 determines whether or not two images of the postcard 2 are photographed. For this reason, the CPU 25 includes a counter and counts the number of times the image data generating unit 10 has taken images. The CPU 25 stores the count value of this counter in the work data storage area 21 d of the memory 21.
When the count value reaches 2, the CPU 25 determines that the front and back sides of the postcard 2 have been photographed, and instructs the image processing device 23 to execute the image composition processing.

Next, the operation of the digital camera 1 according to this embodiment will be described.
When the operator turns on the power of the digital camera 1, the CPU 25 of the digital camera 1 reads the program code from the program code storage device 24 and executes camera basic processing.

The contents of this camera basic projection process will be described with reference to the flowcharts shown in FIGS.
The CPU 25 controls peripheral circuits of the optical lens device 11 to perform camera settings such as the focus, zoom lens position, and aperture of the optical lens device 11, and initialization of the video output device 22 and the like (step S11).

  The CPU 25 initializes data used for initialization of the memory 21, communication, image processing, and the like (step S12).

  The CPU 25 displays a preview image on the display unit 31 and waits until operation information indicating that the operator has pressed the shutter is supplied (step S13).

  When the operator depresses the shutter, the operation unit 32 supplies the operation information to the CPU 25, and the CPU 25 controls the image generation unit to perform photographing (step S14).

  The CPU 25 instructs the image processing device 23 to perform contour acquisition processing (step S15).

  The image processing apparatus 23 is controlled by the CPU 25 and executes the contour acquisition process according to the flowchart shown in FIG.

The image processing device 23 creates a binary edge image according to Equations 1 and 2 (step S41).
The image processing device 23 performs Hough transform using Equation 3 in order to detect a straight line (step S42).

  The image processing apparatus 23 acquires a plurality of coordinates as a contour acquisition candidate with coordinates having a large number of votes in the range of 45 ° ≦ θ225 ° and each of ρ is + and − (step S43). At this time, a candidate whose ρ is + corresponds to the upper side, and a candidate whose ρ is − corresponds to the lower side.

The image processing device 23 creates a voting table in descending order of voting (step S44).
The image processing device 23 rearranges the acquired candidates in order from the coordinate having the maximum distance from the center in the upper, lower, left, and right directions (step S45).

The image processing device 23 calculates the intersections of the upper, lower, left, and right side candidates, and sets the calculated intersections as the four vertices of the contour (step S46).
The image processing device 23 determines whether or not the vertex is within the image of the postcard 2 (step S47).
When it is determined that the vertex is not in the image of the postcard 2 (No in step S47), the image processing device 23 selects the next candidate (step S48).

  If it is determined that the four vertices are in the image of the postcard 2 (Yes in step S47), the image processing apparatus 23 determines whether or not the four vertices of the contour in the image of the postcard 2 have been calculated for all candidates. (Step S49).

  If it is determined that the calculation has failed (No in step S49), the image processing apparatus 23 executes steps S46 to S48 again.

  When it is determined that the calculation is possible (Yes in step S49), the image processing apparatus 23 stores the coordinates of all four vertices of the contour candidate in the work data storage area 21d of the memory 21 (step S50).

  When the image processing device 23 executes the contour acquisition process in this way, the CPU 25 displays a preview image of the contour candidate on the display unit 31 so that the operator can select the correction target (step S16 in FIG. 11).

The CPU 25 controls the display unit 31 to display a contour line (step S17).
When the operator designates the contour, the operation unit 32 supplies this operation information to the CPU 25, and the CPU 25 selects the designated contour (step S18).

  The CPU 25 determines whether or not to correct the contour based on the operation information from the operation unit 32 (step S19).

  When the operation information indicating that the correction button has been pressed is supplied, the CPU 25 determines to correct the contour (Yes in step S19). In this case, the CPU 25 instructs the image processing device 23 to execute the oblique photographing correction process (step S20).

  The image processing device 23 calculates the corrected image size based on the corrected aspect ratio calculated by Equation 14 and the distance between the vertices. The image processing device 23 executes the oblique photographing correction process by calculating the coordinates of the original image based on the corrected coordinates according to Equation 12.

  When the image processing device 23 executes the oblique photographing correction process, the CPU 25 instructs the image processing device 23 to execute the front / back and top / bottom detection processing (step S21).

  The image processing device 23 executes front / back and top / bottom detection processing according to the flowchart shown in FIG.

  The image processing device 23 performs character recognition on the corrected image that has been subjected to the oblique photographing correction processing with the upper, lower, left, and right sides of the upper side of the image, and sets the respective recognition rates to r_U, r_D, r_L, and r_R (step S61).

  The image processing device 23 acquires the highest recognition rate r_U, r_D, r_L, r_R as a result of character recognition from the top, bottom, left, and right (step S62).

The image processing device 23 sets the orientation of the image of the postcard 2 to the direction of the highest recognition rate (step S63).
The image processing device 23 determines whether or not the postcard 2 is used in the portrait orientation based on the aspect ratio and orientation of the image of the postcard 2 (step S64).
When it is determined that the postcard 2 is used in the vertically long direction (Yes in step S64), the image processing apparatus 23 determines that a rectangle or a circle exists in the upper left range of the image of the postcard 2. In this case, the image processing device 23 detects a rectangle or a circle existing within this range using the Hough transform (step S65).

  On the other hand, when it is determined that the postcard 2 is used in the landscape direction (No in step S64), the image processing apparatus 23 determines that a rectangle or a circle exists in the upper right range of the image of the postcard 2. To do. In this case, the image processing device 23 detects a rectangle or a circle existing within this range by using the Hough transform (step S66).

  The image processing device 23 determines whether a rectangle or a circle is detected within this range (step S67).

  When a rectangle or a circle is detected within this range (Yes in step S67), the image processing apparatus 23 determines that the face of the postcard 2 is a table (step S68).

  On the other hand, if no rectangle or circle is detected within this range (No in step S67), the image processing apparatus 23 determines that the face of the postcard 2 is the reverse (step S69).

  When the image processing device 23 performs such top / bottom and front / back detection processing, the CPU 25 controls the display unit 31 so that the image of the postcard 2 is displayed with the orientation set in the direction determined to be the highest recognition rate. (Step S22 in FIG. 11).

  The CPU 25 determines whether or not the operator gives an instruction to correct the orientation of the image of the postcard 2 based on the operation information (step S23).

  If it is determined that there is no instruction to correct the direction (No in step S23), the CPU 25 proceeds to step S25.

  When it is determined that there is an instruction to correct the orientation (Yes in step S23), the CPU 25 instructs the image processing device 23 to rotate the orientation of the image of the postcard 2 to the designated orientation. The image processing device 23 rotates the image of the postcard 2 in the designated direction. Then, the CPU 25 controls the display unit 31 so as to display the image rotated by the image processing device 23 (step S24).

The CPU 25 determines whether or not the number of shots is two (step S25).
If it is determined that the number of shots is not two (No in step S25), the CPU 25 executes steps S14 to S24 again.

  If it is determined that the number of shots is two (Yes in step S25), the CPU 25 instructs the image processing device 23 to execute an image composition process (step S26).

  The CPU 25 controls the display unit 31 to display the image synthesized by the image processing device 23 (step S27).

  Based on the operation information supplied from the operation unit 32, the CPU 25 determines whether or not there has been an instruction to correct the arrangement (step S28).

If it is determined that there has been no instruction to correct the layout (No in step S28), the CPU 25 proceeds to step S30.
If it is determined that there has been an instruction to correct the layout (Yes in step S28), the CPU 25 corrects the layout of the image (step S29).

  The CPU 25 controls the display unit 31 so as to display the image to be stored (step S30).

  The CPU 25 stores this image in the external storage unit 34 (step S31), returns to step S14, and waits until operation information indicating that the shutter has been pressed is supplied to the operator.

  As described above, according to the present embodiment, the digital camera 1 captures the postcard 2, acquires the outline of the image, performs oblique photographing correction, and combines the front and back images.

  Therefore, the contents of the postcard 2 can be converted into data so as to be easily managed without using a flatbed scanner, a computer, or the like.

  In addition, by discriminating the front and back and top and bottom of the postcard 2 and combining the front and back surfaces in a predetermined format, it is possible to save the operator from setting the shooting order and shooting direction.

In carrying out the present invention, various forms are conceivable and the present invention is not limited to the above embodiment.
For example, in the above embodiment, the document is described as the postcard 2. However, the document is not limited to the postcard 2 and may be, for example, a sealed letter, a business card, or the like.

  In the above embodiment, the programs are described as being stored in advance in a memory or the like. However, a program for operating the photographing apparatus as all or a part of the apparatus or executing the above-described processing is a flexible disk, a CD-ROM (Compact Disk Read-Only Memory), a DVD (Digital Versatile Disk). It may be stored in a computer-readable recording medium such as MO (Magneto Optical disk) and distributed, installed in another computer, operated as the above-mentioned means, or the above-mentioned steps may be executed.

  Furthermore, the program may be stored in a disk device or the like included in a server device on the Internet, and may be downloaded onto a computer by being superimposed on a carrier wave, for example.

It is a figure which shows the external appearance of the digital camera which concerns on embodiment of this invention. FIG. 1 is a block diagram showing the configuration of the digital camera. FIG. 3 is a block diagram showing each storage area of the memory shown in FIG. 2. It is a figure which shows the image of a postcard, (a) is a preview image of a postcard, (b) is a binary edge image, (c) shows the image which displayed the outline candidate. It is a figure for demonstrating Hough conversion. It is a figure for demonstrating the basic concept of extraction of a projection parameter, and an affine transformation. It is a figure which shows the area | region where the description regarding the postage stamp and charge is performed, (a) shows the area | region when a short side is up, (b) shows the area | region when a long side is up. It is a figure for demonstrating the Hough transformation in the case of detecting a circle. It is a figure which shows the image composition process of the postcard used vertically. It is a figure which shows the image synthesis process of the postcard currently used horizontally. 3 is a flowchart showing camera basic processing (part 1) executed by the digital camera of FIG. 1; 6 is a flowchart showing camera basic processing (part 2) executed by the digital camera of FIG. 1; It is a flowchart which shows the outline acquisition process which the digital camera of FIG. 1 performs. It is a flowchart which shows the front and back and the up-and-down detection process which the digital camera of FIG. 1 performs.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Digital camera, 2 ... Postcard, 12 ... Image sensor, 21 ... Memory, 23 ... Image processing apparatus, 25 ... CPU, 31 ... Display part, 32 ...・ Operation part

Claims (5)

An image processing apparatus for processing a standard document image in which characters are described on the front surface or the back surface, and a preset identification mark is described in a preset area on the front surface,
The image of the standard document obtained by shooting is corrected to a shape corresponding to the standard document, and the characters described in the standard document are recognized based on the corrected image. The orientation of the standard document is determined based on the recognition rate, the orientation of the image of the standard document is adjusted based on the determination result, the region set in advance on the surface is determined, and the identification mark is detected in the determined region The photographed image obtained by the photographing is determined to be the front image of the standard document, and when the identification mark is not detected in the determined area, the photographed image is the back image of the standard document. a captured image correcting unit for judging,
An image synthesizing unit for synthesizing said surface image of the standard document in which the captured image correcting section determines said rear surface image,
An image processing apparatus comprising: The captured image correction unit acquires a binary edge image from the front image and the back image of the document, acquires the contour of the acquired binary edge image, and calculates the vertex coordinates of the image from the acquired contour. Using the acquired vertex coordinates to projectively transform the shape of the image into a shape corresponding to the document, thereby correcting the shape of each image to a shape corresponding to the document,
The image processing apparatus according to claim 1 . An imaging device for photographing a standard document image in which characters are described on the front surface or the back surface, and a preset identification mark is described in a preset area on the front surface,
A photographing unit for photographing the standard document ;
An image processing unit that performs image processing on an image of a standard document obtained by photographing of the photographing unit,
The image processing unit
The image of the standard document obtained by the photographing, corrected in a shape corresponding to the shape on the fixed document, the shape on the basis of the corrected image, recognizes characters written on the standard document, The orientation of the standard document is determined based on the character recognition rate, the orientation of the image of the standard document is adjusted based on the determination result, the region set in advance on the surface is determined, and the identification mark is added to the determined region. When detected, the photographed image obtained by the photographing is determined as the front image of the standard document, and when the identification mark is not detected in the determined area, the photographed image is the back image of the standard document. a captured image correcting section that determines that,
The image synthesizing unit includes an image synthesizing unit for synthesizing said surface image of the standard document in which the captured image correcting section determines said rear surface image,
Imaging apparatus characterized by comprising a. An image processing method for processing a standard document image in which characters are described on the front surface or the back surface, and a preset identification mark is described in a preset area on the front surface,
The image of the standard document obtained by shooting is corrected to a shape corresponding to the standard document, and the characters described in the standard document are recognized based on the corrected image. The orientation of the standard document is determined based on the recognition rate, the orientation of the image of the standard document is adjusted based on the determination result, the region set in advance on the surface is determined, and the identification mark is detected in the determined region The photographed image obtained by the photographing is determined to be the front image of the standard document, and when the identification mark is not detected in the determined area, the photographed image is the back image of the standard document. A photographic image correction step for determining ;
An image synthesis step of synthesizing the corrected front image and back image;
An image processing method comprising: In the computer of the image processing apparatus that processes the standard document image in which characters are described on the front surface or the back surface, and a preset identification mark is described in a predetermined area on the front surface ,
The image obtained form letters by the photographing, corrected in a shape corresponding to the shape on the fixed document, the shape on the basis of the corrected image, recognizes characters written on the fixed document, character The orientation of the standard document is determined based on the recognition rate, the orientation of the image of the standard document is adjusted based on the determination result, the region set in advance on the surface is determined, and the identification mark is detected in the determined region The photographed image obtained by the photographing is determined to be the front image of the standard document, and when the identification mark is not detected in the determined area, the photographed image is the back image of the standard document. and procedures to determine,
A step of synthesizing said surface image of the standard document in which the captured image correcting section determines said rear surface image,
A program for running

Images