JP4225224B2 - Image processing apparatus, captured image projection apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing device, a captured image projection device, an image processing method, and a program, and more particularly to an image processing device, a captured image projection device, an image processing method, and a program that enable acquisition of an image with excellent visibility.

The original placed on the pedestal is photographed with a digital camera, the digital image data obtained by photographing is subjected to predetermined image processing, etc., and then converted to analog and output, whereby an enlarged image of the original on the screen. A photographed image projection apparatus that projects the image is known (see, for example, Patent Document 1). With such a photographed image projection device, it has become possible to photograph a three-dimensional object and project the image onto a screen, which was not possible with conventional OHP.
JP 2002-354331 A (page 2-3, FIG. 1).

  However, in the conventional photographed image projection apparatus, when a document placed on the pedestal is photographed from an oblique direction, a part of the document sometimes protrudes from the angle of view (shooting range) of the camera. In this case, the zoom magnification of the camera must be reduced in order to keep the entire document within the angle of view of the camera.

  As a result, the ratio of the original image to the entire photographed image becomes small, and the original image projected on the screen also becomes small, making it very difficult to see. Such a problem becomes particularly noticeable when the tilt angle increases. In addition, when the captured image is distorted, the ratio of images other than the document is further increased due to the distortion. As described above, it is difficult to appropriately correct the luminance and color difference of the captured image in which the ratio of the image other than the document is large. As a result, the image of the document projected on the screen becomes more difficult to see.

  The present invention has been made to solve the above-described problems, and an object thereof is to provide an image processing device, a captured image projection device, an image processing method, and a program that enable acquisition of an image with excellent visibility. And

In order to achieve the above object, an image processing apparatus according to the first aspect of the present invention provides:
Data storage means for storing the vertex coordinates of the polygon forming the contour of the pedestal;
A document polygon detection means for detecting the polygon forming the outline of the document from the obtained image by taking the original on the pedestal,
An original polygon detection determining means for determining whether or not a polygon forming the outline of the original is detected by the original polygon detecting means;
When it is determined that the polygon forming the document outline is detected by the document polygon detection determining means , image processing is performed within a range specified by the vertex coordinates of the detected polygon, and the document outline is formed. Image processing means for performing image processing within a range specified by the vertex coordinates stored by the data storage means when it is determined that the polygon to be detected has not been detected.

  Further, in the image processing apparatus, the document polygon detection unit includes a candidate detection unit that detects a straight line parameter candidate for forming a contour of the document from the image, and a candidate detected by the candidate detection unit. Document outline acquisition means for detecting a set of straight line parameters that satisfy a predetermined condition and acquiring a polygon formed by the detected set of straight line parameters as the outline of the original document may be included.

  Furthermore, in the image processing apparatus, the document outline obtaining unit detects a set of two substantially parallel linear parameters from the candidates detected by the candidate detecting unit, and detects the detected two linear parameters. A set of two straight line parameters that are substantially perpendicular to each other may be detected, and a polygon formed by the detected set of straight line parameters may be acquired as the outline of the document.

Further, in the image processing apparatus, the affine parameter acquisition unit that acquires the affine parameter from the vertex coordinates in the range specified by the image processing unit, and the affine parameter acquired by the affine parameter acquisition unit are used for the image. Image correction means for correcting and outputting the inclination of the image by performing affine transformation on the coordinates of each pixel included may be further provided.

Further, in the image processing apparatus, a pedestal polygon detecting means for detecting a polygon forming the outline of the pedestal from an image obtained by photographing the pedestal, and an outline of the pedestal by the pedestal polygon detecting means. A pedestal polygon detection discriminating means for discriminating whether or not a polygon forming the pedestal is detected, and the pedestal polygon detection discriminating means detects that the polygon forming the contour of the pedestal has been detected. Data update means for updating polygon vertex coordinate data forming the contour of the pedestal by writing polygon vertex coordinate data into the data storage means may be further provided.

Further, in the image processing apparatus, the pedestal polygon detecting means detects a set of two substantially parallel linear parameters from an image obtained by photographing the pedestal, and detects the detected 2 Alternatively, a set of two straight line parameters that are substantially perpendicular to the straight line parameters may be detected, and a polygon formed by the detected straight line parameter set may be acquired as the contour of the pedestal.

Furthermore, in order to achieve the above object, a captured image projection apparatus according to a second aspect of the present invention is placed on the pedestal and data storage means for storing polygonal vertex coordinates forming the outline of the pedestal. A document photographing means for photographing the original, a document polygon detecting means for detecting a polygon that forms an outline of the document from an image obtained by photographing by the document photographing means, and the document by the document polygon detecting means. A document polygon detecting / determining means for determining whether or not a polygon forming the contour of the document is detected, and a polygon that forms the contour of the document is detected by the document polygon detecting / determining means, and image processing range designated by the detected polygon vertex coordinates of, if the polygons that form the outline of the document is determined to have not been detected, the top stored by said data storage means Comprising image processing means for image processing in a range specified by coordinates, the image projection means for projecting an image output from said image processing means on a screen, a.

In order to achieve the above object, an image processing method according to a third aspect of the present invention includes a data storage step of storing polygonal vertex coordinates forming a contour of a pedestal, and photographing a document on the pedestal. A document polygon detecting step for detecting a polygon forming the contour of the document from the obtained image, and determining whether or not the polygon forming the contour of the document is detected by the document polygon detecting step. When it is determined that the polygon forming the outline of the original is detected by the original polygon detection determining step and the original polygon detection determining step , the image is within the range specified by the vertex coordinates of the detected polygon. processing, if the polygons that form the contour of the document is determined to have not been detected, the image processing step of image processing in the range specified by the vertex coordinates stored by said data storage means , Comprising a.

In order to achieve the above object, a program according to the fourth aspect of the present invention captures a data storage procedure for storing the vertex coordinates of a polygon forming the outline of a pedestal in a computer and a document on the pedestal. A document polygon detection procedure for detecting a polygon that forms the contour of the document from the image obtained by the operation, and whether or not a polygon that forms the contour of the document is detected by the document polygon detection procedure. When it is determined that the polygon forming the outline of the document is detected by the document polygon detection determination procedure and the document polygon detection determination procedure, the range is specified by the vertex coordinates of the detected polygon. image processing, if the polygons that form the outline of the document is determined to have not been detected, to the image processing in the range specified by the vertex coordinates stored by said data storage means An image processing procedure, thereby running.

  According to the present invention, it is possible to provide an image processing device, a captured image projection device, an image processing method, and a program that enable acquisition of an image with excellent visibility.

  A captured image projection apparatus according to an embodiment of the present invention will be described below with reference to the drawings.

  FIG. 1 is a diagram illustrating a configuration of a captured image projection apparatus according to an embodiment of the present invention. As shown in FIG. 1, the captured image projection apparatus includes a document camera 1, a projector 2, and a computer 5.

  The document camera 1 has a function of photographing a document 4 to be projected as shown in FIG. 1A and a character, a figure, a photograph, etc., such as a screen 3 as an object to be photographed as shown in FIG. 1B. And includes a camera unit 11, a support column 12, a pedestal 13, and an operation table 14.

  The camera unit 11 is for photographing the document 4 and is constituted by a digital camera. The support column 12 is for attaching the camera unit 11, and is rotatably connected to the camera unit 11 and the pedestal 13. The pedestal 13 is for supporting the support column 12.

  FIG. 2 is a block diagram showing the configuration of the document camera 1 shown in FIG. As shown in FIG. 2, the document camera 1 includes an image data generation unit 21 and a data processing unit 22. The image data generation unit 21 and the data processing unit 22 may be provided in the camera unit 11 shown in FIG. 1, and the image data generation unit 21 and the data processing unit 22 include the camera unit 11 and the operation console 14, respectively. And may be provided.

  The image data generation unit 21 captures image data of a shooting target by shooting a shooting target (original 4 or screen 3), and includes an optical lens device 101 and an image sensor 102. Yes.

  The optical lens device 101 includes a lens that collects light and a driving device that drives the lens. The optical lens device 101 collects light from characters, drawings, photographs, and the like of an object to be photographed on the image sensor 102. Form an image.

  The image sensor 102 is for capturing an image formed by condensing light by the optical lens device 101 as digitized image data, and is configured by a CCD (Charge Coupled Device) or the like.

  The data processing unit 22 acquires the image data of the document 4 and the screen 3 captured from the image data generation unit 21 and performs image processing, and outputs the image data subjected to the image processing to the projector 2. A video output device 202, an image processing device 203, an operation unit 204, a program code storage device 205, a CPU (Central Processing Unit) 206, and an interface device 208.

  The memory 201 is for temporarily storing an image from the image sensor 102, image data to be sent to the video output device 202, and the like. The memory 201 is also used as a work memory for the CPU 206. The video output device 202 outputs the image data sent to the computer 5 to a display device (not shown), thereby displaying the image of the document 4 or the screen 3 on the display device.

  The image processing apparatus 203 is for performing image processing such as image data compression, image distortion correction, and image effect on the image data temporarily stored in the memory 201.

  For example, when the camera unit 11 captures characters or the like of the document 4 in an oblique state or rotated state with respect to the photographing surface, the image processing apparatus 203 is inclined or rotated as shown in FIG. By performing distortion correction (trapezoid correction) on the image of the document 4 in the state, an image of the document 4 taken from the front as shown in FIG. 3B is generated.

  In addition, when the camera unit 11 captures the characters on the screen 3 from the left direction, the image processing apparatus 203 distorts the image of the screen 3 captured from the left direction as illustrated in FIG. By performing the correction, an image of the screen 3 taken from the front as shown in FIG. 4B is generated.

  In order to correct distortion, the image processing apparatus 203 cuts a rectangle from the distorted image, and performs projective conversion of the captured image into the cut rectangle. More specifically, the image processing apparatus 203 is controlled by the CPU 206 to mainly perform (1) affine parameter extraction processing for extracting affine parameters from the captured image, and (2) projective transformation of the image using the extracted affine parameters. Then, an image conversion process for correcting distortion is performed, and (3) an image sharpening process for correcting the luminance or color difference to sharpen the image. The details of these processes will be described later.

  The operation unit 204 includes keys and buttons for controlling the function of document projection, and outputs operation information to the CPU 206 when the user operates these switches and buttons.

  In the present embodiment, the operation unit 204 includes a photographing key, a calibration request key, a control key, and a shutter button. The shooting key is a key for selecting a desired shooting mode from a document shooting mode for shooting an image of the document 4 and a board shooting mode for shooting an image of the screen 3.

  The calibration request key is a key for instructing registration of the contour shape of the base 13. The control key includes a YES key for confirming the operation and a NO key for canceling the operation. The shutter button is pressed when shooting an object to be shot. The zoom button is pressed in order to perform zoom control of the camera unit 11 when shooting a small document 4 or enlarging a part of the document 4.

  The program code storage device 205 is for storing a program executed by the CPU 206, and is configured by a ROM (Read Only Memory) or the like.

  The CPU 206 controls each unit in accordance with a program stored in the program code storage device 205. Specifically, the image sensor 102, the memory 201, the video output device 202, the image processing device 203, and the like are controlled based on operation information transmitted from the operation unit 204.

  For example, when the shooting key is pressed by the user, the CPU 206 sets each unit in the document shooting mode or the board shooting mode in response to the operation information transmitted from the operation unit 204 and until the shutter button is pressed. The image sensor 102 is set to the moving image mode. When the shutter button is pressed, the CPU 206 sets a still mode in which an image is read at a high resolution in response to operation information from the operation unit 204.

  The CPU 206 has a function of initializing camera setting parameters such as the focus, exposure, and white balance of the optical lens device 101 to the operating mode. As a result, the scene captured by the camera unit 11 is condensed on the image sensor 102 via the optical lens device 101, and the image sensor 102 is a digital image with a low resolution for moving image shooting from the collected image. Data is created and sent to the memory 201 at, for example, about 30 sheets per second.

  Further, the CPU 206 determines whether or not there is a motion in the image of the document 4 that is the photographing target according to a flowchart described later, and switches between the operating mode and the stationary mode according to the determination result.

  The in-operation mode is a mode that is set when the user intends to place the document 4 or the like that the user wants to project on the pedestal 13 and is a mode that projects the image projected by the camera unit 11 onto the projector 2 as it is.

  In the operating mode, the CPU 206 controls each unit so that, for example, an image having an image resolution of about VGA (Video Graphics Array: 640 × 480 dots) is projected at a speed of 30 fps (frame / second). And As described above, the operating mode is a mode in which real-time characteristics are emphasized although the resolution is low.

  The still mode is a mode that is set after the user places the document 4 and is a mode in which the camera unit 11 takes a high-resolution image and projects a high-resolution still image. When the camera unit 11 is a camera with an imaging resolution of 3 million pixels, the cut-out projected image is an XGA (Extended Graphics Array: 1024 × 768 dots) still image.

  The interface device 208 is for transmitting the compressed image data to the computer 5.

  A projector 2 shown in FIG. 1 forms an image of an original 4 by irradiating a screen 3 with projection light. The computer 5 is connected to the document camera 1 via a cable 15 such as a USB, and is also connected to the projector 2 via a cable 16 such as RGB. The computer 5 receives the image data of the document 4 sent from the document camera 1 via the cable 15 and outputs the received image data to the projector 2 via the cable 16.

  Next, the operation of the captured image projection apparatus according to this embodiment will be described. When the user turns on the power of the captured image projection apparatus, the CPU 206 of the document camera 1 reads the program code from the program code storage device 205 and executes the basic camera processing.

  FIG. 5 is a flowchart of the document camera basic process. In this camera basic process, the CPU 206 first initializes camera setting parameters such as the focus, exposure, and white balance of the camera unit 11 of the document camera 1 as shown in FIG. 5 (step S1). As a result, the scene captured by the camera is condensed on the image sensor 102 via the optical lens device 101, and the image sensor can create a digital image from the collected image.

  Next, the CPU 206 checks the mode switch or the like to determine whether the shooting mode is the document shooting mode or the board shooting mode (step S2). When it is determined that the shooting mode is the document shooting mode (step S2; Yes), the document shooting process is executed (step S3). On the other hand, if it is determined that the shooting mode is the board shooting mode (step S2; No), board shooting processing is executed (step S4).

  The image data obtained by the photographing processes in steps S3 and S4 is stored in a predetermined area of the memory 201. When the photographing process in step S3 or S4 is completed, the CPU 206 returns to step S2, detects the mode switch again, and executes the photographing process in step S3 or S4.

  The CPU 206 reads out the image data obtained by the photographing process in step S3 or S4 from the memory 201, outputs the image data to the outside as an RGB output via the video output device 105, and compresses the image data by the image processing device 203. Thereafter, the data is transferred to the computer 5 via the interface device 108.

  Subsequently, the computer 5 starts up the document processing software installed in the HDD or the like and executes the PC document basic processing. FIG. 6 is a flowchart showing the PC document basic process. In this PC document basic process, as shown in FIG. 6, the computer 5 first determines whether or not the image data transferred from the document camera has been received (step S11), and determines that the image data has been received. In such a case (step S11; Yes), the received image data is decoded (step S12). On the other hand, when it is determined that the image data has not been received (step S11; No), the process waits in a loop until the image data is received.

  Then, the computer 5 determines whether the shooting mode is the document shooting mode or the board shooting mode (step S13). If the computer 5 determines that the shooting mode is the document shooting mode (step S13; Yes), the received image is displayed. Drawing is performed (step S14), and the data of the drawn image is output to the projector 2 to project this image on the screen 3. On the other hand, when it is determined that the shooting mode is the board shooting mode (step S13; No), the received image data is stored in a file or the like (step S15).

  Next, the document photographing process in step S3 will be described in detail. 7 and 8 are flowcharts showing details of the document photographing process. In this document photographing process, the CPU 206 first initializes the work memory in the interface device 208, the image processing device 203, and the memory 201 as shown in FIG. 7 (step S101). Next, the CPU 206 sets a calibration request flag in order to register the contour shape of the pedestal 13 in the initial state (step S102).

  Subsequently, the CPU 206 determines whether or not the calibration request key is pressed by the user (step S103). When it is determined that the calibration request key has not been pressed (step S103; No), zoom processing is executed (step S104). In this zoom process, the CPU 206 determines whether or not the zoom button or the like has been pressed by the user, and performs zoom control of the camera unit 11 if determined to be pressed. Further, when the zoom of the camera unit 11 is changed in the zoom process, the CPU 206 executes a pedestal registration change process. In this pedestal registration change process, the CPU 206 adjusts a linear parameter indicating the contour shape of the pedestal 13 stored in the memory 201 in accordance with the zoom ratio before and after the zoom process.

  Then, the CPU 206 performs shooting by the camera unit 11 and acquires image data (step S105). In the initial state, the document 4 is not placed on the pedestal 13, and the camera unit 11 captures only the pedestal 13 and acquires image data of the pedestal 13. Thereafter, the CPU 206 transfers image data obtained by photographing to the memory 201 from the image sensor 102.

  Subsequently, the CPU 206 reads the image data from the memory 201, compares the image data with the image data obtained by the previous shooting, and determines whether or not there is a motion in the image (step S106). Specifically, the movement of the image is detected by determining whether or not the difference between the image data is equal to or greater than a predetermined threshold.

  If it is determined that there is a motion in the image (step S106; Yes), the operation mode is set (step S107), and the captured image is displayed on the display device as it is. On the other hand, when it is determined that there is no motion in the image (step S106; No), it is checked whether the in-operation mode is set or the still image mode is set, and whether the image has moved until the previous time. Is determined (step S108). If the CPU 206 determines that the image has moved until the previous time (step S108; Yes), the CPU 206 sets the still mode (step S109).

  Subsequently, the CPU 206 checks the calibration request flag to determine whether or not a calibration request has been made (step S110). If it is determined that a calibration request has been made (step S110; Yes) ), A pedestal registration process is executed (step S111). In this pedestal registration process, the CPU 206 acquires and registers the contour shape of the pedestal 13 from the captured image. Once the contour shape of the pedestal 13 is registered as described above, the positional relationship between the camera unit 11 and the pedestal 13 varies, for example, the tilt angle of the camera unit 11 with respect to the pedestal 13 varies. The contour shape of the base 13 is not registered again until the key is pressed.

  Thereafter, the CPU 206 clears the calibration request flag, returns to step S103, and determines again whether the calibration request key has been pressed. If it is determined that the calibration request key has been pressed by the user (step S103; Yes), it is determined that the positional relationship between the camera unit 11 and the pedestal 13 has changed, and the pedestal 13 after the change is determined. In order to register the contour shape, the process returns to step S102, and a calibration request flag is set. Then, the CPU 206 executes the pedestal registration process in step S111 again to register the shape of the pedestal after the change.

  After the registration of the contour shape of the pedestal 13 is made, the user places the document 4 on the pedestal 13 and tries to project the document on the document 4. Until the user sets the document 4, the CPU 206 copies the image data obtained by photographing to a predetermined area of the memory 201, compresses the image data with the image processing device 203, and the interface. The data is transmitted to the computer 5 via the device 208 at a speed of about 30 frames, for example. Then, the computer 5 performs a decoding process or the like on the image data transferred from the document camera 1 to acquire uncompressed data, and displays the captured image on a display or the like.

  Thereafter, when the user finishes setting the document 4, the CPU 206 determines that there is no movement in step S <b> 106 (step S <b> 106; No), reads an image taken from the memory 201, and the image processing apparatus 203. A quadrangular contour extraction process is executed (step S112). In this quadrangle contour extraction process, the CPU 206 extracts coordinate points (rectangle contours) at the four corners of the image of the photographing target from the photographed image of the photographing target.

  Subsequently, the CPU 206 executes image conversion processing (step S113). In this image conversion process, the CPU 206 performs clipping of a subject to be imaged and projective conversion to a front image. Then, the CPU 206 executes an image sharpening process (step S114). Through the processing of these steps S101 to S114, the document camera 1 can detect the front surface as shown in FIG. 3B from the image of the document 4 taken in the oblique state or rotated state as shown in FIG. The image of the document 4 taken from is acquired.

  Thereafter, the CPU 206 compresses the image data by the image processing device 203 and transmits it to the computer 5 via the interface device 208 (step S115). When the user switches the mode switch from the document shooting mode to the board shooting mode (step S116; No), the CPU 206 ends the document shooting process.

  Next, the board photographing process in step S4 will be described in detail. FIG. 9 is a flowchart showing details of the board photographing process. In this board photographing process, the CPU 206 first initializes the working memory in the interface device 208, the image processing device 203, and the memory 201 as shown in FIG. 9 (step S121). The CPU 206 waits for the user to press the shutter (step S122). When the shutter is pressed (step S122; Yes), the camera unit 11 captures the screen 3 and acquires image data (step S123). Thereafter, the CPU 206 transfers image data obtained by photographing to the memory 201 from the image sensor 102.

  Subsequently, the CPU 206 reads out the photographed image from the memory 201, and in the same manner as when performing the document photographing process, the image processing device 203 performs the rectangular outline extraction process (step S124), the image change process (step S125), and the image clearness. Processing (step S126) is sequentially executed. Thereafter, the CPU 206 compresses the image data by the image processing device 203 and transmits it to the computer 5 via the interface device 208 (step S127). When the user switches the mode switch from the board shooting mode to the document shooting mode (step S128; No), the CPU 206 ends the board shooting process.

  Next, image processing executed by the image processing apparatus 203 will be described. First, a basic concept (realization method) for affine transformation used for image processing in the image processing apparatus 203 will be described.

  Affine transformation is widely applied to image spatial transformation. In this embodiment, projective transformation is performed using two-dimensional affine transformation without using three-dimensional camera parameters. This is because the coordinates (u, v) before conversion are related to each other by the following equation (1) by conversion such as movement, enlargement / reduction, and rotation. Projective transformation can also be performed by this affine transformation.

  The final coordinates (x, y) are calculated by the following formula 2.

  Expression 2 is an expression for projective transformation, and the coordinates (x, y) are degenerated toward 0 according to the value of z ′. That is, the parameter included in z ′ affects the projection. This parameter is a13, a23, a33. Since other parameters can be normalized by the parameter a33, a33 may be 1.

  FIG. 10A is a diagram showing the coordinates of the vertices of the quadrangular shooting target (original 4 or screen 3) shown in FIG. FIG. 11 is a diagram for explaining a relationship between a quadrangle photographed by the camera unit 11 and an image of an actual photographing object.

  In FIG. 11, a UVW coordinate system is a three-dimensional coordinate system of an image obtained by photographing with the camera unit 11. The A (Au, Av, Aw) vector and the B (Bu, Bv, Bw) vector represent the object to be photographed as vectors in the three-dimensional coordinate system UV-W. The S (Su, Sv, Sw) vector indicates the distance between the origin of the three-dimensional coordinate system UV-W and the object to be imaged.

  The projection screen shown in FIG. 11 is used for projecting an image of an object to be photographed. If the coordinate system on the projection screen is (x, y), the image projected on the projection screen may be considered as the image photographed by the camera unit 11. It is assumed that the projection screen is positioned vertically at a distance f from the W axis. An arbitrary point P (u, v, w) on the object to be photographed is connected to the origin by a straight line, and there is a point that intersects the straight line and the projection screen, and the XY coordinate of the intersection is represented by p (x, y). At this time, the coordinate p is expressed by the following equation 3 by projective transformation.

  From Equation 3, the relationship shown in the following Equation 4 is obtained from the relationship between the points P0, P1, P2, P3 and the projection points p0, p1, p2, p3 on the projection screen as shown in FIG.

  At this time, the projection coefficients α and β are expressed by the following equation (5).

  Next, projective transformation will be described. An arbitrary point P on the object to be imaged is expressed by the following equation 6 using S, A, and B vectors.

  When the relational expression of Expression 4 is substituted into Expression 6, the coordinates x and y are expressed by the following Expression 7.

  When this relationship is applied to the affine transformation formula, the coordinates (x ′, y ′, z ′) are expressed by the following equation (8).

  By substituting m and n into Equation 8, the corresponding point (x, y) of the captured image is obtained. Since the corresponding point (x, y) is not necessarily an integer value, the pixel value may be obtained using an image interpolation method or the like.

  For m and n, an image size (0 ≦ u <umax, 0 ≦ v <vmax) for outputting the corrected image p (u, v) is given in advance, and a method of adjusting the image according to the image size is considered. It is done. According to this method, m and n are expressed by the following equation (9).

  However, the aspect ratio of the corrected image to be created does not match the aspect ratio of the shooting target. Here, the relationship between the corrected image p (u, v) and the values of m and n is expressed by the following equation 10 from the equation 4.

  If the focal length f of the lens, which is a camera parameter, is known, the aspect ratio k can be obtained according to Equation 10. Therefore, if the image size of the corrected image p (u, v) is (0 ≦ u <umax, 0 ≦ v <vmax), m and n are obtained according to the following equation 11 to obtain the same as the object to be imaged. An aspect ratio k can be obtained.

  When the camera has a fixed focal point, the value of the focal length f of the lens can be obtained in advance. When a zoom lens or the like is present, the value of the focal length f of the lens changes depending on the zoom magnification of the lens. Therefore, a table indicating the relationship between the zoom magnification and the focal length f of the lens is created and stored in advance. Then, the focal length f can be read based on the zoom magnification, and projective transformation can be performed according to Equations 10 and 11.

  In order to perform such affine transformation, the CPU 206 first executes the quadrangular contour extraction processing in steps S112 and S124. FIG. 12 is a flowchart showing details of the quadrangular contour extraction processing.

  In this quadrangular contour extraction process, the CPU 206 first controls the image processing device 203 to generate a reduced luminance image from the input image in order to reduce the number of image processing operations as shown in FIG. 12 (step S201). . Next, the CPU 206 generates an image of only a straight line from the generated reduced luminance image, and sets this as an edge image of the shooting target (step S202). Subsequently, the CPU 206 corrects the distortion aberration of the generated edge image (step S203).

  Subsequently, the CPU 206 detects the straight line parameter included in the edge image from the edge image of the photographing object by extracting the peak value by Hough transforming the edge image corrected in step S203 (step S203). S204). Then, the CPU 206 creates a quadrangle that is a candidate for forming the contour of the object to be photographed from the detected straight line parameters (step S205).

  Thereafter, the CPU 206 generates candidate rectangles in this way, and assigns priorities to the generated rectangles (step S206). Then, the CPU 206 selects a rectangle according to the priority order, determines whether the selected rectangle has been extracted (step S207), and determines that the rectangle has not been extracted (step S207; No), the rectangle outline The extraction process ends.

  On the other hand, if it is determined that the quadrangle has been extracted (step S207; Yes), the quadrangle extracted from the image processing device 203 is acquired and transmitted to the video output device 202, and a quadrangle preview image is displayed on the display device ( Step S208).

  Subsequently, based on the operation information transmitted from the operation unit 204, the CPU 206 determines which switch of the YES key and the NO key has been pressed (step S209). If it is determined in step S209 that the NO key has been pressed (step S209; No), the image processing apparatus 203 designates the next candidate rectangle (step S210).

  On the other hand, if it is determined that the YES key has been pressed (step S209; Yes), an affine parameter is calculated from the extracted quadrangular vertices (step S211).

  Next, the quadrangle contour extraction process will be described more specifically. An example of a reduced luminance image generated by the CPU 206 using the image processing apparatus 203 is shown in FIG. The image processing apparatus 203 generates an edge image as shown in FIG. 13B from such a reduced luminance image by using an edge detection filter called a Roberts filter (step S202). The Roberts filter is a filter that detects an edge of an image by weighting two 4-neighbor pixels, obtaining two filters Δ1 and Δ2, and averaging them.

  14A shows the coefficient of the filter Δ1, and FIG. 14B shows the coefficient of the filter Δ2. When the coefficients of the two filters Δ1, Δ2 are applied to the pixel value f (x, y) of a certain coordinate (x, y), the converted pixel value g (x, y) Represented by

  In the edge image generated in step S202, a straight line may not be extracted due to distortion. Therefore, the CPU 206 performs distortion correction on the edge image (step S203). The reason for performing distortion correction not on the original image but on the edge image is to increase the processing speed. The distortion correction calculation is to calculate for each pixel which pixel in the original image becomes the pixel to be examined after distortion correction. At that time, in the case of an edge image, only the coordinates skipped at a constant interval are investigated, and processing can be performed at a high speed because there are few pixels.

  The edge image shown in FIG. 13B includes straight line parameters. The image processing apparatus 203 performs a Hough transform from the edge image to detect a straight line parameter (step S204).

  The Hough transform is a transform that votes points constituting a straight line on the XY plane to the ρ-θ plane defined by the following equation (13).

  When θ is changed from 0 to 360 ° in the coordinates (x, y) of each point, the same straight line is represented by one point on the ρ-θ plane. For this reason, it is possible to determine a ρ-θ coordinate having a large number of votes as a straight line. At that time, the number of votes is the number of pixels on a straight line, which corresponds to the length of the straight line. Therefore, ρ−θ having an extremely small number of votes, that is, a short straight line can be excluded from the straight line candidates.

  Thus, in the method using the Hough transform, the processing speed decreases as the number of points or angles to be investigated increases. In order to avoid this decrease in processing speed, the edge image is reduced by processing the coordinates of the investigation target at a constant interval in both the X and Y axis directions at the time of edge detection. As a result, the number of survey targets can be reduced.

  Furthermore, the investigation angle can be reduced in another way. That is, when considering the coordinate system with the center of the image as the origin in the edge image to be examined, ρ also takes a negative value, so it is sufficient to measure the angle 0 ≦ θ <180 ° and the remaining 180 °. ≦ θ <360 ° can be expressed by taking a negative value of ρ.

  However, when a square is actually captured, each side exists vertically and horizontally when the square is at the center of the image. Therefore, rather than investigating in the range of 0 ° ≦ θ <180 °, as shown in FIG. 15, it is more effective to measure the upper and lower sides and the left and right sides in the range represented by the following formula 14, respectively. It is.

  As is clear from Equation 14, cos θ of 45 ° ≦ θ <135 °, sin θ of 135 ° ≦ θ <225 °, sin θ of 45 ° ≦ θ <135 °, and cos θ of 135 ° ≦ θ <225 ° Since the minus numbers correspond to each other, the sin table and the cos table can be shared.

  Further, a straight line detected at 45 ° ≦ θ <135 ° in FIG. 16A is indicated by a thick line, and a straight line detected at 135 ° ≦ θ <225 ° is indicated by a thick line in FIG. Depending on whether is a positive value or a negative value, it is possible to specify the upper and lower sides or the left and right sides. Therefore, it is possible to select the sides constituting the central square more efficiently.

  Based on such an idea, the CPU 206 detects a straight line and identifies a quadrangle. Note that this quadrangle detection processing differs depending on whether the object to be photographed is photographed from directly above, such as document photography, or when the document camera 1 is photographed with the document camera 1 tilted with respect to the object to be photographed, such as board photography. It becomes.

  Further, the pedestal registration process in step S111 executed prior to the quadrangular contour extraction process when the calibration key is pressed in the document photographing process is also based on the above-described concept. First, the pedestal registration process will be described with reference to the flowchart shown in FIG.

  In this pedestal registration process, the CPU 206 first generates a reduced luminance image from the input image as shown in FIG. 17, and then generates an edge image of the pedestal 13 from the generated reduced luminance image (step S221). Next, the CPU 206 performs Hough transform on the edge image and performs voting from the XY plane to the ρ-θ plane (step S222). In the document camera 1, four linear parameters indicating the shape of the outline of the pedestal 13 on the ρ-θ plane are registered in advance. The CPU 206 registers the design value of this linear parameter in the memory 201 as indicating the contour shape of the base 13 (step S223).

  Subsequently, the CPU 206 detects a linear parameter included in the edge image of the pedestal 13 actually captured by extracting a peak value from the data obtained by performing the Hough transform on the edge image (step). S224).

  Then, the CPU 206 determines whether or not a straight line parameter is detected in step S224 (step S225). If a straight line parameter is detected (step S225; Yes), the straight line parameter registered in the memory 201 is determined. Is rewritten from the design value to the detected linear parameter (actually measured value) (step S226). Thereby, the measured value of the straight line parameter is registered in the memory 201 as indicating the shape of the outline of the base 13.

  On the other hand, when the straight line parameter is not detected (step S225; No), step S226 is skipped and the pedestal registration process is terminated as it is. For this reason, the design value of the straight line parameter remains registered in the memory 201 as indicating the shape of the outline of the pedestal 13.

  Note that when projecting a small document 4 or enlarging and photographing a part of the document 4, the CPU 206 increases the zoom of the camera unit 11 in the zoom process of step S104. Accordingly, since the image of the pedestal 13 photographed by the camera unit 11 is enlarged, the CPU 206 needs to adjust the linear parameter registered in the memory 201 in the above-described pedestal registration change process. FIG. 18 is a flowchart showing details of the pedestal registration change process.

  In this pedestal registration change process, as shown in FIG. 18, the CPU 206 first determines whether or not the zoom of the camera unit 11 has been changed in the zoom process of step S104 (step S231). If it is determined in step S231 that the zoom of the camera unit 11 has been changed (step S231; Yes), the zoom ratio of the camera unit 11 before and after the zoom process is obtained, and the zoom ratio is determined according to the obtained zoom ratio. Then, the straight line parameters registered in the memory 201 are adjusted (step S232). Specifically, when the zoom ratio of the camera unit 11 before zoom processing and after zoom processing is “2”, the CPU 206 doubles each value of the linear parameter registered in the memory 201.

  On the other hand, if it is determined in step S231 that the zoom of the camera unit 11 has not been changed (step S231; No), the CPU 206 skips step S232 and ends the pedestal registration change process.

  As described above, when the zoom of the camera unit 11 is changed by the zoom process, the CPU 206 adjusts the linear parameter according to the zoom ratio of the camera unit 11 before the zoom process and before the zoom process. The contour shape of the pedestal 13 registered in the pedestal registration process shown in FIG. 17 can be changed. When the adjusted straight line parameter exceeds the angle of view of the image, the straight line parameter exceeding the angle of view may be excluded from the candidates. Further, a straight line parameter exceeding the angle of view may be written in the memory 201 as it is. Even if it does in this way, since CPU206 does not detect the linear parameter exceeding this angle of view in step S224, a problem on a process does not arise.

  Next, a quadrangle detection process (rectangular detection process at the time of document shooting) when shooting a document will be described with reference to a flowchart shown in FIG. In this document photographing square detection process, the CPU 206 first generates a reduced luminance image from the input image as shown in FIG. 19, and then generates an edge image of the document 4 from the generated reduced luminance image (step). S301). Next, the CPU 206 performs Hough transform on the edge image and performs voting from the XY plane to the ρ-θ plane (step S302).

  Then, the CPU 206 acquires a plurality of coordinates having a large number of votes when 45 ° ≦ θ <135 ° as candidates for straight line parameters forming the upper and lower sides (step S303). Similarly, the CPU 206 acquires a plurality of coordinates having a large number of votes when 135 ° ≦ θ <225 ° as candidates for straight line parameters forming the left and right sides (step S304).

  Subsequently, the CPU 206 detects a straight line parameter close to the straight line parameter forming the contour of the pedestal 13 registered in the memory 201 from the candidates acquired in steps S303 and S304, and the detected straight line parameter is determined from the candidates. Remove (step S305). Specifically, a linear parameter whose difference from the linear parameter forming the contour of the base 13 is about 0 to 2 or 3 pixels in ρ and about 0 to 2 ° in θ is excluded from the candidates.

  Then, the CPU 206 selects, from among the candidates in the X-axis direction and the candidates in the Y-axis direction that remain in step S305, those whose values of ρ are different and whose θ is within a predetermined range (step S306). ). Specifically, two linear parameters having different values of ρ and having a difference of θ of about ± 10 ° are detected, and the difference of θ from these detected linear parameters is 90 ° ± 10. Two linear parameters having a difference of + and-in the value of ρ are detected. Then, as shown in FIG. 20A, the CPU 206 acquires a quadrangle formed by these detected four straight line parameters as a quadrangle of the photographed document 4. In this way, the CPU 206 can detect a square from the edge image.

  The CPU 206 determines whether or not a quadrangle is detected in step S306 (step S307). If it is determined that a quadrangle is detected (step S307; Yes), the CPU 206 proceeds to step S310.

  On the other hand, if it is determined in step S307 that a quadrangle has not been detected (step S307; No), the CPU 206 again includes the X-axis direction candidate and the Y-axis direction candidate including the straight line parameter removed in step S305. Are selected from the edge images, and the squares are detected from the edge images (step S308).

  CPU 206 further determines whether or not a quadrangle has been detected in step S308 (step S309). If it is determined in step S307 or S309 that a quadrangle has been detected (step S307; Yes, step S309; Yes), the CPU 206 obtains the coordinates of each vertex of the detected quadrilateral of the document 4 (step S310).

  On the other hand, when it is determined in step S309 that no quadrangle has been detected (step S309; No), the CPU 206 obtains the coordinates of each vertex of the quadrilateral of the base 13 registered in the memory 201 (step S311).

  Next, a quadrangle detection process (board detection quadrangle detection process) in the case of executing board shooting will be described with reference to the flowchart shown in FIG. Note that the processing from steps S311 to S314 in the board photographing quadrangle detection processing shown in FIG. 21 is the same as the processing from steps S301 to S304 in the document photographing quadrilateral detection processing shown in FIG. Is omitted.

  The CPU 206 identifies those having the same absolute value | ρ | of ρ from the candidates for the X-axis direction and the candidate for the Y-axis direction acquired in step S314, and prioritizes them in descending order of the absolute value. Is set (step S315). This is because the opposite sides are not always parallel when the screen 3 is photographed obliquely. By specifying the top, bottom, left, and right sides as those having a large absolute value of | ρ |, a straight line priority order for specifying a region having the maximum area as shown by a bold line in FIG. 20B is obtained. be able to.

  The CPU 206 identifies the quadrangle constituting the above-described maximum area as the quadrangle of the screen 3 that has been shot (step S316).

  Next, the image motion discrimination process executed in step S106 will be described. The CPU 206 determines whether or not there is a motion in the image of the document 4 in order to switch the shooting mode. In order to make such a determination, the CPU 206 obtains an image change amount MD from the previously captured image. The image change amount MD is an amount representing how much the image being taken has changed compared to the previous image. There are several methods for calculating this amount. In this captured image projection apparatus, the sum of absolute values of differences between pixels is obtained as the image change amount MD from the data of the previously captured image.

  That is, if the previous pixel data is Pn-1 (x, y) and the current pixel data is Pn (x, y) (1 ≦ x ≦ 640, 1 ≦ y ≦ 480), the image change amount MD is This is expressed by the following equation (15).

  Further, a threshold value MDTH shown in Expression 16 is set in advance as a threshold value for determining the presence / absence of motion in comparison with the image change amount MD. The captured image projection device determines that there is no motion if the image change amount MD is less than the threshold value MDTH, and determines that there is motion if the image change amount MD is greater than or equal to the threshold value MDTH.

  Next, a method for obtaining the affine parameters from the detected original 4 or the square of the base 13 registered in the memory 201 will be described. Using the coordinates (x0, y0), (x1, y1), (x2, y2), (x3, y3) of the four points of the detected original 4 or the registered quadrilateral of the pedestal 13, Equation 5 , Eq. 8, affine parameters that are each element in the matrix shown in Eq. 8 can be obtained.

  At this time, since the scale of m and n is in the range of 0 ≦ m ≦ 1, 0 ≦ n ≦ 1, the coordinate unit is corrected. Assume that the U-axis side and V-axis side image sizes are scaled by Usc and Vsc, respectively, without changing the center of the image. At this time, assuming that the center of the image is uc and vc, the scale conversion of the image is expressed by the following Expression 17.

  When the equation 17 is rewritten on the u and v scales, the coordinates (x ′, y ′, z ′) of the converted rectangle are expressed by the following equation 18.

  Here, Af is an affine transformation matrix, and the affine transformation matrix Af is expressed by the following equation (19).

  When the maximum image (screen) size that can be output is given by (umax, vmax), uc and vc are expressed by the following equation (20).

  Further, when vmax / umax> k and vmax / umax ≦ k, an image having a desired aspect ratio k is obtained by performing image scaling according to the following equations 21 and 22.

  Based on such an idea, the CPU 206 calculates an affine parameter from the vertices of the rectangle. FIG. 22 is a flowchart showing details of the affine parameter extraction processing.

  The CPU 206 calculates projection coefficients α and β according to Equation 5 from the four vertex coordinates (x0, y0), (x1, y1), (x2, y2), and (x3, y3) of the quadrangle (step S321). ) And the aspect ratio k of the document 4 is calculated (step S322).

  Next, the CPU 206 designates the center point (uc, vc) of the image according to Equation 20 (Step S323), and compares the maximum image size vmax / umax with the aspect ratio k expressed by Equation 12 (Step S324). ).

  When vmax / umax> k (step S324; Yes), assuming that the aspect ratio k is not changed, the CPU 206 determines that the maximum image size vmax on the V-axis side (vertical) is larger than the image size of the object to be photographed. judge. Then, the CPU 206 calculates usc and vsc according to Equation 21 so that the maximum image size on the U-axis side matches the image size of the object to be imaged, and determines the scale of the image of the object on the V-axis side ( Step S325).

  When vmax / umax ≦ k (step S324; No), assuming that the aspect ratio k is not changed, the CPU 206 determines that the maximum image size umax on the U-axis side (horizontal) is larger than the image size of the subject to be photographed. judge. Then, the CPU 206 obtains usc and vsc according to Equation 22 so that the maximum image size on the V-axis side matches the image size of the object to be photographed, and determines the scale of the image of the object to be photographed on the U-axis side. (Step S326).

  Subsequently, the CPU 206 calculates Formula 19 from the calculated usc, vsc, uc, vc and the vertex coordinates (x0, y0), (x1, y1), (x2, y2), (x3, y3) of the four points of the rectangle. Accordingly, the affine transformation matrix Af is obtained (step S327), and each element of the affine transformation matrix Af is used as the affine parameter A to obtain the affine parameter A (step S328).

  Next, an image processing method for creating a corrected image using the obtained affine parameters will be described. First, when projective transformation or other affine transformation is performed using the affine parameters, as shown in FIG. 23, the point p (x, y) of the original image is transformed by projective transformation using the transformation matrix Ap (after). Assume that it corresponds to the point P (u, v) of the image. In this case, it is better to obtain the point p (x, y) of the original image corresponding to the point P (u, v) of the converted image than to obtain the point P of the converted image corresponding to the point p of the original image. preferable.

  Note that when the coordinates of the point P of the converted image are obtained, an interpolation method based on the bilinear method is used. In the bilinear interpolation method, the coordinate point of the other image (transformed image) corresponding to the coordinate point of one image (original image) is searched, and the (pixel) values of four points around the coordinate point of the one image are found. Is a method for obtaining a (pixel) value of a point P (u, v) of the converted image. According to this method, the pixel value P of the point P of the converted image is calculated according to the following Expression 23.

  In order to obtain the point p (x, y) of the original image corresponding to the point P (u, v) of the converted image, the CPU 206 executes an image conversion process. FIG. 24 is a flowchart for explaining details of the image conversion processing.

  The CPU 206 initializes the pixel position u of the converted image to 0 (step S241) and initializes the pixel position v of the converted image to 0 (step S242).

  Next, the CPU 206 substitutes the pixel position (u, v) of the converted image using the affine parameter A obtained in Expression 19, and obtains the pixel position (x, y) of the original image according to Expression 2. (Step S243). Subsequently, the CPU 206 obtains a pixel value P (u, v) from the obtained pixel position (x, y) by the bilinear method according to Equation 23 (step S244).

  Then, the CPU 206 increments the corrected image coordinate v by one (step S245).

  Subsequently, the CPU 206 compares the coordinate v of the corrected image with the maximum value vmax of the coordinate v to determine whether or not the coordinate v of the corrected image is equal to or greater than the maximum value vmax (step S246). When it is determined that v is less than the maximum value vmax (step S246; No), the process returns to step S243.

  Thereafter, the CPU 206 repeats the processing of steps S243 to S245, and when it is determined that the coordinate v has reached the maximum value vmax (step S246; Yes), the coordinate u of the corrected image is incremented by one (step S247). .

  Subsequently, the CPU 206 compares the coordinate u with the maximum value umax of the coordinate u, determines whether the coordinate u is equal to or greater than the maximum value umax (step S248), and if the coordinate u is less than the maximum value umax. When it determines (step S248; No), it returns to step S242.

  Thereafter, the CPU 206 repeats the processes of steps S242 to S247, and when it is determined that the coordinate u has reached the maximum value umax (step S248; Yes), the image conversion process is terminated.

  Next, a process for extracting an image sharpening parameter from the image thus obtained and an image sharpening process performed using this parameter will be described. The image sharpening process is a process for obtaining a clearer image.

  The image sharpening parameter is a variable necessary for image sharpening processing such as a maximum value, a minimum value, a peak value of a luminance histogram, a peak value of a color difference histogram, and an average value.

  In order to extract the image sharpening parameters, it is necessary to cut out the image of the actual original portion excluding the outline from the entire image and generate a histogram of luminance and color difference. First, a process necessary for extracting an image sharpening parameter will be described.

  As shown in FIG. 25, the image of the object to be photographed includes not only a substantial image such as a photograph, a figure, and a character, but also a pedestal 13 and a shadow of the sheet of the object to be photographed. There may be images that are not related to the content. An inner frame and an outer frame are set in this entire image, and an inner frame in which an actual document portion, a pedestal 13, a shadow of a sheet of the object to be photographed, and the like are reflected in the inner frame in which a substantial image of the object to be photographed is present. Between the outer frame and the outer frame. The outline of the object to be photographed in the periphery is generally black. In some cases, it is preferable to have such peripheral data in the image data.

  However, if a histogram is generated including an image of the peripheral part of the object to be photographed, stretching of the histogram may not function effectively.

  For this reason, only when extracting the image sharpening parameters, the image of the actual original portion is cut out to generate a histogram of brightness and color difference, and the image sharpening parameters are extracted from the histogram and image sharpening is performed for the entire image. Processing shall be performed. This is a more effective parameter.

  Specifically, if the image data including the peripheral portion is M rows and N columns and the number of dots between the outer frame and the inner frame is K dots in both the X and Y directions, on the X axis side, Pixel data from K to M-K rows that are pixel data of the actual document portion is acquired, and pixel data from K to NK columns that are pixel data of the actual document portion is acquired on the Y-axis side. The pixel data of the actual manuscript part is (M-2 * K) row and (N-2 * K) column data.

  A luminance histogram and a color difference histogram are generated for the image of the actual document portion cut out in this way.

  The luminance histogram shows the distribution of luminance values (Y) existing in the actual document portion, and is generated by counting the number of pixels in the actual document portion for each luminance value. FIG. 26A shows an example of a luminance histogram. In FIG. 26A, the horizontal axis represents the luminance value (Y), and the vertical axis represents the number of pixels. In order to sharpen an image, it is necessary to obtain a maximum value (Ymax), a minimum value (Ymin), and a peak value (Ypeak) as image sharpening parameters.

  The maximum value counts the number of pixels for each luminance value, and is a value indicating the maximum luminance among luminance values having a predetermined number or more set in advance, and the minimum value is a predetermined number or more This is a value indicating the minimum luminance among the luminance values having the counted value. The peak value is a luminance value that maximizes the count value. The peak value is considered to indicate the luminance value of the background color of the document 4 or the screen 3 to be photographed.

  The color difference histogram indicates the distribution of the color differences (U, V) existing in the actual document portion, and is generated by counting the number of pixels in the actual document portion for each color difference. FIG. 26B shows an example of the color difference histogram. In FIG. 26B, the horizontal axis indicates the color difference, and the vertical axis indicates the number of pixels. Also in the case of the color difference histogram, a peak value (Upeak, Vpeak) of the color difference where the count value of the pixel becomes maximum appears. This peak value is also considered to indicate the color difference of the background color of the object to be imaged. In order to sharpen an image, it is necessary to obtain a peak value and an average value (Umean, Vmean) of a color difference histogram as image sharpening parameters. The average value is a color difference value having an average count value.

  Note that in order to obtain an image with high visibility by sharpening the image, the correction effect differs depending on the background color of the object to be photographed, so the image effect correction method must be changed depending on the background color of the object to be photographed. . For this reason, it is necessary to determine the background color of the object to be photographed. The background color of the object to be photographed is determined from the peak values of the luminance histogram and the color difference histogram.

  Here, the background color of the photographing object is classified into three. The first is a case where the background color is white, such as a whiteboard or notebook. The second is a case where the background color is black, such as a blackboard. The third is a case where the background color is other than white or black, such as a magazine or a pamphlet. Specifically, the background color of the document 4 is determined according to the following determination formula.

  The white determination condition is expressed by the following Expression 24, and when the condition shown in Expression 24 is satisfied, the background color of the photographing object is determined to be white (W).

  The black determination condition is expressed by the following Expression 25, and when the condition shown in Expression 25 is satisfied, the background color of the shooting target is determined to be black (B).

  If the conditions shown in Equations 24 and 25 are not satisfied, the background color of the subject is determined to be color (C). For example, the color threshold is set to 50 and 128, the white determination threshold is set to 128, and the black determination threshold is set to 50, for example.

  Based on such a concept, the CPU 206 executes image sharpening parameter extraction processing. FIG. 27 is a flowchart showing details of the image sharpening parameter extraction processing.

  The CPU 206 counts the number of pixels having each luminance (Y) value in the actual document portion, generates a luminance histogram as shown in FIG. 26A (step S331), and generates the luminance histogram from the generated luminance histogram. The maximum value (Ymax), minimum value (Ymin), and peak value (Ypeak) of the luminance are acquired (step S332).

  Next, the CPU 206 creates a histogram of color differences (U, V) as shown in FIG. 26B (step S333). Subsequently, the CPU 206 obtains the peak value (Upeak, Vpeak) of the color difference (U, V) as an image sharpening parameter (step S334) and obtains the average value (Umean, Vmean) of the color difference (step S335).

  Then, the CPU 206 discriminates the background color of the object to be photographed from these peak values (Ypeak, Upeak, Vpeak) of the image histogram according to the judgment condition formulas shown in Formulas 24 and 25 (Step S336), and the image sharpening parameter. And the background color data of the object to be photographed are stored in the memory 201 (step S337).

  Next, the CPU 206 performs an image sharpening process using the image sharpening parameters extracted in this way. As described above, in order to effectively perform the image sharpening process, it is necessary to change the processing contents depending on the background color.

  When the background color is white, such as a whiteboard or notebook, luminance conversion as shown in FIG. When the background color is black, such as a blackboard, luminance conversion as shown in FIG. When the background color is other than white or black, such as a magazine or a pamphlet, the conversion as shown in FIG. In FIGS. 28A, 28 </ b> B, and 28 </ b> C, the horizontal axis indicates the input value of the pixel value, and the vertical axis indicates the output value of the pixel value.

  When the background color is white, as shown in FIG. 28A, the inclination angle of the luminance conversion line is changed with the peak value as a boundary. The predetermined luminance value is set to 230, for example, and the input luminance peak value is raised to the luminance value 230. Then, the maximum value is raised to the maximum brightness. Therefore, the luminance conversion line is represented by two line segments as shown in FIG.

  When the background color is black, luminance conversion is performed so that the peak value becomes a certain luminance value (20) as shown in FIG. Also in this case, as shown in FIG. 28B, the luminance conversion line is represented by two line segments.

  When the background color is a color other than white or black, as shown in FIG. 28C, the minimum value or less and the maximum value or more are cut and expressed as one line segment as in the normal enlargement process. The luminance conversion line is set as follows.

  It should be noted that a conversion table of background luminance (Y) and output (Y ′) may be set in advance and stored in the memory 201 as described above. According to the created conversion table, each output value is obtained from the value of each input pixel, and image sharpening processing is performed. In the image thus converted, bright pixels are brighter and dark pixels are darker, so that the luminance distribution is widened and the image is excellent in visibility.

  In addition, depending on the photographed image, the white balance of the digital camera may not be adjusted properly, and for example, the color may change to yellow or the like as a whole. This color change cannot be corrected only by performing the image sharpening process of the luminance histogram.

  In this case, color adjustment is performed to obtain a preferable image. FIG. 29 is a diagram illustrating an example of a luminance conversion graph for performing color adjustment. In FIG. 29, the horizontal axis indicates the input value of the color difference, and the vertical axis indicates the output value of the color difference. In the color adjustment, luminance conversion is performed according to the luminance conversion graph shown in FIG. 29 so that the average values (Umean, Vmean) of the color differences (U, V) shown in FIG.

  When the values of the color differences U and V are both 0, the color is an achromatic color, and color adjustment is performed so that the peak values (Upeak, Vpeak) are 0. That is, the color conversion line is represented by two line segments. The output value U ′ for the input value U is given by the following equation (26).

  The same applies to the color difference V.

  Next, as the image sharpening process, when the background is not sufficiently whitened only by the above-described luminance extension, the background is whitened, and the color of the portion considered to be the background in the image is set to white (Y: 255, U : 0, V: 0) or a color close to that.

  FIG. 30A is a diagram illustrating a luminance value increase rate with respect to the luminance value of a pixel, with the luminance value of a certain pixel as a reference (0). The horizontal axis represents the luminance value, and the vertical axis represents the luminance value raising rate. FIG. 30B is a diagram showing the color difference and the change rate of the color difference. The horizontal axis indicates the color difference, and the vertical axis indicates the change rate of the color difference.

  In the figure, C0 indicates a range from 0 where the luminance and the color difference are increased by 100%, and C1 indicates a range where the increase rate is varied according to the luminance value. As shown in FIGS. 30A and 30B, the color of the pixel whose luminance (Y) value is equal to or greater than a certain value (Yw) and whose color difference (U, V) is within a certain range (C0) is defined as a whitening range. The brightness value is raised and the color difference is set to zero. If such whitening of the background is performed, the image sharpening process becomes very effective when the background is not sufficiently whitened only by extending the luminance.

  Based on such a concept, the CPU 206 executes an image sharpening process. FIG. 31 is a flowchart showing details of the image sharpening process.

  The CPU 206 reads the stored image sharpening parameter from the memory 201 (step S251).

  Next, the CPU 206 adjusts the luminance histogram (step S252). Specifically, the CPU 206 determines whether or not the background is white. If the background is determined to be white, the CPU 206 makes the background whiter and the visibility is improved as shown in FIG. Luminance conversion is performed to adjust the luminance histogram. On the other hand, when it is determined that the background is not white, the CPU 206 determines whether or not the background is black, and when it is determined that the background is black, the CPU 206 performs luminance conversion as shown in FIG. Adjust the brightness histogram. On the other hand, if it is determined that the background is not black, the CPU 206 performs luminance conversion as shown in FIG. 28C and adjusts the histogram according to the background color of the shooting target.

  Subsequently, the CPU 206 performs color adjustment by performing conversion as shown in FIG. 25 on the image adjusted in this way (step S253), and subsequently executes background whitening processing (step S254). ).

  FIG. 32 is a flowchart showing details of the background whitening process performed by the CPU 206. Note that imax and jmax in the figure are the sizes of the x and y axes of the input image.

  In this background whitening process, the CPU 206 first initializes the count value j to 0 (step S341) and initializes the count value i to 0 (step S342). Next, the CPU 206 determines whether or not the luminance (Y) of the pixel (i, j) of the input image is equal to or greater than a certain value (Yw) (step S343).

  When it is determined that the luminance (Y) is equal to or greater than a certain value (Yw) (step S343; Yes), the CPU 206 determines whether or not the absolute values of the color differences U and V are less than the predetermined value C0 (step S344). ). On the other hand, if it is determined that the luminance (Y) is less than a certain value (Yw) (step S343; No), the CPU 206 proceeds to step S348.

  When it is determined that the absolute values of the color differences U and V are not less than the predetermined value C0 (step S344; No), the CPU 206 has the absolute values of the color differences U and V of the pixel (i, j) of the input image less than the predetermined value C1. It is determined whether or not (step S345).

  When it is determined in steps S344 and S345 that the absolute values of the color differences U and V are less than the predetermined value C0 (step S344; Yes), the CPU 206 sets the luminance value (Y) to 255 and the color difference (u, v ) Is set to 0, the value of the pixel (i, j) is rewritten (step S346), and the process proceeds to step S348.

  When it is determined in steps S344 and S345 that the absolute values of the color differences U and V are greater than or equal to the predetermined value C0 and less than the predetermined value C1 (step S344; No, step S345; Yes), the CPU 206 determines that the luminance value Y = Y + a * (255−Y), the value of the pixel (i, j) is rewritten (step S347), and the process proceeds to step S348.

  If it is determined in steps S344 and S345 that the absolute values of the color differences U and V are greater than or equal to the predetermined value C1 (step S344; No, step S345; No), the CPU 206 proceeds to step S348 without changing the luminance value. move on.

  Then, the CPU 206 increments the count value i (step S348). Subsequently, the CPU 206 compares the count value i with the maximum value imax and determines whether or not the count value i has reached the maximum value imax (step S349).

  When it is determined that the count value i has not reached the maximum value imax (step S349; No), the CPU 206 returns to step S343.

  Thereafter, the processes in steps S343 to S348 are repeatedly executed, and when it is determined that the count value i has reached the maximum value imax (step S349; Yes), the CPU 206 increments the count value j (step S350).

  Subsequently, the CPU 206 compares the count value j with the maximum value jmax to determine whether or not the count value j has reached the maximum value jmax (step S351).

  When it is determined that the count value j has not reached the maximum value jmax (step S351; No), the CPU 206 returns to step S342.

  Thereafter, the processes of steps S342 to S350 are repeatedly executed, and when it is determined that the count value j has reached the maximum value jmax (step S351; Yes), the CPU 206 ends the whitening process of the background.

  As described above, according to the present embodiment, the CPU 206 generates an edge image from an image obtained by photographing the document 4 on the pedestal 13 and forms the contour of the document 4 from the generated edge image. Detect the rectangle to be used. Next, the CPU 206 determines whether or not the quadrilateral of the original 4 has been detected from the edge image. If it is determined that the quadrilateral of the original 4 has been detected, the CPU 206 obtains the coordinates of each vertex of the detected quadrilateral of the original 4. On the other hand, if it is determined that it has not been detected, the coordinates of each vertex of the quadrangle that forms the contour of the pedestal 13 registered in the memory 201 are obtained.

  Then, the CPU 206 acquires affine parameters from the obtained coordinates of the respective vertices of the quadrangle, affine-transforms the coordinates of each pixel of the captured image using the acquired affine parameters, and corrects the inclination of the captured image.

  For example, as shown in FIG. 3A, when the document 4 is within the angle of view (shooting range), the CPU 206 can detect the square of the document 4 from the edge image. The image is cut out along the outline. Then, the CPU 206 performs an inclination correction or an image sharpening process on the cut-out image, and then outputs it. As a result, an image as shown in FIG. 3B is projected on the screen 3.

  On the other hand, when the document 4 protrudes from the angle of view as shown in FIG. 33A, the CPU 206 cannot detect the rectangle of the document 4, and therefore the rectangle registered in the memory 201 (the outline of the base 13). ) To crop the image. Then, the CPU 206 performs an inclination correction or an image sharpening process on the cut-out image, and then outputs it. As a result, an image as shown in FIG. 33B is projected on the screen 3.

  As described above, when the document 4 is within the angle of view, the image is cut out along the outline of the document 4, and when it is beyond the angle of view, the image is cut out along the outline of the pedestal 13. Is done. For this reason, the CPU 206 does not need to reduce the zoom magnification of the camera unit 11 even when the image of the photographed document 4 protrudes from the angle of view, so that the ratio of the document 4 to the entire photographed image can be increased. The image that is cut out and projected onto the screen 3 also becomes larger. As a result, the captured image projection device of the present embodiment can project an image with excellent visibility onto the screen 3.

  Even if the captured image is distorted, the ratio of the document 4 to the entire captured image can be increased as described above. Therefore, the captured image projection apparatus according to the present embodiment can save the captured image. The luminance and color difference can be corrected appropriately, and as a result, the visibility of the image projected on the screen 3 can be further improved.

  Further, as shown in FIG. 34A, the camera unit 11 photographs the document 4 placed on the pedestal 13 from directly above, but when there is a light reflection or the like, FIG. ) Take a picture from a slightly oblique position to reduce the reflection of light.

  FIG. 35A shows an example of an image obtained when the document 4 is photographed from directly above, and FIG. 35B shows an example of an image obtained when the document 4 is photographed from a slightly oblique position. Is shown.

  The CPU 206 extracts the outline of the document 4 from the photographed images shown in FIGS. 35A and 35B and cuts out the image of the document 4 by the above-described method, so that the image shown in FIG. Images can be acquired. As a result, the captured image projection apparatus according to the present embodiment is less susceptible to the effects of light reflection. Furthermore, the photographed image projection apparatus according to the present embodiment can obtain the same effect by extracting the contour of the pedestal 13 and cutting out the image even when the document 4 protrudes from the angle of view. .

  Furthermore, when the positional relationship between the camera unit 11 and the pedestal 13 changes due to a change in the tilt angle of the camera unit 11 with respect to the pedestal 13, when the user presses the calibration request key, the CPU 206 captures the pedestal 13. A set of linear parameters forming the contour of the pedestal 13 is detected from the image obtained by doing so and registered in the memory 201. As a result, when the positional relationship between the camera unit 11 and the pedestal 13 changes, the CPU 206 can sequentially update the set of linear parameters registered in the memory 201 accordingly. By using the set of linear parameters that are sequentially updated in this manner, the captured image projection apparatus according to the present embodiment performs the cut-out of the image of the document 4 from the captured image and the inclination correction. Even when the positional relationship with the pedestal 13 changes, the front image of the document 4 can be appropriately acquired from the captured image.

  In addition, this invention is not limited to the said embodiment, A various deformation | transformation and application are possible. Hereinafter, modifications of the above-described embodiment applicable to the present invention will be described.

  In the above embodiment, the support column 12 is rotatably connected to the camera unit 11 and the pedestal 13. However, the present invention is not limited to this, and the support column 12 may be fixedly connected to the camera unit 11 and the pedestal 13, and connected so as to be locked at a predetermined position by a notch mechanism or the like. May be. As described above, when the positional relationship between the camera unit 11 and the pedestal 13 is determined in advance, the pedestal registration process in step S111 is omitted, and a set of linear parameters corresponding to each positional relationship is registered in the memory 201. Each time the positional relationship fluctuates, the set of linear parameters to be used may be switched sequentially.

  Further, in the above embodiment, the CPU 206 uses the straight line parameters close to the straight line parameters forming the contour of the pedestal 13 stored in the memory 201 in the pedestal registration process in step S111 from the candidates acquired in steps S303 and S304. I was off. However, the present invention is not limited to this, and the CPU 206 removes in advance the straight line parameters that form the contour of the pedestal 13 from the edge image obtained in step S301, and detects them as candidates in steps S303 and S304. It may not be done. In addition, as illustrated in FIG. 36, the CPU 206 may cut out an image slightly inside the outline of the pedestal 13 and detect a square from the cut out image.

  Further, in the above embodiment, the shape of the outline of the document 4 is a quadrangle. However, the present invention is not limited to this, and the shape of the document is arbitrary, and may be, for example, a pentagon. In this case, the captured image projection apparatus may acquire a pentagon from the outline of the document 4 and extract projection parameters from the vertex coordinates of the pentagon.

  In the above embodiment, the image processing process is applied to the photographic image projection apparatus. However, the present invention is not limited to this, and may be applied to other photographic apparatuses such as a digital camera. .

  Furthermore, in the above-described embodiment, the program executed by the CPU 236 or the like is stored in advance in the ROM or the like. However, the present invention is not limited to this, and a program for causing a computer to operate as all or part of the apparatus or to execute the above-described processing is stored in a flexible disk, a CD-ROM (Compact Disk Read-Only). It is stored and distributed in a computer-readable recording medium such as a memory (DVD), a digital versatile disk (MO), or a magneto optical disk (MO), and this is installed in another computer and operated as the above-mentioned means, or These steps may be executed.

  Further, the storage medium for storing the program is not limited to the above-described ones. For example, a next-generation optical disk storage medium using a blue laser such as Blue-Ray-Disc (R) or AOD (Advanced Optical Disc), a red laser Needless to say, the present invention can be implemented using various types of large-capacity storage media that will be developed in the future, such as HD-DVD9 using a blue-ray and Blue-Laser-DVD using a blue-violet laser.

  Furthermore, the program may be stored in a disk device or the like included in a server device on the Internet, and may be downloaded onto a computer by being superimposed on a carrier wave, for example.

It is a figure which shows the structure of the picked-up image projection apparatus which concerns on embodiment of this invention, (a) is a figure at the time of document photography, (b) is a figure at the time of board photography. It is a block diagram which shows the structure of the document camera shown in FIG. It is a figure for demonstrating the function of the image processing apparatus shown in FIG. It is a figure for demonstrating the function of the image processing apparatus shown in FIG. It is a flowchart which shows a document camera basic process. It is a flowchart which shows PC document drawing basic processing. 6 is a flowchart showing details of the document photographing process shown in FIG. 5. 6 is a flowchart showing details of the document photographing process shown in FIG. 5. It is a flowchart which shows the detail of the board | substrate imaging | photography process shown in FIG. It is a figure for demonstrating the clipping process of the image processing apparatus shown in FIG. It is explanatory drawing of the basic idea of affine parameter extraction and affine transformation. It is a flowchart which shows the detail of the square outline extraction process shown in FIG. It is a figure for demonstrating a reduced luminance image and an edge image. It is a figure for demonstrating the function of a Roberts filter. It is a figure for demonstrating the straight line detection using Hough transform, (a) shows the process which detects an upper and lower side, (b) has shown the process which detects a right and left side. It is an image figure which shows that upper and lower sides or right and left can be determined depending on whether ρ is a positive value or a negative value. It is a flowchart which shows the detail of the base registration process shown in FIG. It is a flowchart which shows the detail of a base registration change process. It is a flowchart which shows the detail of a rectangle detection process at the time of document photography. It is a figure for demonstrating specification of the rectangle after a straight line detection, (a) shows the case of document photography, (b) has shown the case of board photography. It is a flowchart which shows the detail of a rectangle detection process at the time of board photography. It is a flowchart which shows the detail of the affine parameter extraction process shown in FIG. It is a figure for demonstrating the inverse transformation for obtaining the original image from the image after projective transformation. It is a flowchart which shows the detail of the image conversion process shown in FIG. It is a figure which shows the example of an image cut out by the image processing apparatus shown in FIG. FIG. 26A is a diagram illustrating a luminance histogram, and FIG. 26B is a diagram illustrating a color difference histogram. It is a flowchart which shows the detail of an image sharpening parameter extraction process. FIG. 28A is a diagram showing image sharpening processing when the background color is white, and FIG. 28B is a diagram showing image sharpening processing when the background is black. c) is a diagram showing image sharpening processing when the background is other than white or black. It is a figure which shows an example of the luminance conversion graph for performing color adjustment. It is a figure for demonstrating whitening of a background. It is a flowchart which shows the detail of the image sharpening process shown in FIG. It is a flowchart which shows the detail of the background whitening process shown in FIG. It is a figure for demonstrating the function of the image processing apparatus shown in FIG. It is a figure for demonstrating the function of the document camera shown in FIG. It is a figure for demonstrating the function of the image processing apparatus shown in FIG. It is a figure for demonstrating the function of the image processing apparatus in the modification of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Document camera, 2 ... Projector, 3 ... Screen, 4 ... Document, 11 ... Camera part, 13 ... Base, 201 ... Memory, 203 ... Image processing apparatus, 204 ... Operation part, 206 ... CPU

Claims (9)

Data storage means for storing the vertex coordinates of the polygon forming the contour of the pedestal;
A document polygon detection means for detecting the polygon forming the outline of the document from the obtained image by taking the original on the pedestal,
An original polygon detection determining means for determining whether or not a polygon forming the outline of the original is detected by the original polygon detecting means;
When it is determined that the polygon forming the document outline is detected by the document polygon detection determining means , image processing is performed within a range specified by the vertex coordinates of the detected polygon, and the document outline is formed. When it is determined that the polygon to be detected has not been detected, image processing means for image processing in a range specified by the vertex coordinates stored by the data storage means,
An image processing apparatus comprising: The document polygon detection means includes candidate detection means for detecting straight line parameter candidates that form the outline of the document from the image;
Document outline acquisition means for detecting a set of straight line parameters satisfying a predetermined condition from candidates detected by the candidate detection means and acquiring a polygon formed by the detected line parameter set as the outline of the original. Including,
The image processing apparatus according to claim 1. The document outline obtaining unit detects a set of two substantially parallel linear parameters from the candidates detected by the candidate detecting unit, and two straight lines substantially perpendicular to the detected two linear parameters. A set of parameters is detected, and a polygon formed by the detected set of straight line parameters is acquired as an outline of the document;
The image processing apparatus according to claim 2. Affine parameter acquisition means for acquiring affine parameters from vertex coordinates in a range specified by the image processing means;
Image correction means for correcting and outputting the inclination of the image by affine transformation of the coordinates of each pixel included in the image using the affine parameter acquired by the affine parameter acquisition means;
The image processing apparatus according to claim 1, further comprising: A pedestal polygon detecting means for detecting a polygon that forms an outline of the pedestal from an image obtained by photographing the pedestal;
Pedestal polygon detection discriminating means for discriminating whether or not a polygon forming the outline of the pedestal has been detected by the pedestal polygon detecting means;
When it is determined that the polygon that forms the contour of the pedestal is detected by the pedestal polygon detection determining means, the vertex coordinate data of the detected polygon is written in the data storage means, thereby forming the contour of the pedestal Data updating means for updating vertex coordinate data of the polygon to be further included,
The image processing apparatus according to claim 1 , wherein the image processing apparatus is an image processing apparatus. The pedestal polygon detecting means detects a set of two substantially parallel linear parameters from an image obtained by photographing the pedestal, and is substantially perpendicular to the detected two linear parameters. Detecting a set of two linear parameters, and obtaining a polygon formed by the detected linear parameter set as an outline of the pedestal;
The image processing apparatus according to claim 5. Data storage means for storing the vertex coordinates of the polygon forming the contour of the pedestal;
A document imaging means for photographing an original placed on the pedestal,
A document polygon detecting means for detecting a polygon forming the contour of the document from an image obtained by photographing by the document photographing means;
An original polygon detection determining means for determining whether or not a polygon forming the outline of the original is detected by the original polygon detecting means;
When it is determined that the polygon forming the document outline is detected by the document polygon detection determining means , image processing is performed within a range specified by the vertex coordinates of the detected polygon, and the document outline is formed. When it is determined that the polygon to be detected has not been detected, image processing means for image processing in a range specified by the vertex coordinates stored by the data storage means,
Image projecting means for projecting the image output from the image processing means onto a screen;
A photographic image projection apparatus comprising: A data storage step for storing the vertex coordinates of the polygon forming the outline of the pedestal;
A document polygon detection step of detecting a polygon which forms the outline of the document from the obtained image by taking the original on the pedestal,
An original polygon detection determining step for determining whether or not a polygon forming the outline of the original is detected by the original polygon detecting step;
When it is determined in the original polygon detection determination step that a polygon forming the outline of the original has been detected , image processing is performed within the range specified by the vertex coordinates of the detected polygon to form the outline of the original An image processing step of performing image processing in a range specified by the vertex coordinates stored by the data storage means,
An image processing method comprising: On the computer,
A data storage procedure for storing the vertex coordinates of the polygon forming the contour of the pedestal;
A document polygon detection procedure for detecting the polygon forming the outline of the document from the obtained image by taking the original on the pedestal,
A document polygon detection determination procedure for determining whether or not a polygon forming the contour of the document has been detected by the document polygon detection procedure;
When it is determined that the polygon forming the outline of the original is detected by the original polygon detection determination procedure , image processing is performed within a range specified by the vertex coordinates of the detected polygon, and the outline of the original is formed. An image processing procedure for performing image processing in a range specified by the vertex coordinates stored by the data storage means when it is determined that the polygon to be detected has not been detected ;
A program for running