JP2007109102A - Image processing method and image processing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To automatically scale an input image to an output image size even for such an image as involves an error in relative positioning between a subject and an imaging device, such as a digital camera. <P>SOLUTION: An image processing device comprises a means for extracting connected components in an input image, a means for calculating a set of connected components in an area corresponding to an output image, and a means for calculating a set of connected components in an area corresponding to a specified scrolling amount. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing method, a computer program, and an image processing apparatus, and more particularly to a technique suitable for use in enlarging or reducing an image.

  In the future, as an image processing method for displaying an image on which an input image is enlarged or reduced and displayed on an output image as an input image, such as an image input by a scanner or the like, in which there is no relative displacement between the subject and the imaging device, As shown in Patent Document 1, there is one that divides lines and characters based on a histogram.

JP 2000-163044 A JP 2003-288588 A Japanese Patent Laid-Open No. 2003-50005

  However, an image processing method that automatically enlarges / reduces an input image in accordance with the output image size is desired even when there is a relative displacement between the subject and the imaging device, such as an image taken with a digital camera. It is rare.

  The present invention has been made in view of the above-described problems, and an object thereof is to enable an input image to be automatically enlarged / reduced in accordance with an output image size.

  In order to meet this demand, the present invention is an image processing method that corrects distortion and inclination of an input image and outputs the image by enlarging or reducing the image. The image processing method includes an input process for inputting an image, an output process for outputting an image, An instruction step for instructing scrolling of an image output in the output step, an extraction step for extracting a connected component for the image input in the input step, and a connected component of a region corresponding to the image output in the output step Provided is an image processing method comprising: a first calculation step for obtaining a set; and a second calculation step for obtaining a set of connected components in a region corresponding to the scroll amount instructed in the instruction step. To do.

  According to the present invention, even when there is a relative positional shift between the subject and the imaging device, the magnification of the input image can be automatically set, and the scroll range can be automatically set.

(First embodiment)
Embodiments of the present invention will be described with reference to the drawings.
FIG. 17 is a diagram illustrating an internal configuration example of a digital camera as the present embodiment.
In FIG. 17, reference numeral 1701 denotes an imaging unit including a lens, a shutter, a diaphragm, a CCD, and the like. Reference numeral 1702 denotes an A / D converter that converts an analog video signal imaged by the imaging unit 1701 into digital image data. A signal processing circuit 1703 performs processing such as gamma correction and white balance (WB) correction on the image data output from the A / D converter 1702, and outputs the data to the bus 1704. The image data is stored in the memory 1706. Is done.

  Reference numeral 1705 denotes a compression circuit, which compresses and encodes an image signal in accordance with a well-known JPEG format. Reference numeral 1706 denotes a memory which stores the image signal input from the signal processing circuit 1703 and outputs it to the circuits of the respective devices via the buses 1704 and 1709. Also, the image signal from each circuit is stored.

  A display unit 1707 displays an image input from the memory 1706, an image stored in the storage device 411, or a menu screen.

  Reference numeral 1710 denotes an operation unit that operates and controls the operation of the digital camera 1700, and includes various switches such as a menu button and a playback button. Reference numeral 1711 denotes an MPU (Micro Processing Unit) which controls each circuit of the apparatus according to software stored in a memory (not shown). In the present embodiment, the processes shown in FIGS.

  A storage device 1712 is a flash memory card or the like that can be detached from the camera, stores an image signal input via the bus 1709, reads out the stored image signal, and outputs it to each circuit via the bus 1709. .

  Reference numeral 1713 denotes an external communication interface that transfers image signals to / from an external information device 1715 via a transmission path 1714 such as USB (Universal Serial Bus), IEEE 1394, wireless LAN, or Bluetooth.

FIG. 1 is a flowchart showing a processing procedure for a part of simple image processing.
In step S107, it is determined what to do. If an image is input, the image is input in step S101, and a connected component of the input image is detected in step S102.

  If the display range is set, the display range is set in step S103, and the range to be output is automatically calculated in step S104. If the display range is to be moved, the display range is selected in step S105, and the display range is moved in step S106.

  An example in which there is a relative displacement between the subject 202 and the digital camera 201 and distortion or tilt occurs in the imaging result 203 is shown in FIG. An example of the imaging result 203 at this time is shown in FIG.

A digital camera 201 is shown in FIG.
The digital camera 201 includes a display 401, a display range moving unit 402, a display area moving unit 403, an image display request unit 404, a display range enlargement / reduction request unit 405, and an imaging device 406.

6 to 8 are flowcharts showing processing procedures inside the digital camera 201.
Assume that the imaging device 406 has obtained the imaging result 203. At this time, the digital camera 201 is driven in step S601 in FIG.

  Next, in step S602, a connected component is detected for the input image. Thereafter, in the case where one character is formed by a plurality of connected components such as “I”, “ko”, and “ga”, adjacent connected components are joined together to obtain a connected component indicating one character.

  Next, in step S603, the input image is divided into regions. An example of an area obtained by this area division is shown at 901 in FIG. For example, Patent Document 2 describes an area division method for an image having perspective distortion and inclination.

  In step S604, a geometric transformation matrix for correcting the tilt and distortion of the input image is obtained. For example, Patent Document 2 and Patent Document 3 describe a method for deriving a geometric transformation matrix for correcting inclination and distortion and examples of the geometric transformation matrix.

  Next, in step S605, an inverse transformation matrix of the geometric transformation matrix is obtained. This matrix indicates a conversion matrix from the output image to the input image. Next, in step S606, a transition is made to image output.

  In FIG. 7, what is to be performed is determined in step S706, and it is assumed that the result of this determination is transition to image output in step S605. Alternatively, when it is determined to output an image, such as when an image output is requested by the image display requesting unit 404, step S701 is driven.

  Next, in step S702, an area to be displayed is determined from the areas obtained in step S603. The determination method is arbitrary. Next, in step S703, the process proceeds to display range setting.

  On the other hand, as a result of the determination in step S706, if the display area moving unit 403 determines to move the display area, such as when the display area is requested to move, step S704 is driven. In step S705, the display area is moved in accordance with the requested movement of the display area. Next, in step S703, the process proceeds to display range setting.

  Next, what is to be performed is determined in step S815 in FIG. 8, and if it is determined that the display range is to be set as a result of this determination, such as when the display range is set in step S703, step S801 is performed. Driven. Next, in step S802, a display range initial value is determined. The display range initial value is obtained from the size of the connected component obtained in step S602 and the number of characters and lines to be displayed on the display 401 at a time.

  That is, when the display area 901 in the display area obtained in step S603 is to be displayed, the origin 902 is set in the display area 901 in the input image. Then, from this origin 902, an area 1001 occupied by connected components corresponding to the number of characters and the number of lines to be displayed at once in the display 401 is obtained and set as a display range initial value.

  The number of characters and the number of lines to be displayed at a time in the display 401 are values set in advance or increased or decreased by the display range expansion / reduction request unit 405. In setting the origin 902, the orientation of the image may be automatically determined, or a means for the user to explicitly specify the orientation may be provided.

  On the other hand, as a result of the determination in step S815, when it is determined that the display range is to be moved, such as when the display range moving means 402 requests movement of the display range, step S803 is driven. In step S804, the display range is moved in accordance with the requested movement of the display range. This display range is obtained from the currently displayed range 1001, the movement amount instructed by the display range moving means 402, and the connected component obtained in step S602. That is, as shown in FIG. 10, a display area is an area 1002 that is moved from the currently displayed range 1001 by an area occupied by a connected component corresponding to the movement amount instructed by the display range moving unit 402.

  On the other hand, as a result of the determination in step S815, when it is determined to enlarge or reduce the display range, such as when the display range enlargement / reduction request unit 405 requests expansion or reduction of the display range, step S805 is driven. In step S806, the display range is enlarged or reduced. The enlargement or reduction of the display range is performed based on the size of the connected component obtained in step S602 and the number of characters and lines to be displayed on the display 401 at a time. That is, the number of characters and the number of lines to be displayed at once in the display 401 are increased or decreased according to the requested image enlargement or reduction.

  Next, the origin 902 is set in the display area 901 in the input image shown in FIG. A region 1001 occupied by connected components corresponding to the number of characters and the number of lines to be displayed on one screen from the origin 902 is set as a display range. The origin 902 to be set is preferably the center or one of the four corners of the currently displayed area.

  On the other hand, as a result of the determination in step S815, when it is determined that the display range is to be pre-read, such as when the CPU is in an idle state or the like and a pre-read of the display range is requested, step S807 is driven. Next, in step S808, the next movement is estimated from the movement history of the display range and the display area, and the area after the movement is selected. Alternatively, an area in the vicinity of the currently displayed range is set as a display range for prefetching.

  Next, in step S809, it is searched whether or not a part or all of the display range 1001 is included in the corrected image history 1101 that is a set of corrected images 1102 as shown in FIG. Only the display range difference 1003 between the corrected image history 1101 and the display range 1001 is set as a range to be corrected. Also, a portion 1103 included in the corrected image history 1101 in the display range 1001 is obtained.

  Next, in step S810, the inclination and distortion of the display range difference 1003 are corrected. This applies the transformation matrix obtained in step S604 to the display range difference 1003 obtained in step S809. As a result, a converted image 1104 is obtained.

  In step S811, the converted image 1104 is added to the corrected image history 1101. At this time, the display range difference 1003 and the converted image 1104 are recorded in association with each other to facilitate the search in step S809. Further, when the storage area is limited and the converted image 1104 cannot be recorded, the corrected image 1102 existing at the farthest place from the cost range difference 1003 is deleted.

  In step S812, the converted image 1104 is combined with the portion 1103 included in the corrected image history 1101 in the display range 1001. Next, in step S813, the obtained image is enlarged or reduced according to the resolution of the display 401 to obtain an output image 1105.

  Next, in step S814, the output image 1105 is output to the display 401. However, in step S807, that is, when the pre-reading of the movement of the display range is driven, output to the display 401 should not be performed.

As described above, in the digital camera 201 according to the present embodiment, first, image correction of the range 1001 displayed in the display 401 is performed on the captured image FIG. As a result, the output image 1105 is obtained faster and with a smaller storage area than when the entire image is corrected. In addition, the converted image and operation history are recorded, and prefetching is performed based on the operation history, so that even when the user operates the display range moving unit 402 or the display area moving unit 403, a quick response is made.
The image processing apparatus in this embodiment is not limited to a digital camera, and may be a mobile phone, a portable information terminal, or other devices.

(Second Embodiment)
A second embodiment will be described. Note that the description of the same parts (FIGS. 2 to 4 and 17) as in the first embodiment will be omitted.
The form of the present embodiment is shown in FIG. In FIG. 5, the digital camera 201 and the image correction server 501 can communicate with each other by some means such as USB, LAN, wireless LAN, Bluetooth, and the Internet.

12 to 13 are flowcharts of processing procedures inside the image correction server 501.
Also, flowcharts of processing procedures inside the digital camera 201 are shown in FIGS.

  In the digital camera 201, it is assumed that the imaging device 406 has obtained the imaging result 203. At this time, step S1401 in FIG. 14 is driven. In step S <b> 1402, the imaging result 203 is transmitted to the image correction server 501.

  When the image correction server 501 receives an image, step S1201 in FIG. 12 is driven, and an imaging result 203 is obtained. Next, in step S1202, a connected component is detected for the input image. The method for deriving this connected component is the same as in step S602.

  In step S1203, the input image is divided into regions. An example of an area obtained by this area division is shown at 901 in FIG. For example, Patent Document 2 discloses an area dividing method for an image having perspective distortion and inclination.

  In step S1204, a geometric transformation matrix for correcting the inclination and distortion of the input image is obtained. For example, Patent Document 2 and Patent Document 3 describe a method for deriving a geometric transformation matrix for correcting inclination and distortion and examples of the geometric transformation matrix.

  Next, in step S1205, an inverse transformation matrix of the geometric transformation matrix is obtained. This matrix indicates a conversion matrix from the output image to the input image. In step S <b> 1206, information related to the end of image input and the divided area is transmitted to the digital camera 201.

  Next, in FIG. 15, it is determined what to do in step S1507. As a result of this determination, if the digital camera 201 determines to receive a divided area, such as when image input is completed and information regarding the divided area is received, step S1501 is driven. On the other hand, if it is determined in step S1507 that there is an image output request, such as when image output is requested by the image display requesting unit 404, step S1502 is driven.

  Next, in step S1503, an area to be displayed is determined from the areas obtained in step S1203. The determination method is arbitrary. Next, in step S1504, the process proceeds to display range setting.

  On the other hand, as a result of the determination in step S1507, when the display area moving unit 403 determines to move the display area, for example, when the display area is requested to move, step S1505 is driven. In step S1506, the display area is moved in accordance with the requested movement of the display area. Next, in step S1504, the process proceeds to display range setting.

  Next, in FIG. 16, it is determined what to do in step S <b> 1617, and as a result of this determination, it is determined in the digital camera 201 that the display range is set, such as when the display range is set in step S <b> 1504. If so, step S1601 is driven. Next, in step S1602, a display range initial value determination request is transmitted.

  On the other hand, if it is determined in step S1617 that the display range enlargement / reduction is requested by the display range enlargement / reduction request unit 405 in the digital camera 201, step S1603 is executed. Driven. Next, in step S1604, a display range expansion / contraction request is transmitted.

  On the other hand, if it is determined in step S1617 that the display range is to be moved in the digital camera 201, such as when the display range is moved by the display range moving unit 402, step S1605 is driven. In step S1608, a display range movement request, a movement direction, and a movement amount are transmitted.

  On the other hand, as a result of the determination in step S1617, if it is determined in the digital camera 201 that the CPU is in an idle state and the display range movement is prefetched such as when the display range movement is prefetched, step S1606 is performed. Is driven. Next, in step S1607, the next movement is estimated from the movement history of the display range and the display area.

  In step S1608, a display range movement request, a movement direction, and a movement amount are transmitted.

  In step S1609, the display range is received. Next, in step S1610, a corrected image is searched from the corrected image history. This search method and image correction range derivation method are the same as step S809 of the first embodiment. In step S 1611, the image correction range and the image correction request are transmitted to the image correction server 501.

  Next, in step S1612, a corrected image is received. In step S1613, the converted image is added to the corrected image history. This addition method is the same as step S811 of the first embodiment.

  Next, in step S1614, the corrected image and the corrected image are combined. This addition method is equivalent to step S812 in the first embodiment. Next, in step S1615, the obtained image is enlarged or reduced in accordance with the output image. This enlargement / reduction method is equivalent to step S813 of the first embodiment.

  Next, in step S1616, the output image 1105 is output to the display 401. However, in step S1606, that is, when the pre-reading of the movement of the display range is driven, output to the display 401 should not be performed.

  Next, in FIG. 13, it is determined what to do in step S1311. As a result of this determination, if the image correction server 501 determines to determine the display range initial value, such as when a display range initial value determination request is received, step S1301 is driven. Next, in step S1302, a display range initial value is determined. The method for deriving the display range initial value is the same as that in step S802 of the first embodiment. In step S 1305, the display range is transmitted to the digital camera 201.

  On the other hand, as a result of the determination in step S1311, when the image correction server 501 determines to move the display range, such as when a display range movement request, a movement direction, and a movement amount are received, step S1303 is driven. In step S1304, the display range is moved according to the received movement direction and movement amount. The method for deriving the display range after movement is the same as step S804 in the first embodiment. In step S1305, the display range is transmitted to the digital camera 201.

  On the other hand, as a result of the determination in step S1311, when the image correction server 501 determines that the display range is to be expanded, such as when a display range expansion request is received, step S1306 is driven. In step S1307, the display range is enlarged or reduced. This enlargement or reduction of the display range is equivalent to step S806 in the first embodiment. In step S1305, the display range is transmitted to the digital camera 201.

On the other hand, as a result of the determination in step S1311, if the image correction server 501 determines that image correction is to be performed, such as when an image correction range and an image correction request are received, step S1308 is driven. Next, in step S1309, the inclination and distortion within the range are corrected. This correction method is the same as step S810 in the first embodiment. Next, in step S1310, the converted image is transmitted to the digital camera 201.
As described above, in the digital camera 201 according to the present embodiment, only the image correction of the range 1001 displayed in the display 401 is first performed on the captured image FIG. As a result, the output image 1105 is obtained faster and with a smaller storage area than when the entire image is corrected. In addition, the converted image and operation history are recorded, and prefetching is performed based on the operation history, so that even when the user operates the display range moving unit 402 or the display area moving unit 403, a quick response is made.

  In addition, since processing such as Connected Component analysis and image correction is performed on the server side, a large memory and a fast CPU speed are not required on the digital camera 201 side.

(Other embodiments according to the present invention)
The image processing apparatus in the embodiment is not limited to a combination of a digital camera and a server. Each means constituting the image processing apparatus and each step of the image processing method in the embodiment of the present invention described above can be realized by operating a program stored in a RAM or ROM of a computer. This program and a computer-readable recording medium recording the program are included in the present invention.

  Further, the present invention can be implemented as, for example, a system, apparatus, method, program, or recording medium. Specifically, the present invention may be applied to a system including a plurality of devices. The present invention may be applied to an apparatus composed of a single device.

  Note that the present invention supplies a software program that implements the functions of the above-described embodiments directly or remotely to a system or apparatus. This includes the case where the system or the computer of the apparatus is also achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Examples of the recording medium for supplying the program include a floppy (registered trademark) disk, a hard disk, an optical disk, and a magneto-optical disk. Further, there are MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R) and the like.

  As another program supply method, there is a method of connecting to a homepage on the Internet using a browser of a client computer. The computer program itself of the present invention or a compressed file including an automatic installation function can be downloaded from the homepage by downloading it to a recording medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  As another method, the program of the present invention is encrypted, stored in a recording medium such as a CD-ROM, distributed to users, and encrypted from a homepage via the Internet to users who have cleared predetermined conditions. Download the key information to be solved. It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. Furthermore, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can be realized by the processing.

  As another method, the program read from the recording medium is first written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Then, based on the instructions of the program, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are also realized by the processing.

It is a flowchart which shows a process sequence about a part of simple image processing. It is a figure which shows the example which distortion and inclination generate | occur | produce in an imaging result. It is a figure which shows the imaging result at the time of imaging in the state of FIG. It is a figure which shows the external appearance of the digital camera in the 1st Embodiment of this invention. It is a figure which shows the digital camera and image server in the 2nd Embodiment of this invention. It is a flowchart which shows the process sequence inside the digital camera in the 1st Embodiment of this invention. It is a flowchart which shows the process sequence inside the digital camera in the 1st Embodiment of this invention. It is a flowchart which shows the process sequence inside the digital camera in the 1st Embodiment of this invention. It is a figure which shows the example of the area | region obtained by the area | region division | segmentation in the 1st Embodiment of this invention. It is a figure which shows the example which moved the range currently displayed in the 1st Embodiment of this invention. FIG. 5 is a diagram illustrating a set of corrected images in the first embodiment of the present invention. It is a flowchart which shows the process sequence inside the image correction server in the 2nd Embodiment of this invention. It is a flowchart which shows the process sequence inside the image correction server in the 2nd Embodiment of this invention. It is a flowchart which shows the process sequence inside the digital camera in the 2nd Embodiment of this invention. It is a flowchart which shows the process sequence inside the digital camera in the 2nd Embodiment of this invention. It is a flowchart which shows the process sequence inside the digital camera in the 2nd Embodiment of this invention. It is a figure which shows the example of an internal structure of the digital camera in the 1st Embodiment of this invention.

Explanation of symbols

201 Digital Camera 202 Subject 203 Imaging Result 1700 Digital Camera 1701 Imaging Unit 1702 A / D Converter 1703 Signal Processing Circuit 1704 Bus 1705 Compression Circuit 1706 Memory 1707 Display Unit 1709 Bus 1710 Operation Unit 1711 MPU
1712 Storage device 1713 External communication interface 1714 Transmission path 1715 External information device

Claims (3)

An image processing method that corrects distortion and tilt of an input image and outputs the image after enlarging or reducing,
An input process for inputting an image;
An output process for outputting an image;
An instruction step for instructing scrolling of the image output in the output step;
An extraction step of extracting connected components for the image input in the input step;
A first calculation step for obtaining a set of connected components in the region corresponding to the image output in the output step; and a second calculation step for obtaining a set of connected components in the region corresponding to the scroll amount instructed in the indicating step. An image processing method comprising:   A computer program causing a computer to execute each step of the image processing method according to claim 1. An image processing apparatus that corrects distortion and inclination of an input image and outputs an enlarged or reduced image.
An input means for inputting an image;
An output means for outputting an image;
Instruction means for instructing scrolling of an image output by the output means;
Extraction means for extracting connected components for an image input by the input means;
First calculation means for obtaining a set of connected components in a region corresponding to an image output by the output means;
An image processing apparatus comprising: a second calculation unit that obtains a set of connected components in a region corresponding to the scroll amount instructed by the instruction unit.

Images