JP2013141053A - Image processing device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to easily find a desired image from among photographed images obtained by photographing the same object.SOLUTION: When a photographed image is input, processing for clipping an area including a subject and perspective correction processing are performed to generate a corrected image. Next, the corrected image generated this time is compared with a corrected image generated in the past to determine a group (image group) of similar images, and further, definition representing how much a character or a figure in the image is discriminable is calculated. Then, an image file which has the photographed image before correction, and has the corrected image and the obtained information as meta information is generated, and a corrected image with the highest definition is determined as a representative image.

Description

  The present invention particularly relates to an image processing apparatus, an image processing method, and a program thereof suitable for use in searching for a desired image.

  Conventionally, when a meeting or a meeting is held in a company, a school, a research institution or the like, it is widely performed to summarize the contents of the meeting or the meeting using a whiteboard. At this time, it is often performed to take a whiteboard using a digital camera or the like and leave it as a conference record. However, when a person stands in front of a whiteboard or when an object is placed, it is often impossible to shoot from the front and to shoot from an angle. The images taken in this way are often not suitable for storage or display because the content written on the whiteboard is distorted.

  Therefore, in order to solve such a problem, it has been performed to correct a captured image to an image as if it was captured from the front by perspective correction of distortion of the photographed board document content. For example, Patent Document 1 discloses a technique for storing an image before parse correction and an image after parse correction.

JP 2008-99147 A

Camera & Imaging Products Association CIPA DC-007 Multi-Picture Format

  However, the contents written on the whiteboard due to flash or lighting are often not clear. For example, as shown in FIG. 12, an image 1201 in which illumination is reflected and a blurred image 1202 due to out-of-focus or camera shake may be obtained by the perspective correction. When such an image with unclear blackboard content is obtained, it is generally performed to shoot the whiteboard a plurality of times while changing the shooting conditions.

  On the other hand, in the technique described in Patent Document 1, an image before parse correction and an image after parse correction are stored in separate files. For this reason, even if the file names are related, when the whiteboard is shot a plurality of times while changing the shooting conditions, the files relating to the contents of the blackboard increase. Also, in meeting materials, image correction may be manually adjusted in order to make it easier to see the contents of the board, so in many cases not only the corrected image but also the image before correction is required. Therefore, it is troublesome to find a desired file from a plurality of files.

  An object of the present invention is to make it possible to easily find a desired image from photographed images obtained by photographing the same object.

  The image processing apparatus according to the present invention includes a correction image generation unit that generates a correction image obtained by extracting and parsing the subject from a plurality of captured images of the same subject, and a correction image generated by the correction image generation unit. Calculating means for calculating the sharpness of the image, and determining means for determining, as a representative image, the corrected image having the highest sharpness among the corrected images whose sharpness is calculated by the calculating means. .

  According to the present invention, it is possible to easily find the clearest image from among captured images obtained by photographing the same object such as a whiteboard.

It is a figure which shows the structural example of the image processing system in embodiment. It is a figure explaining a mode that the whiteboard is image | photographed diagonally from two imaging devices. It is a figure which shows an example of a picked-up image and the image after correction | amendment. It is a block diagram which shows the internal structural example of the image processing apparatus which concerns on embodiment. 6 is a flowchart illustrating an example of an operation procedure of the image processing apparatus according to the embodiment. It is a figure which shows an example of a some picked-up image. It is a figure which shows an example of the image after the parse correction of the image shown in FIG. It is a figure which shows an example of the table which put together the image group with respect to a correction image, and the value of a sharpness. FIG. 7 is a diagram illustrating an example of an image file in which meta information is added to the captured image illustrated in FIG. 6. In an embodiment, it is a figure showing an example of UI displayed on a display. In an embodiment, it is a figure showing other examples of UI displayed on a display. It is a figure which shows an example of the image which illumination reflected and reflected, and the blurred image.

  Hereinafter, the present invention will be described in detail based on preferred embodiments with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations. In the following embodiments, a whiteboard is assumed as a subject to be photographed. However, the application destination of the present invention is not limited to a whiteboard, and a planar photographing object such as a paper document or a business card. Applicable in general.

(First embodiment)
FIG. 1 is a diagram illustrating a configuration example of an image processing system 7 in the present embodiment.
As shown in FIG. 1, the image processing system 7 according to the present embodiment includes an imaging device 2, an image processing device 3, and a display device 4. The imaging device 2 and the image processing device 3 are connected to each other through a wireless communication path 5 such as a wireless LAN, but need not be wireless and may be wired. In addition, the image processing apparatus 3 and the display apparatus 4 are connected to each other via a wired communication path 6 such as HDMI, but are not necessarily wired and may be wireless.

  The photographing apparatus 2 is provided with a photographing button 21 and a viewfinder 22 such as a liquid crystal panel. In the example shown in FIG. 1, the whiteboard 1 is arranged as an object to be photographed by the photographing apparatus 2, but it is not particularly limited to the whiteboard, and is flat like a business card or paper material placed on a desk. It may be a thing. The object to be photographed is displayed on the viewfinder 22, and a photographed image is generated by pressing the photographing button 21, and the data is input to the image processing device 3 through the wireless communication path 5. Then, the image signal and GUI output from the image processing device 3 are sent to the display device 4 through the wired communication path 6, and the captured image is displayed.

  Here, as shown in FIG. 2, when the whiteboard 1 is photographed obliquely from the two photographing devices 2a and 2b, distorted images such as images 31a and 31b shown in FIG. 3 are obtained, respectively. In the image processing apparatus 3 according to the present embodiment, these distorted images are input, distortion correction is performed by image processing such as projective transformation, and an image 34 as if taken from the front is generated. At this time, the projective transformation parameters are calculated by detecting feature points of the input image.

FIG. 4 is a block diagram illustrating an internal configuration example of the image processing apparatus 3 according to the present embodiment.
In FIG. 4, a UI device 401 is a mouse, a digitizer, a keyboard, or the like, and is used for inputting a user instruction to the image processing apparatus 3. The CPU 402 develops a program from the program storage area 406 to the RAM 403, and interprets and executes the program by the CPU 402, thereby controlling various controls and calculations in the image processing apparatus 3, UI display, and the like.

  The communication IF 404 is a communication interface with the imaging device 2 and the display device 4. The UI display unit 405 is an LED, a liquid crystal panel, or the like that displays the state of the image processing device 3 and the processing content, but may be mounted so as to control and display the display device 4. Further, the image processing apparatus 3 according to the present embodiment includes a program storage area 406 and a data storage area 407. Specifically, these storage areas can be realized using a hard disk or a flash memory, but it goes without saying that the present invention does not depend on a specific storage medium.

  Since the image processing system 7 of the present embodiment is often used at the time of a conference, in actual operation, the image processing system 7 is integrated with a conference system that performs minutes creation, conference information management, and the like on the image processing device 3. It may be a configuration.

Next, the operation procedure of the image processing apparatus 3 in the present embodiment will be described with reference to the flowchart of FIG. Each process shown in FIG. 5 is performed under the control of the CPU 402.
First, in step S <b> 501, the process waits until a photographed image is input from the photographing apparatus 2 via the communication IF 404. When a captured image is input, region extraction processing and perspective correction processing are performed in S502. The process of S502 can be realized by using, for example, a distortion correction process described in Patent Document 1.

  Here, as an example of the captured image, when the captured images 601 to 606 illustrated in FIG. 6 are input in S501, corrected images 701 to 703, 705, and 706 illustrated in FIG. 7 are generated by the processing of S502. Here, since a clear rectangular area does not exist in the captured image 604 shown in FIG. 6, it is assumed that a corrected image is not generated.

Next, in S503, the corrected image generated this time is compared with the corrected image generated in the past, and a group (image group) of close images is determined. Thereby, it is possible to combine a plurality of images obtained by photographing the same target region under different photographing conditions into one image group. As an image comparison method, for example, the comparison is performed by calculating a difference value. When the luminance component of the pixel of the current corrected image is I (x, y) and the luminance component of the pixel of the past corrected image is I _P (x, y), the difference Δ between the images in a certain region K is expressed by the following equation: Calculated by (1).

  In the example illustrated in FIG. 7, the corrected images 701 to 703 are determined as images close to each other, and are classified as, for example, the image group A. Similarly, the corrected images 705 and 706 are also determined to be close images, for example, the image group X. In the present embodiment, the images are compared by calculating the difference value using Equation (1), but other methods may be used as long as the images can be compared.

  Next, in S504, a sharpness level representing how many characters and figures in the image can be distinguished is calculated. As a method for calculating the sharpness, it is possible to use the recognition accuracy obtained as a result of character recognition or to use the degree of the high-frequency component of the image. However, if the sharpness can be calculated, these methods are used. It is not limited.

  In the example illustrated in FIG. 7, the sharpnesses of the corrected images 701 to 703, 705, and 706 are calculated as 2, 1, 3, 1, 3, for example. In this case, the higher the numerical value of the sharpness, the clearer the image. FIG. 8 shows a table in which image groups and sharpness values for the corrected images 701 to 703, 705, and 706 are summarized.

  Next, in S505, the information obtained in S502 to S504 is associated with each captured image as meta information. Such meta information can be stored in, for example, a JPEG application marker segment.

FIG. 9 is a diagram illustrating an example of an image file in which meta information is added to the photographed image illustrated in FIG. 6.
As shown in FIG. 9, the image file 901 includes a captured image 601 and its meta information 911. Further, the meta information 911 includes the corrected image 701 obtained in S502 of FIG. 5, the image group information obtained in S503, and the sharpness obtained in S504. If the corrected image is too large to fit in the thumbnail area of the meta information, it may be resized and reduced or stored in a multi-picture format as described in Non-Patent Document 1.

  Similar to the image file 901, the other image files 902, 903, 905, and 906 are composed of captured images and meta information. Unlike the other meta information 912, 913, 915, and 916, the meta information 914 of the image file 904 does not include image group or sharpness information, and the captured image 604 is resized in the thumbnail area. Thus, a reduced image 924 is stored.

  Then, the image file to which the meta information is added by the processing of FIG. 5 is recorded in the data storage area 407 under the control of the CPU 402. When the processing in FIG. 5 is completed, the corrected image with the highest definition is determined as a representative image from these image files, and the representative image is output to the display device 4 via the communication IF 404 to perform display control. Do.

FIG. 10 is a diagram illustrating an example of a UI displayed on the display device 4 when the image processing device 3 outputs the image file thus created to the display device 4 and displays the list.
As shown in FIG. 10, when image group information is included in the meta information of each image file, an image having the highest definition in the same image group is displayed as a representative image. In the example shown in FIG. 10, the corrected image is displayed as a thumbnail image in the image frame 1001, but the captured image itself may be displayed. Further, the image 924 that does not have image group information may be hidden and displayed.

  Also, buttons that display other than the representative image, such as buttons 1002 and 1003 shown in FIG. 10, may be displayed as a UI. For example, when the button 1003 is pressed, a corrected image 705 belonging to the same image group as the image 706 is also displayed as shown in FIG. 11, for example, and the button 1003 is changed to a button 1101 for returning only the representative image to the display. Try to switch.

  As described above, according to the present embodiment, since the image with the highest definition in the same image group is determined as the representative image, an image captured by reflecting light or the like or a blurred image is included. In this case, an image used as a meeting material can be easily found.

  In the present embodiment, the example in which the processing illustrated in FIG. 5 is performed by the image processing device 3 has been described. However, the processing may be performed inside the imaging device 2. In this case, the thumbnail image list may be displayed on the viewfinder 22 of the photographing apparatus 2 instead of being displayed on the display apparatus 4. At this time, it is desirable to configure the viewfinder 22 as a touch panel so that pointing with a finger is possible.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

401 UI device 402 CPU
403 RAM
404 Communication IF
405 UI display section 406 Program storage area 407 Data storage area

Claims (6)

Generating means for generating a corrected image obtained by cutting out the subject from each of a plurality of captured images of the same subject and correcting the perspective;
Calculating means for calculating the sharpness of the corrected image generated by the generating means;
An image processing apparatus comprising: a determining unit that determines, as a representative image, a corrected image having the highest sharpness among the corrected images having the sharpness calculated by the calculating unit.   The image processing apparatus according to claim 1, further comprising a recording unit that records the captured image and the corrected image related to the captured image in a single file in association with each other.   The image processing apparatus according to claim 2, wherein the recording unit records, as meta information of the captured image, a corrected image obtained by cutting out the subject and performing perspective correction.   The image processing apparatus according to claim 1, further comprising display control means for displaying the representative image determined by the determination means on a display device. A generation step of generating a corrected image obtained by cutting out the subject from a plurality of captured images of the same subject and performing perspective correction;
A calculation step of calculating the sharpness of the corrected image generated in the generation step;
An image processing method comprising: a determining step of determining, as a representative image, a corrected image having the highest sharpness among the corrected images whose sharpness has been calculated in the calculating step. A generation step of generating a corrected image obtained by cutting out the subject from a plurality of captured images of the same subject and performing perspective correction;
A calculation step of calculating the sharpness of the corrected image generated in the generation step;
A program that causes a computer to execute a determination step of determining, as a representative image, a corrected image having the highest sharpness among the corrected images for which the sharpness has been calculated in the calculation step.

Images