JP2011248494A - Picture synthesizing apparatus, picture synthesizing method and program for the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable, when overlapping areas are detected among a plurality of partial pictures consecutively taken of characters and/or pictures handwritten by a copyist with a camera mounted on the hand or elsewhere and a whole picture is to be reconstructed by superposing the overlapping areas, elements that would obstruct detection of the overlapping areas to be excluded without having to fit a special device or to perform a special operation and the overlapping areas to be precisely detected even from pictures with little feature amount.SOLUTION: Pictures affected by camera shake or parts of pictures accidentally showing a hand or a pen, which would obstruct detection of overlapping areas among a plurality of partial photographic pictures, are detected and removed, and furthermore corners (crossings, edges and bends) of handwritten segments are utilized as feature points.

Description

  The present invention belongs to a technique for constructing a panoramic image from a moving image, and particularly relates to a technique for recording an image for a handwritten document and a figure. Further, it belongs to a technical system called image mosaicing that obtains a panoramic image by combining a plurality of consecutive captured images.

  Recording, digitizing, and recognizing handwritten documents and figures (hereinafter collectively referred to as handwritten documents) are recognized as important research issues from the viewpoint of document information processing and recording of individual information. R & D has been conducted and commercialized.

  Already, digitizing handwritten documents using an image scanner, a tablet, a mouse, or the like is routinely performed. There is also a product that digitizes handwritten documents using a camera mounted on the pen tip and a paper on which a special pattern is printed (Non-patent Document 1).

  In addition, by recording handwritten documents on whiteboards and blackboards, which are as important as on paper, by removing the appearance of people appearing in the camera images set up to capture the whiteboards and blackboards. It has also been proposed to record handwritten documents (Non-Patent Documents 2 and 3).

  In addition, there is an example in which recording and recognition of a handwritten document is realized by giving a tablet function to the whiteboard (Non-Patent Document 4).

  On the other hand, many studies have been conducted to automatically record a wearer's daily life using a small sensor attached to the body and use it as a life log (Non-Patent Documents 5 and 6). These studies have envisioned a society where it is routine to wear one or more sensors on the body to enrich life, and in fact, these sensors should not be special in the future. is there.

  The present invention envisions such a future, and acquiring and recording information using a camera or sensor attached to the body as shown in FIG. 1 is nothing special. And

  However, when the camera is mounted on the body as shown in FIG. 1 and an attempt is made to obtain a full view of the handwritten document being written, each camera image has a problem that only a small portion of the handwritten document being written can be photographed. . Such a problem is a universal problem when only a part of the entire view can be taken with a single shot, such as making a map from an aerial photograph or taking a landscape photograph. In order to solve this problem, a technique for obtaining a panoramic image by combining partial images called image mosaicing has been studied and put into practical use. The technique of mosaicing generally pays attention to the similarity of partial images, detects similar small portions (overlapping portions), and superimposes them to obtain a panoramic image. In recent years, in order to evaluate the similarity of partial images, multiple intensity gradient-based local features such as SIFT (Scale invariant faeture trancate, Non-Patent Document 7) and SURF (Speeded up robust feature, Non-Patent Document 8) have been collected. In general, a method for obtaining a panoramic image by superimposing images based on the similarity of points at which features exist (hereinafter, feature points) is used.

Anoto, "Anoto", [online], [Search April 26, 2010], Internet <URL: http://www.anoto.com/> L. He and Z. Zhang, "Real-Time Whiteboard Capture and Processing Using a Video Camera for Remote Collaboration", IEEE Transactions on Multimedia, 2007, Vol.9, No.1, p.198-206 P.E.Dickson, W.R.Adrion, and A.R.Hanson, "Whiteboard Content Extraction and Analysis for the Classroom Environment", IEEE International Symposium on Multimedia, 2008, p.702-707 Marcus Liwicki and Horst Bunke, "Recognition of Whiteboard Notes-Online, Offline and Combination", World Scientific Publishing, 2008, p.1-11 L. Bao and S.S. Intille, "Activity Recognition from User-Annotated Acceleration Data", Proceedings of PERVASIVE 2004, 2004, p.1-17 T. Maekawa, Y. Yanagisawa, Y. Kishino, K. Kamei, Y. Sakurai, and T. Okadome, "Wristband Type Sensor Device for Recognizing Activities that Involve Object Use", Proceedings of European Conference on Ambient Intelligence, 2009, Poster paper M. Brown and D.G.Lowe, "Recognizing Panoramas", International Conference on Computer Vision-Volume 2, 2003, p.1218-1225 H. Bay, T. Tuytelaars, and L. Van Gool, "SURF: Speeded Up Robust Features", European Conference on Computer Vision, 2006, p.404-417

  In the techniques of Non-Patent Documents 1 to 4, a specially prepared device such as a pen, paper, tablet, or scanner is required, which is expensive. The user must also learn how to operate these special devices.

  According to techniques such as Non-Patent Documents 5 and 6, a handwritten document can be automatically recorded using a camera and a sensor attached to the wrist. However, a camera that can be worn on the wrist has a small viewing angle and cannot capture all the handwritten documents being drawn. Therefore, it is necessary to obtain a panoramic image by integrating partial images of a wide range of images that are covered as the camera moves. In addition, when the camera is mounted on a wrist or the like, a hand or pen to be drawn is reflected between the paper surface and the camera. Furthermore, the writing operation is not always stable, and it may operate at a high speed that cannot be tracked with the video performance of a camera that can be worn on the wrist, etc. In such a case, an appropriate partial image should be taken. Is difficult. In addition, when considering drawing simple hand-drawn figures on white paper, it is difficult to obtain a natural gradient compared to the case of complex images such as natural images. Therefore, it is difficult to obtain stable feature points in feature extraction systems such as Non-Patent Document 7 (SIFT) and Non-Patent Document 8 (SURF) which are currently mainstream, and an appropriate composite image cannot be obtained. was there. In particular, it is difficult to find similar feature points from stroke images taken from distant positions.

  An object of the present invention is to detect an overlapping area of a plurality of partial images obtained by continuously capturing characters or pictures handwritten by a writer using a camera attached to a hand or the like. When reconstructing the entire image by superimposing, elements that hinder the detection of overlapping areas can be eliminated without wearing a special device or performing special operations, and even images with small feature quantities are duplicated An object of the present invention is to provide an image composition device, an image composition method, and a program thereof that can accurately detect a region.

  According to a first aspect of the present invention, there is provided an image synthesizing device for synthesizing a large connected image from a plurality of images continuously captured in accordance with the movement of a part from a camera attached to a predetermined part on the side where the wearer writes. An image composition apparatus comprising signal input means for detecting the intensity and position of movement of the predetermined part.

  A second aspect of the present invention is the image composition apparatus according to the first aspect of the present invention, wherein the signal input means is an acceleration sensor.

  In a third aspect of the present invention, in the image composition device according to the first or second aspect of the present invention, when the background of the captured image is a paper surface or a whiteboard, the color of the paper surface or the whiteboard is detected, An image synthesizing apparatus comprising foreground removal means for removing pixels having different colors as a reflected image.

  According to a fourth aspect of the present invention, in the image composition device according to the third aspect of the present invention, in addition to a luminance gradient-based local feature, as a feature point serving as a clue to synthesize an image, And a feature extraction unit that extracts an intersection, a bending point, and an end point, and an image synthesis unit that synthesizes an image by detecting an overlapping area between two images using the feature point. An image composition device.

  According to a fifth aspect of the present invention, there is provided an image input step of continuously capturing images in accordance with movement of a part from a camera attached to a predetermined part on the side where the wearer writes, and movement of the predetermined part. A signal input step for detecting intensity and position, an image selection step for selecting an image of a desired image quality from a plurality of images captured in the image input step based on detection information in the signal input step, and an image selection step A foreground removal step for detecting the color of the paper surface or whiteboard when the background of each image selected in is a paper surface or a whiteboard, and removing pixels of a different color as a reflected image; A feature extraction step for extracting feature points for each image obtained in the removal step, and a region overlapping between the two images is detected using the feature points, and the regions overlap. An image combining step of combining the respective images to Migihitsuji, an image synthesizing method of execution.

  According to a sixth aspect of the present invention, in the image composition method according to the fifth aspect of the present invention, the feature points extracted in the feature extraction step are intersections of lines, which are characteristic of black and white binary graphics in addition to luminance gradient-based local features. And an inflection point and an end point.

  A seventh aspect of the present invention is a program for causing a computer to function as the image composition apparatus according to any one of the first to fourth aspects of the present invention.

  According to the image synthesizing apparatus, the image synthesizing method, and the program thereof according to the present invention, an image with a camera shake and a hand or a pen in the image that interfere with the detection of an overlapping area of a plurality of captured partial images are captured. By detecting and removing parts, it is possible to eliminate elements that hinder the detection of overlapping areas, and to use feature points by using corners (intersections, end points, bending points) of handwritten line segments as feature points Even in an image with few images, the overlapping area can be detected with high accuracy. Therefore, the entire image can be reconstructed with high accuracy without mounting a special device or performing a special operation.

The figure which shows the image which mounted | wore the body with the camera and the sensor. 1 is a block diagram illustrating a configuration example of an image configuration apparatus 100. FIG. FIG. 3 is a diagram showing an example of a processing flow of the image construction apparatus 100. The figure which shows the image which images a handwritten stroke continuously. The figure which shows the image which superimposes the overlapping area of a partial image, and reconfigure | reconstructs the whole image. The figure which shows the example of the corners (an intersection, an end point, a bending point, etc.) extracted as a feature from an image.

  FIG. 2 is a block diagram showing a configuration example of the image construction apparatus 100 of the present invention, and FIG. 3 is a processing flow example thereof. The image construction apparatus 100 includes an image input unit 21, a signal input unit 22, an image selection unit 23, a foreground removal unit 24, a feature extraction unit 25, an image composition unit 26, and an image output unit 27.

  The image input unit 21 is a camera that is attached to a predetermined natural part such as a writer's hand, wrist, or lower arm, and the stroke that the writer draws on the paper or the like by hand as shown in FIG. Images are continuously captured (S1). For example, if it is attached to the wrist, it is possible to photograph the paper surface regardless of the position of the paper surface, and it is possible to capture fine patterns and characters because it is photographed from near the paper surface. Note that the amount of data can be reduced by converting captured image data into a histogram or the like. The signal input unit 22 is an acceleration sensor, for example, and is mounted at substantially the same position as the image input unit 21 to detect the intensity and position of the movement of the part where the camera is mounted (S2). Instead of the acceleration sensor, another device that can detect the intensity of operation, such as a position sensor, a three-dimensional mouse, a device including an infrared light receiving unit, or a Doppler radar, may be used. The amount of data can be reduced by applying fast Fourier transform to waveform data such as acceleration data and extracting only the Fourier coefficients.

  While conventional input devices such as tablets and scanners are special devices for electronically recording written figures, such cameras and sensors can be naturally used for other purposes as well. Wearing one or more cameras or sensors on the body is considered not to be special for the recording of daily life as well as writing operations in the future. High nature.

  The image selection unit 23 determines whether or not the image input from the image input unit 21 is an image suitable for synthesis that does not include blur and the like, and outputs the image determined to be a suitable image (S3). . Specifically, first, a blurred image is extracted based on the operation information detected by the signal input unit 22 and discarded. When an acceleration sensor is used as the signal input unit 22, the time during which the hand is moving is detected from the amount of change from the moving average of the acceleration data, and an image taken at a time when the amount of change is larger than the arbitrarily set threshold value. Discard as blurred image. The moving average can be obtained from the acceleration data series. Extraction of a blurred image using an acceleration sensor is a technique already used in a digital camera or the like (for example, Reference 1). As described above, after a high-quality image is roughly selected based on the operation information of the camera mounting unit, a blurred image is further detected and discarded by image processing. There are many existing methods for blur detection by image processing (for example, Reference 2). The image selection unit 23 outputs the image selected as described above. In the present invention, the feature extraction unit 25 extracts a plurality of feature points from each image and combines them. However, an image with few feature points may be an image that does not capture a blurred image or a stroke. Since there are many such images, the image selection unit 23 may discard such images in advance.

[Reference 1] US Pat. No. 4,673,276 [Reference 2] H. Tong, M. Li, H. Zhang and C. Zhang, "Blur Detection for Digital Images Using Wavelet Transform", Proceedings of IEEE International Conference on Multimedia and Expo, 2004, p.17-20

  The foreground removal unit 24 removes an image such as a hand or a pen reflected in front of each image selected by the image selection unit 23 (S4). Specifically, this can be realized by discarding a foreground image (region) having a color that is significantly different from the background paper color or a foreground image (region) having a color close to the skin color. More specifically, each image is divided into grid-like small areas, and it is determined for each small area whether or not it is a foreground image from the degree of similarity to the color. The similarity is, for example, the Euclidean distance in the HSV color space, and is determined by setting an arbitrary threshold value. The paper color can be detected, for example, by searching for the most frequently used color in the first image in the continuous image series acquired by the image input unit 21. The skin color is set in advance.

  The feature extraction unit 25 extracts feature points for taking corresponding points of each image for each image from which the foreground has been removed by the foreground removal unit 24 (S5). The features referred to here are not only features such as SURF and SIFT, which are luminance gradient-based local features, but also features that are often found in black and white binary graphics such as handwritten documents, such as corners (intersections, end points, Inflection points). For the extraction of intersections, end points, inflection points, and the like of line segments, extraction processing may be performed on small areas other than the small areas determined by the foreground extraction unit 24 to be foreground images.

  The image compositing unit 26 finds overlapping regions of the images by calculating feature points that match between the images extracted by the feature extracting unit 25, and synthesizes the images so that they overlap. (S6). Specifically, projection conversion between images is obtained. In other words, for images 1 and 2 having overlapping regions as shown in FIG. 5, transformation is applied to image 2 to obtain a transformation parameter (homography matrix) that overlaps with the overlapping region with image 1. By the conversion, a reconstructed (synthesized) image shown in the right part of FIG. 5 is created. More specific description will be given below.

  One image is considered as a node of a tree and a projection transformation between two images as an edge. In order to create a reconstructed image using all the obtained images, it is necessary to create a spanning tree from these images. If a spanning tree can be created, projection transformation between arbitrary images can be obtained by sequentially applying projection transformation (edges) between images (nodes). Therefore, in order to create a spanning tree, the image composition unit 26 first compares images using feature points that can be processed at high speed. And about the part which could not create a spanning tree only by the comparison using a feature point, an overlapping area | region is discovered using the corner extracted from the image. As described above, in comparison using feature points, it is difficult to find similar feature points from stroke images photographed from distant positions, and therefore, it is difficult to find overlapping regions between images. However, by using the corner extracted from the image, it is possible to find an overlapping region even between stroke images taken from a distant position. In the following, a comparison method using feature points and an overlapping region discovery method using corners will be described.

<Comparison method using feature points>
The comparison is performed using a high-speed feature extraction method such as SURF. In addition, it is desirable that the feature extraction / description method is invariable to some extent with respect to rotation and brightness change such as SURF. Then, a projection transformation is obtained from a pair of similar feature points obtained from each image. Since the obtained feature point pairs contain many errors, robust estimation is used to derive the projection transformation (see References 3 and 4).

[Reference 3] MA Fischler and RC Bolles, "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography", Communications of the ACM, 1981, Vol.24, p.381-395
[Reference 4] R. Hartley and A. Zisserman, "Multiple view geometry in computer vision", Cambridge University Press, 2003

<A method for finding overlapping areas using corners>
Taking the two images (a) and (b) shown in FIG. 6 as an example, the portions indicated by ◯ are corners. Find correspondences between corners between images and calculate transformation parameters (homography matrix). For example, four corresponding corner pairs (p1, p1 '), (p2, p2'), (p3, p3 '), (p4, p4') between the image of (a) and the image of (b) ), The homography matrix can be calculated by solving the simultaneous equations (see References 5 and 6).

[Reference 5] J. Shi and C. Tomasi, "Good features to track", IEEE Conference on Computer Vision and Pattern Recognition, 1994, p.593-600
[Reference 6] CE Springer, "Geometry and analysis of projective spaces", Freeman, 1964

  Then, the image synthesized by the image synthesis unit 26 is output from the image output unit 27 (S7).

  As described above, according to the present invention, the overlapping area is detected for a plurality of partial images obtained by continuously capturing characters or pictures written by the writer by hand using a camera attached to the hand, When reconstructing the whole image by superimposing the overlapping areas, it is a part of the image with hand shake and the hand or pen in the image that hinders detection of the overlapping area of the captured partial images By removing and detecting elements, it is possible to eliminate elements that hinder the detection of overlapping areas, and feature quantities can be obtained by using corners (intersections, end points, bending points) of handwritten line segments as feature points. Even with a small number of images, an overlapping region can be detected with high accuracy. Therefore, the entire image can be reconstructed with high accuracy without wearing a special device or performing a special operation.

  Each of the above processes is not only executed in time series according to the description, but may also be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. In addition, it can change suitably in the range which does not deviate from the meaning of this invention.

  When each of the above devices is realized by a computer, the processing contents of the functions that each device should have are described by a program. The program is stored in, for example, a hard disk device, and necessary programs and data are read into a RAM (Random Access Memory) at the time of execution. The read program is executed by the CPU. In this way, each processing content is realized on the computer. Note that at least a part of the processing content may be realized by hardware.

Claims (7)

An image synthesizer that synthesizes a large connected image from a plurality of images continuously captured according to the movement of the part from a camera attached to a predetermined part on the side where the wearer writes,
An image synthesizing apparatus comprising signal input means for detecting the intensity and position of movement of the predetermined part. The image composition apparatus according to claim 1,
The image synthesizing apparatus, wherein the signal input means is an acceleration sensor. In the image composition device according to claim 1 or 2,
When the background of the imaged image is a paper surface or a whiteboard, foreground removal means is provided that detects the color of the paper surface or the whiteboard and regards pixels of a different color as a reflected image and removes them. An image composition device. The image composition device according to claim 3.
Feature extraction means for extracting line intersections, inflection points, and end points characteristic of black-and-white binary graphics, in addition to luminance gradient-based local features, as feature points that serve as clues for synthesizing images;
Image compositing means for compositing images by detecting an overlapping area between two images using the feature points;
An image composition apparatus comprising: An image input step for continuously capturing images according to the movement of the part from a camera attached to a predetermined part on the side where the wearer writes;
A signal input step for detecting the intensity and position of the movement of the predetermined part;
An image selection step for selecting an image of a desired image quality from a plurality of images captured in the image input step based on the detection information in the signal input step;
Foreground removal step of detecting the color of the paper surface or whiteboard when the background of each image selected in the image selection step is a paper surface or whiteboard, and considering pixels of a different color as a reflected image and removing them When,
A feature extraction step for extracting feature points for each image obtained in the foreground removal step;
An image combining step of detecting an overlapping area between two images using the feature points, and combining the images so that the areas overlap;
The image composition method to execute. The image composition method according to claim 5, wherein
The image synthesizing method characterized in that the feature points extracted in the feature extraction step include, in addition to the luminance gradient-based local features, line intersections, bend points, and end points characteristic of black and white binary graphics.   A program for causing a computer to function as the image composition device according to claim 1.