JP2012060452A - Image processor, method therefor and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide electronic data for which a handwritten object added to projected electronic data is combined in such a form that a positional relation is close to an original description shape.SOLUTION: A frame extraction processing unit 401 extracts a frame comprising four straight lines from an object image 400. A projection area frame shape 402 is equivalent to the contour of a projection area extracted from the frame extraction processing unit 401. An electronic data specifying unit 403 specifies projected electronic data 404 from the object image 400, an object extraction processing unit 405 extracts a handwritten object 406 from the object image 400 and the electronic data 404, and an object arrangement processing unit 407 arranges the handwritten object 406 in the electronic data 404. In this embodiment, the handwritten object 406 is arranged in the electronic data 404 to attain output data 408. In this case, the output data 408 are in the same structure as the electronic data 404 but are different in the point that the handwritten object 406 is included.

Description

  The present invention relates to an image processing apparatus, a method thereof, and a program, and more particularly to an image processing apparatus for storing a document image processing result, a method thereof, and a program thereof.

  In recent years, conference materials have been distributed electronically by the development of computers and networks, and conference materials distributed electronically by a projector such as a projector on a projection board that can be handwritten such as a whiteboard during a conference. Is often projected as it is. In this type of meeting, participants add letters and diagrams to the whiteboard onto which electronic data is projected and proceed with the discussion. The handwritten whiteboard is photographed with a photographing device such as a digital camera. The shooting data may be used as minutes.

  Since the shooting data is distorted depending on the shooting angle and it is unclear which part of the original meeting material is associated, it is inconvenient to substitute the minutes. Therefore, a technique has been proposed in which such handwritten image data is converted into data that can be distributed electronically as minutes.

  For example, a method has been proposed in which when writing is performed on a projected image plane, projection is temporarily stopped and a handwritten image that has been entered is captured and combined to generate image data (see, for example, Patent Document 1). In addition, since the photographing data cannot normally be photographed from the front of the projection plate due to the positional relationship between the photographing device and the whiteboard, the projection data is distorted. Therefore, when the composition is simply combined, the original electronic data and the handwritten image are displaced. . In order to remove this deviation, a method has been proposed in which distortion is acquired by an outer frame acquisition unit of a projected image to correct the distortion of the image (see, for example, Patent Document 2).

JP 2006-108813 A JP 2007-017543 A

  However, in a system for projecting conference materials using a conventional projection apparatus, the handwritten location is often not only inside the projection area projected onto the projection plate but outside the projection area. For example, when projecting onto a whiteboard, the conference material is projected onto a certain area, but writing characters and pictures by hand in areas other than that area is usually performed in a meeting. Japanese Patent Application Laid-Open No. 2004-151561 describes the composition of handwritten content (or handwritten object) inside the projection area, but the processing when there is a handwritten object outside the projection area is unknown. In an actual meeting, there is a high possibility of handwriting outside the projection area, so the location of the handwritten object is not limited to the inside or outside of the projection area. Is desirable.

  In addition, when a conference is actually performed using a non-fixed portable projector, the projector is often operated in a state where the projection plate and the projector are not correctly facing each other. In an image shot in such a situation, there is distortion due to the positional relationship between the projection device and the projection plate in addition to distortion due to the positional relationship between the projection plate and the imaging device, and this distortion is generally different. It is. Therefore, there is a problem that when the image is corrected based on the outer frame of the projection area, the shape of the handwritten part is distorted. On the other hand, when the frame of the projection plate is used for correction in order to eliminate the distortion of the handwritten part, there is a problem that the position is shifted when the corrected image is combined with the projection image data. Since only one of the methods of Patent Document 2 can be corrected, there is a problem that the other distortion remains.

  In order to solve such a problem, the image processing apparatus of the present application projects a projection unit that projects electronic data onto a projection plate capable of handwriting, and a projection obtained by projecting electronic data on a handwritten object that is an image handwritten on the projection plate. A photographing means for photographing together with an image to generate image data, an electronic data identifying means for extracting a projection image photographed together with a handwritten image from the image data, and identifying electronic data of the extracted projection image; A frame extraction unit that extracts a frame of the projection area projected on the projection plate by the projection unit from the image data, a handwritten object is extracted from the image data, and the extracted handwritten object is converted into an object inside the projection area, and the projection area. Means for separating and holding objects outside the object, objects inside the extracted projection area, and outside the projection area And object, characterized by comprising an output means for outputting with the electronic data specified by the electronic data identification means.

  It is possible to obtain electronic data obtained by synthesizing a handwritten object added to the projected electronic data in a form having a positional relationship close to the original description shape.

It is a figure which shows the environment where the image input into the image processing apparatus of one Embodiment of this invention is acquired. It is a figure which shows the example of the input image input into an image processing apparatus. It is a figure which shows the structural example of an image processing apparatus. It is a figure which shows the block diagram explaining the process of one Embodiment of this invention. Block Diagram FIG. 4 is a diagram showing a flowchart of FIG. FIG. 3 is a diagram showing an example of an image obtained by performing edge enhancement on the input image 200 of FIG. It is a figure which shows the example of the image which cut out the white board area | region from the input image, and correct | amended distortion. FIG. 6 is a diagram showing a display example of electronic data in which the first embodiment is applied to a display / edit program. It is a figure which shows the example of a slide preview display of a display / edit program. It is a figure which shows the example of electronic data display which applied Embodiment 2 to the display and edit program. FIG. 10 is a diagram showing an example of electronic data display in which the third embodiment is applied to a display / edit program. It is a figure which shows the example of an electronic data display which applied Embodiment 4 to the display and edit program. It is a figure which shows the example of an electronic data display which applied Embodiment 5 to the display and edit program. It is a figure which shows the example of electronic data display which applied Embodiment 6 to the display and edit program.

[Example 1]
Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram illustrating an environment in which an image input to an image processing apparatus according to an embodiment of the present invention is acquired.

  Referring to FIG. 1, a white plate 101 serves as both a projection plate and an entry plate, and an entry is made on the white plate 101 with a pen 102. The projection device 103 projects an image on the white plate 101, and the personal computer 104 transmits the image to be projected to the projection device 103. A photographing device 105 such as a digital camera photographs the white plate 101 including the projection area, and the projection data 110 is projected onto the white plate 101 from the projection device 103.

  FIG. 2 is a diagram illustrating an example 200 of an image transmitted to the image processing apparatus of the present embodiment. An image 200 photographed by the photographing apparatus 105 includes a white board region frame 201 and a projection region frame 202.

  Similarly, the input image 200 includes a projected image object 203 (“Slide Title”) of the projected image, and handwritten objects 204 (“Handwriting”) and 205 (“memo”) written by the pen 102. Further, an inclination, that is, linear distortion occurs due to the relationship between the position of the photographing device 105 and the position of the object, and the projection area frame 202 and the projection image object 203 are related to the white plate 101 from the positional relationship between the projection device 103 and the white plate 101. Distortion different from the distortion of is caused. Here, the handwritten object 204 exists inside the projection area frame 202, and the handwritten object 205 exists outside the projection area frame 202.

  FIG. 3 is a diagram showing a configuration example of an image processing apparatus 300 for implementing the present invention. The image processing apparatus 300 includes an external storage medium I / F 301 for inputting captured image data and a CPU 302 for executing an image processing program for performing the processing of the present invention on the image data. Further, a work memory when executing such an image processing program, a memory 303 used for temporary storage of data, and a hard disk 304 for storing the program and data are provided. The configuration of the image processing apparatus 300 described above is an example, and image processing may be executed using the personal computer 104, or image processing may be executed on an electronic circuit such as the photographing apparatus 105.

  FIG. 4 is a block diagram illustrating the processing of the present invention. The target image 400 is input through the external storage medium I / F 301 as an image captured by the image capturing apparatus 105 and stored in an external storage medium. The frame extraction processing unit 401 extracts a frame composed of four straight lines from the target image 400. The projection area frame shape 402 corresponds to the outline of the projection area extracted by the frame extraction processing unit 401. The electronic data specifying unit 403 specifies electronic data 404 projected from the target image 400. The object extraction processing unit 405 extracts the handwritten object 406 from the target image 400 and the electronic data 404.

  The object placement processing unit 407 places the handwritten object 406 in association with the inside of the electronic data 404.

  In this embodiment, the handwritten object 406 is arranged in the electronic data 404 and used as output data 408. Here, the output data 408 has the same structure as the electronic data 404, but differs in that a handwritten object 406 is included.

  The output data 408 is composed of data 409 to 413 and can be displayed / edited by a display / editing program in the PC 104. The data 409 is used for layout editing in the display / editing program, and includes a projection display area 410 and a non-projection display area 411. The display / edit program uses the data 412 for text editing, and displays only the data on the projection display area on the display of the PC 104 in the projection display mode. In the editing mode, the projection display area 410, the non-projection display area 411, and the text editing data 412 of the layout data are displayed. For example, an application such as Microsoft Office PowerPoint (registered trademark) can be used as the display / editing program.

  FIG. 5 is a flowchart showing details of the processing of this embodiment based on the block diagram of FIG. Hereinafter, an example of generating the output data 408 using the target image 200 of FIG. 2 as the target image 400 will be described with reference to the flowchart of FIG.

  In step S501, the electronic data specifying unit 403 specifies the electronic data projected by the projection device 103 when the shooting device 105 takes an image, and the process proceeds to step S502. The electronic data may be specified by searching by a search technique such as a full text search using a target file or character string which is the electronic data projected in the image stored in the PC 104. Specifically, the key information for the search is acquired from an image obtained by correcting the projection area frame 202 of the captured image with projection distortion, or a character string obtained as a result of OCR processing of the image. Can do. Alternatively, the user may be prompted to input the file name of the target file, and specified using the input result. Here, the width w and the height h of the projection area frame 202 of the electronic data are specified together with the specification of the electronic data itself.

  In step S502, the frame extraction processing unit 401 extracts unequal side rectangles corresponding to the projection area frame 202 and the white plate area frame 201 from the captured image. Extraction of a rectangle corresponding to a frame can be performed as follows using a known method. That is, pixels corresponding to the boundary portion of the white plate region in the target image are made to stand out by an edge enhancement method using a Sobel or Laplacian filter. A sufficiently long four-side straight line constituting a quadrangle is extracted from this enhanced image using a known straight line extraction method such as a Hough transform or a minimum approximation method. An example of an image obtained by performing edge enhancement on the target image 200 of FIG. 2 is shown in FIG. An unequal side rectangle 601 is a rectangle corresponding to the projection area frame 202. As shown in FIG. 6, a white plate region frame 602 is extracted outside the projection region frame 601 from the edge-enhanced image. In step S501, the inner frame of the two extracted frames is selected as the projection area frame, and the outer frame is selected as the white area frame.

  In step S504, a distortion correction value for correcting distortion of the captured image is extracted and distortion correction processing is performed. Here, the distortion correction value is used in a so-called reverse perspective transformation calculation that corrects an inequilateral quadrangular region that occurs when a rectangular region is projected onto a two-dimensional plane with a three-dimensional angle to the original rectangular region. It is a parameter of the transformation matrix. For example, as disclosed in Japanese Patent Application Laid-Open No. 2003-288588, by substituting the coordinates of the four vertices before conversion and the coordinates of the four vertices after conversion into the arithmetic expression of inverse perspective transformation, solving the obtained simultaneous equations Thus, the parameters of the transformation matrix are extracted.

  In this embodiment, the coordinates of the four vertices before conversion are the coordinates of the four vertices of the projection area frame 601 extracted in step S502. The coordinates of the four vertices after conversion are the four corners of the image area of the output size w × h with respect to the projection display area portion of the projection data specified in step S501, that is, (0,0), (w, 0), ( w, h), (0, h). Each parameter of the matrix obtained by solving the simultaneous equations created using both coordinates is set as a distortion correction value.

  Next, a white board area is cut out from the image photographed using the white board area frame extracted in S502, and is perspective-transformed using a distortion correction value to generate an image in which distortion is corrected. FIG. 7 is a diagram illustrating an example of an image after correction is performed in this manner.

  In step S504, the handwritten object extraction processing unit 405 divides the handwritten image and projection data from the captured image as objects. The divided objects store data related to shape and position.

  The object division process is performed as follows. First, the distortion corrected image generated in S503 is converted into a binary image so that the object image is black and the background image is white. This conversion can be performed by any of those known in the art. For example, a binary image can be obtained by acquiring a luminance histogram of a multilevel image and converting it to a binarized image using a threshold value obtained adaptively from those peaks, but the present invention is not limited to this. Next, a labeling image is generated by a known method. For example, the following methods are known. That is, the pixels of each line forming the binary image are scanned, and the start position and the end position are stored in a table for a section in which black pixels are continuous. When this process is completed for all lines, a labeling image can be generated by giving the same label to the section in contact between the lines and forming a region in which black pixels having the same label are continuous. Here, each pixel block having the same label in the labeling image is regarded as an object.

  Step S505 is a step that means that the iterative process up to the repetitive termination step S510 is executed for each of the objects extracted in step S504.

  In step S506, the handwritten object extraction processing unit 405 determines whether or not the target object is a handwritten image object. If it is an object of a handwritten image, the process proceeds to step S507. Otherwise, the process proceeds to step S508. The determination of whether or not the object is a handwritten image can be used by any method known in the art. For example, it is possible to determine in advance that the projected image and the handwritten image have different colors and determine whether the color of each object acquired from the captured image is the color for the projected image or the color for the handwritten image. it can. Alternatively, the determination may be made in consideration of the characteristics of the handwritten image based on the magnitude of the fluctuation of the line and the unevenness of the density. Furthermore, it may be determined by designating a handwritten color in advance and comparing the color information of the target object with the handwritten designated color information. In this case, if it is determined that the color is different, it can be determined as a projection object. Further, at the time of shooting an image, projection by the projection device may be stopped for handwriting judgment, and only a handwritten image may be taken and used for judgment. Furthermore, the position information of the content in the projection display area of the electronic data 404 specified earlier may be compared, and if there is no match, it may be determined as a handwritten object.

  The above determination results are held accompanying the data of each object. In this embodiment, the objects 204 and 205 in the photographed image 200 are determined as handwritten objects and the process proceeds to S507, and the object 203 is determined to be a non-handwritten object, that is, a projected object, and the process proceeds to S510.

  In step S507, the object placement processing unit 407 determines whether the position of the handwritten object is inside or outside the projection area frame. If the position of the handwritten object is inside the projection area frame, the process proceeds to S508. If not, that is, if the position is outside the projection area, the process proceeds to S509. In this embodiment, since the handwritten object 204 is located within the projection area 202, the process proceeds to S508, and since the object 205 is located outside the projection area 202, the process proceeds to S509.

  In step S508, the object arrangement processing unit 407 converts the outline of the line portion of the object, the core line, and the like into vector path data by a known vector conversion technique, and arranges it in the projection display area 410 on the layout data of the electronic data 404. Here, as a known vector conversion technique, there is a method disclosed in Japanese Patent No. 3026592. In Japanese Patent No. 3026592, horizontal and vertical inter-pixel vectors are detected based on the state of a target pixel and its neighboring pixels while raster scanning an image. Next, based on the connection state between these inter-pixel vectors, a technique for generating information that describes the circulation of connected pixel data called an outline vector as a set of inter-pixel vectors by extracting the outline of the image data. It is disclosed.

  In step S509, the object placement processing unit 407 converts the object into character code data using a known character recognition technique, and places it in the text editing data 412 of the electronic data 404.

  Step S510 is a repeated termination step. If there is an unprocessed object, the process proceeds to S505, and if all the objects have been processed, the process proceeds to the end.

  FIG. 8 shows an example in which output data 408 obtained by applying the first embodiment is displayed by a display / edit program. The window 800 is an entire display window of the display / editing program, and includes windows 801 to 807 described below. In the layout editing window 801, the slide preview 802 and the print preview window 803 are switched in the display / editing program of this embodiment, and FIG. 8 shows a state in which the editing window 805 is displayed. Further, the window 800 includes a text editing window 806 and a summary information display window 807.

  In the layout editing window 801, a display according to the contents of the layout editing data 409 of the output data 408 is made. In the first embodiment of the present invention, a projection display area 804 of the specified electronic data layout data and an object 808 obtained by vectorizing the handwritten object 204 are drawn. In the text editing window 806, display according to the contents of the text editing data 412 of the output data 408 is performed. In the embodiment of the present invention, an object 809 obtained by character recognition of the handwritten object 205 is drawn.

  FIG. 9 shows an example in which a slide preview 802 of the display / edit program is displayed. Only the projection display area 410 in the layout data 409 of the output data 408 is displayed, and the projection non-display area 411 and the text editing data 412 are not displayed.

  As described above, according to the present embodiment, a handwritten object is extracted from a white board image including a projection screen, and the data conversion format of the handwritten object is stored inside and outside the projection area and stored in the original electronic data of the projection screen. Generate different output data.

  In the display / editing program, the electronic data generated in this manner is received and held in association with the original electronic data as handwritten objects in the projection area as vector data and handwritten objects outside the projection area as character code data. . As a result, it is possible to provide the minutes in a reusable state.

[Example 2]
In the first embodiment, the character code data is stored by recognizing a handwritten object outside the projection area, but may be vectorized and stored as in the projection area. In this embodiment, the objects outside the vectorized projection area retain the same positional relationship as the actually captured image and are arranged in the projection non-display area 412 of the electronic data 408. That is, the handwritten object in the projection non-display area is vectorized to hold the data of the object itself, and the display position is held in relation to the projection display area.

  FIG. 10 shows an example in which output data 408 obtained by applying the second embodiment to the display / edit program is displayed. An object 1001 is a vectorization of the handwritten object 205 and is arranged outside the projection display area 804. At this time, the positional relationship between the object 1001 and the projection display area 804 holds the positional relationship between the original handwritten object 205 and the projection area frame 202.

  According to the present embodiment, the generated electronic data can be edited in the display / editing program in a state in which a handwritten object outside the projection area is close to the time of photographing.

[Example 3]
In the second embodiment, the electronic data is stored while maintaining the positional relationship of the handwritten object outside the projection area, but may be stored in the projection display area. In this embodiment, a handwritten object outside the projection area can be displayed in the projection display area 804.

  FIG. 11 shows an example in which output data 408 obtained by applying the third embodiment to the display / edit program is displayed. The object 1101 is a vectorization of the handwritten object 205, and the place where the object 1101 is arranged is a place on the projection display area 804 where no other object exists.

  According to the present embodiment, since the generated electronic data displays a handwritten object in the projection display area in the display / editing program, it is possible to provide the handwritten object outside the projection area in a state where it can be displayed during preview.

[Example 4]
In the third embodiment, the handwritten object outside the projection area is stored in the electronic data while being shifted in the projection display area so that the handwritten object can be displayed on the preview window. However, the object may be animated. In this embodiment, an animation is applied to a handwritten object outside the projection area.

  FIG. 12 shows an example in which output data 408 obtained by applying the fourth embodiment to the display / edit program is displayed. FIG. 12A shows a layout display window, and FIG. 12B shows a preview window. An object 1201 is a vectorization of the handwritten object 205, and is arranged in the projection non-display area in the layout display window. When the window 800 is switched to the preview window 802, the object 1201 is placed on the projected display area 804 of the layout data, and a margin area where no other object exists is extracted and arranged and moved. When the layout display window 801 is switched to, the object 1201 is moved to the projection non-display area.

  According to the present embodiment, the generated electronic data can be provided in a display / editing program so that a handwritten object outside the projection area can be displayed while maintaining the positional relationship of the object during editing and displayable during preview. .

[Example 5]
In the fourth embodiment, the handwritten object outside the projection area is stored in the electronic data so that it can be displayed on the editing window and the preview window, but it may be stored in an area that can be displayed during printing. In this embodiment, a handwritten object outside the projection area is displayed on the print preview 803 so that it can be printed out.

  FIG. 13 shows an example in which output data 408 obtained by applying the fourth embodiment to the display / edit program is displayed in the print preview window. An object 1301 is obtained by vectorizing the handwritten object 205, and the positional relationship between the projection area and the object 813 is arranged in the print preview window 803 while maintaining the positional relationship between the original handwritten object 205 and the projection area frame 202.

  According to the present embodiment, the generated electronic data can be provided in a displayable / editable program in a printable state while maintaining the positional relationship with the projection area for a handwritten object outside the projection area.

[Example 6]
In the fifth embodiment, the handwritten object outside the projection area is stored in the electronic data so that it can be printed. However, the projection display area of the original electronic data may be expanded. In this embodiment, a handwritten object outside the projection area is processed.

  FIG. 14 shows an example in which output data 408 obtained by applying the sixth embodiment to the display / edit program is displayed. A window 1401 is an expanded projection display area, and is enlarged to include a handwritten object 1402 outside the projection area.

  According to the present embodiment, the generated electronic data can be provided in a display / editing program in a state in which the handwritten object outside the projection area maintains the positional relationship with the projection area in both the editing window and the preview window. .

  An image processing device that projects electronic data on a hand-drawn projection plate, captures the handwritten state, and creates minutes.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed. The present invention can also be realized by a plurality of processors cooperating to perform processing.

Claims (11)

Projection means for projecting electronic data onto a hand-drawn projection plate;
Photographing means for photographing a handwritten object, which is an image handwritten on the projection plate, together with a projection image obtained by projecting the electronic data, and generating image data;
An electronic data specifying means for extracting a projection image taken together with the handwritten image from the image data, and specifying electronic data of the extracted projection image;
A frame extraction means for extracting a frame of a projection area projected on the projection plate by the projection means from the image data;
Extraction means for extracting the handwritten object from the image data, and holding the extracted handwritten object separately for an object inside the projection area and an object outside the projection area;
An image processing apparatus comprising: output means for outputting an object inside the extracted projection area and an object outside the projection area together with the electronic data specified by the electronic data specifying means.   The image processing apparatus according to claim 1, wherein the output unit outputs an object inside the projection area to an area where the specified electronic data is displayed.   3. The image according to claim 1, further comprising handwritten character recognizing means for recognizing an object outside the projection area to obtain a character code and holding the character code in association with the specified electronic data. Processing equipment.   The image processing apparatus according to claim 3, wherein the output unit outputs the character code acquired by the handwritten character recognition unit outside an area where the electronic data is displayed. Further comprising a blank extracting means for extracting a blank area in which nothing is displayed in the display area of the electronic data;
The image processing apparatus according to claim 1, wherein the output unit outputs an object outside the projection area to the margin area.   The image processing apparatus according to claim 5, wherein the output unit further includes a mode for outputting an object outside the projection area outside the display area of the specified electronic data.   The image processing apparatus according to claim 1, wherein the output unit outputs by printing.   The image processing apparatus according to claim 1, wherein the output unit performs output by displaying on a display device.   The output means enlarges an area where the specified electronic data is displayed, and outputs an object outside the projection area into the enlarged display area. The image processing apparatus according to any one of 8. A projection step of reading out and projecting electronic data from the storage means on a handwritten projection plate,
A photographing step of photographing a handwritten object, which is an image handwritten on the projection plate, together with a projection image obtained by projecting the electronic data to generate image data;
An electronic data specifying step of extracting a projection image taken together with the handwritten image from the image data, and specifying electronic data of the extracted projection image in the storage means;
A frame extraction step for extracting a frame of a projection area projected on the projection plate in the projection step from the image data;
An extraction step of extracting the handwritten object from the image data and dividing the extracted handwritten object into an object inside the projection area and an object outside the projection area and holding the storage means in the storage unit;
An image processing method comprising: an output step of outputting an object inside the extracted projection area and an object outside the projection area together with the electronic data identified in the electronic data identification step. A program for causing an image processing apparatus to execute an image processing method, the image processing method comprising:
A projection step of reading out and projecting electronic data from the storage means on a handwritten projection plate,
A photographing step of photographing a handwritten object, which is an image handwritten on the projection plate, together with a projection image obtained by projecting the electronic data to generate image data;
An electronic data specifying step of extracting a projection image taken together with the handwritten image from the image data, and specifying electronic data of the extracted projection image in the storage means;
A frame extraction step for extracting a frame of a projection area projected on the projection plate in the projection step from the image data;
An extraction step of extracting the handwritten object from the image data and dividing the extracted handwritten object into an object inside the projection area and an object outside the projection area and holding the storage means in the storage unit;
A program comprising: an output step of outputting an object inside the extracted projection region and an object outside the projection region together with the electronic data specified in the electronic data specifying step.

Images