JP2007194724A - Projection type display apparatus, written image extract method and program thereof, and computer-readable information recording medium with program recorded thereon - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a projection type display apparatus capable of attaining highly accurate color conversion and to provide a written image extract method and a program thereof, and a computer-readable recording medium in which the program is recorded. <P>SOLUTION: The projection type display apparatus includes: a screen 1; a CCD camera 2 for photographing a written image formed on the screen 1; and an extraction section 42 that extracts information of a part corresponding to the written image from an image photographed by the CCD camera 2 and replaces colors of the extracted written image with representative colors, and the extraction section 42 carries out averaging by replacing a pixel value of a target pixel of the written image with an average of pixel values of the written pixels within a square region around the target pixel. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a projection display device written image extraction method for projecting and displaying image light on a screen or the like, a program thereof, and a computer-readable information recording medium on which the program is recorded.

  As a conventional technique related to extracting information written on a screen, “Reader Printer” disclosed in Patent Document 1, “Electronic Blackboard System” disclosed in Patent Document 2, and “Projection Type” disclosed in Patent Document 3 Display device and program ".

  In the invention disclosed in Patent Document 1, an image projected on a transmission screen or an image written with a pen or the like on a transparent member provided on the front surface of the transmission screen is provided in the vicinity of the transmission screen. Scanning is performed using a relatively large linear light receiving sensor.

  The invention disclosed in Patent Document 2 electronically captures an image drawn on an electronic blackboard with a pen or the like, and combines it with an image projected on the electronic blackboard with a projector.

The invention disclosed in Patent Document 3 projects from the back of the screen, writes on the display screen, images with a built-in imaging device (CCD camera or the like), and digitizes to obtain image information.
Japanese Patent Publication No.7-46198 JP-A-10-297166 JP 2003-143348 A

  However, in the invention disclosed in Patent Document 1, since the light receiving sensor moves on the screen surface and reads the written information, the apparatus becomes large.

  In the invention disclosed in Patent Document 2, it is necessary to move the writing surface to pass through the scanner unit in order to read the writing information, which takes time to read.

  Since the invention disclosed in Patent Document 3 performs a process of subtracting the projection image from the written image obtained by the imaging unit, it is not possible to remove noise that occurs due to the shadow of an obstacle near the screen and varies depending on the situation.

Furthermore, when a character written with a pen such as a marker pen is enlarged, the pixels vary, and there is a possibility that the determination accuracy becomes very low when color conversion is performed with a representative color.
This is because the writing information is diffused when the character itself written with the marker pen has a variation in density and is transmitted from the writing surface to the photographing surface.

  Thus, conventionally, there has not been provided a projection display device capable of performing color conversion with high accuracy, a written image extraction method and a program thereof, and a computer-readable information recording medium on which the program is recorded.

  The present invention has been made in view of such problems, and provides a projection display device capable of performing color conversion with high accuracy, a writing image extraction method and a program thereof, and a computer-readable information recording medium on which the recording is recorded. With the goal.

  In order to achieve the above object, according to a first aspect of the present invention, a projection surface on which a projection image is formed, a writing surface on which a writing image can be formed overlapping with the projection image, and a writing surface are formed. Projection type having imaging means for photographing a written image, extraction means for extracting information of a portion corresponding to the written image from an image photographed by the imaging means, and color conversion means for replacing the color of the extracted written image with a representative color In the display device, the color conversion unit averages the pixel value of the pixel of interest of the written image by replacing the pixel value of the pixel of interest in the square area centered on the pixel of interest. The present invention provides a projection display device having an averaging means.

  In the first aspect of the present invention, it is preferable that the extraction unit removes the background portion from the image captured by the imaging unit.

  In any of the above-described configurations of the first aspect of the present invention, the extraction unit causes the imaging unit to capture a plain image of the projection plane when the apparatus is activated, stores the luminance distribution in the image, and based on this, the imaging unit It is preferable to correct the luminance distribution of the photographed written image. The representative colors are preferably black, blue, and red. In addition, the color conversion means determines in which order the colors of the written image are to be converted in the order of black, blue, and red. It is more preferable to provide means for performing the above. In addition, the difference between the maximum value and the minimum value of the color components of each RGB color after averaging in the target pixel is calculated, and when the difference is equal to or less than a predetermined value, the color conversion means determines the color of the target pixel. If the luminance value of blue is the maximum among the luminance values of the color components of RGB after averaging in the pixel of interest, the color conversion means converts the color of the pixel of interest into blue Or, it is more preferable that the color conversion unit converts the color of the pixel of interest into red when the luminance value of red is the maximum among the luminance values of the RGB color components after averaging in the pixel of interest. .

  In order to achieve the above object, the present invention provides, as a second aspect, a projection process for forming a projection image, a writing process for forming a writing image overlapping with the projection image, and a writing formed in the writing process. Write image extraction including image capture processing for capturing an image, extraction processing for extracting information on a portion corresponding to the write image from the image captured by the image capture processing, and color conversion processing for replacing the color of the extracted write image with a representative color In the color conversion process, the pixel value of the target pixel of the written image is averaged by replacing it with the average value of the pixel values of the write pixel in the square area centered on the target pixel. The present invention provides a method for extracting written images.

  In the second aspect of the present invention, in the extraction process, it is preferable to remove the background portion from the image captured in the imaging process.

  In any of the above-described writing image extraction methods according to the second aspect of the present invention, in the extraction processing, the luminance of the writing image captured by the imaging processing is based on the luminance distribution in the plain image of the projection plane recorded in advance. It is preferable to correct the distribution. The representative colors are preferably black, blue, and red. In addition, in the color conversion process, the representative color of the written image is determined in the order of black, blue, and red. More preferably. In addition, the difference between the maximum value and the minimum value of the RGB color components after averaging in the target pixel is calculated, and when the difference is equal to or smaller than the predetermined value, the color of the target pixel is calculated in the color conversion process. If the luminance value of blue is the maximum among the luminance values of the RGB color components after averaging in the pixel of interest, the color of the pixel of interest is converted to blue in the color conversion process Alternatively, it is more preferable that the color of the pixel of interest is converted to red in the color conversion process when the luminance value of red among the color components of each RGB color component after averaging in the pixel of interest is the maximum. .

  In order to achieve the above object, the present invention provides, as a third aspect, a program for causing a computer to execute the written image extraction method according to any one of the second aspect of the present invention.

  In order to achieve the above object, the present invention provides, as a fourth aspect, a computer-readable information recording medium on which a program of the written image extraction method according to the third aspect of the present invention is recorded. Is.

  ADVANTAGE OF THE INVENTION According to this invention, the projection display apparatus which can perform color conversion with high precision, the writing image extraction method, its program, and the computer-readable information recording medium which recorded this can be provided.

FIG. 1 shows a projection display device according to a preferred embodiment of the present invention.
A projection display device shown in FIGS. 1A and 1B includes a screen (projection surface) 1 that forms a predetermined projection image, a writing surface that can form a writing image so as to overlap the projection image, and a writing surface. And an imaging device that captures a written image formed thereon.

FIG. 2 shows a cross section of the projection display device according to the present embodiment.
The imaging device 2 is installed so that the entire screen 1 on which the projection device 3 projects an image can be taken within the angle of view. As shown in FIG. 3, the imaging device 2 and the projection device 3 are connected to a control device 4. The projection display device according to the present embodiment can be realized by using a CCD camera as the imaging device 2, a projector as the projection device 3, a computer as the control device 4, and causing the computer to execute a control program. The installed hardware can be applied as the control device 4.
Hereinafter, a CCD camera is used as the imaging device 2, a projector is used as the projection device 3, and a computer is used as the control device 4. The CCD camera 2, projector 3, and computer 4 are also used.

  The user can directly write on the screen 1 a sentence, a figure, or the like to be added to the projected image while referring to the image from the computer 4 projected by the projector 3. When saving the written content together with the projected image, the user inputs a command to the control program at an arbitrary timing (such as clicking a dialog button). This command input may be executed not by software processing by a program but by providing a dedicated switch.

  When a command is input by the user, a photographing command is transmitted from the photographing unit 41 to the CCD camera 2. The CCD camera 2 that has received the imaging command from the computer 4 captures an image on the screen 1, and the obtained image is sent to the computer 4. The computer 4 processes the image received from the CCD camera 2 by the extraction unit 42 and extracts the written image. The computer 4 sends the extracted image to the combining unit 43 and combines it with the previously projected image. The projection unit 44 transmits a projection command for the synthesized image to the projector 3. The projector 3 that has received the projection command from the computer 4 projects the composite image on the screen.

  By this operation, the image written on the screen 1 by the user is converted into electronic information.

FIG. 4 shows a flow of operations of the extraction unit 42 of the projection display device according to the present embodiment.
<Projection Light Black Display Processing: Step S101>
When the user inputs an image capture command, the computer 4 projects a black image from the projector 3. This is an operation for putting the projector 3 in a light-shielding state in order to avoid unnecessary information from being reflected in an image taken by the CCD camera 2 thereafter. If the projector 3 has a mute function, it can also be used.

<Screen surface shooting process: Step S102>
When the projector 3 is in a light shielding state, the screen 1 is photographed by the CCD camera 2. FIG. 5 shows an example of an image of the screen 1 on which writing has been performed.

<Background removal processing: Step S103>
FIG. 6 shows an example of an image in which a shadow appears due to the user holding his hand near the screen 1. In the background removal process, noise such as shadows is removed, and only a writing part with a pen or the like is extracted.
The part written by the pen touches the surface, the outline is clear, the inside density is high density (low gradation when converted to gray scale), and the background and shadow parts are low density Therefore, the writing part and the other part are separated by the binarization process.

In general, binarization processing is performed to extract a target portion by removing a background portion from a target image. That is, a certain density value is set as a threshold value, and the background area and the target area are divided. As a method for determining the threshold value, a mode method, a p-tile method, a differential histogram method, and the like are known, and these are methods for fixing the threshold value to a constant value.
In the present invention, it is preferable to use a variable threshold process because the way in which shadows and noise are generated changes depending on obstacles and human behavior during photographing. This is a method of scanning the pixels in the image and changing the threshold value for each pixel of interest. In the present embodiment, the average density value of the local region for N × N pixels around the pixel of interest is used as a threshold value.

FIG. 7 shows an example of the local region when N = 3.
An average value g_ave of the green component g values of nine pixels in the 3 × 3 region including the target pixel is calculated, and if the g value of the target pixel is lower than this value, it is an extraction target (that is, a writing portion). If the g value is high, the background, shadow, or noise is determined. If it is determined that the background, shadow, or noise is detected, the target pixel is replaced with white (that is, RGB (255, 255, 255)).
By this processing, as shown in FIG. 8, the background portion can be removed and only the writing portion can be extracted.

<Shading correction processing: Step S104>
When photographing with the CCD camera 2, external light incident from the front surface of the screen 1 is used, so that unevenness in the amount of light may occur depending on the ambient light conditions depending on the installation location. FIG. 9 shows an example of an image of the screen 1 that has not been written. This image has a high luminance on the left side, and a density gradient is generated in the image.
In order to avoid the threshold value set when extracting the writing portion from being changed depending on the position in the image due to the unevenness of the light amount, the writing image is subjected to shading correction using the plain image. The density gradient can be corrected by determining the ratio of the density value of the pixel value of the writing portion to the pixel value of the plain image and replacing the pixel value with a value converted to a width of 0 to 255.

Depending on the ambient light conditions, bright and dark portions, that is, luminance gradients are generated in the image. A writing portion written in a portion where the luminance of the plain image is high has a high luminance, and a writing portion written in a portion where the luminance of the plain image is low is photographed with a low luminance.
It is assumed that the luminance gradient on a certain line (y = j) in the plain image is in the state shown in FIG. When writing is performed in this environment, as shown in FIG. 11, even if writing is performed using a pen of the same color, the luminance value of the writing portion is not constant due to the influence of the luminance gradient, and the luminance value varies depending on the location. Occurs.
In this way, even if writing is performed with the same color, the captured image has a large or small RGB value. Therefore, the brightness gradient of the written captured image is corrected based on the brightness gradient of the plain image.

Since the angle of view of the plain image captured in advance is the same as that of the captured image after writing, the coordinates (I, J) on the plain image are the coordinates ( I, J) corresponding to one to one. Assuming that the pixel values of a pixel in the written photographed image are Rc, Gc, and Bc, and the pixel values on the plain image having the same coordinates are Rn, Gn, and Bn, the corrected pixel values R, G, and B are
R = (Rc / Rn) × 255
G = (Gc / Gn) × 255
B = (Bc / Bn) × 255
It is expressed.
If the luminance gradient is corrected in this way, the luminance difference within the same color is eliminated as shown in FIG. 12, and a written photographed image with a clear hue as shown in FIG. 13 is obtained, and a higher quality written extracted image is obtained. Is obtained.

  A plain image for correcting unevenness in the amount of light may be taken automatically when the apparatus is activated, or may be taken at an arbitrary timing in accordance with a user operation.

<Color conversion processing: Step S105>
FIG. 14 shows the flow of operations in the color conversion process. The color conversion process is divided into three processes: an averaging filter process (step S201), a color determination process (step S202), and a correction process (step S203).

FIG. 15 shows an operation flow in the averaging filter processing.
First, an image pImage after shading is acquired (step S301). Next, an area of the image filtimg after the averaging filter is secured (step S302). Then, pImage is copied to filtimg (step S303). If the image pImage after shading is white (step S304 / Yes), it moves to the next pixel position in raster order without doing anything.
When the image pImage after shading is not white (step S304 / No), the average value of the pixels that are not white in the N × N pixel area centered on the pixel of interest is replaced with a prime value of the attention (steps S305 to S309). ). Specifically, the value of totalcount is initialized to 0 (step S305). If pImage is not white (step S306 / No), totalcount is incremented, and the value of pImage at that position is added to filtimg (step 307). Thereafter, if the filter range at the current position has not ended (step S308 / No), the pImage is moved to the next position in the filter range, and then the process returns to step S306, and the above processing is repeated until the filter range ends. .

  When the filter range ends (step S308 / Yes), the average value of pixels other than white in the filter range is calculated by dividing the value of filtimg by the value of totalcount, and the pixel value of the pixel of interest is replaced with this value ( Step S310). Thereafter, pImage is returned to the current position (step S311). If the process at the final pixel position is not completed (No at Step 312), the process moves to the next pixel position in raster order (Step S313), and the process at the final pixel position is completed (Yes at Step S312). To do.

  FIG. 16 shows the flow of operations in the color discrimination process. First, an image filtimg after the averaging filter process is acquired (step S401). Next, an area of the image tmpimg after color discrimination is secured (step S402). Then, pImage is copied to tmpimg (step S403). Next, the RGB maximum value maxrgb and color maxcol of filtimg are calculated (step S404). Next, the RGB minimum value minrgb and its color mincol of filtimg are calculated (step S405). Next, a difference diffrgb between maxrgb and minrgb is calculated (step S406). When tmpimg is white (step S407 / Yes), pImage is substituted (step S414). When tmpimg is not white (step S407 / No), when (mincol is blue) and (maxcol is green or diffrgb <thA (thA is an arbitrary threshold) or | rg | <thB (thB is an arbitrary threshold)) ( (Step 408 / Yes), tmpimg is determined to be black (step S411). Note that thA and thB are arbitrary thresholds, but when these are experimentally changed, thA is 10 ≦ thA ≦ 30, more preferably thA = 20, and thB is 5 ≦ thB ≦ 15, more preferably The color discrimination result was the best when thB = 10. Further, when maxcol is blue and mincol is red (step S409 / Yes), tmpimg is determined to be blue (step S412). When maxcol is red (step S410 / Yes), tmpimg is determined to be red (step S413). When the color discrimination at the last pixel position is completed, the process is terminated. If none of the conditions is satisfied in steps S408 to S410 (step 410 / No), pImage is substituted into tmpimg (step S414). If the processing at the final pixel position is not completed (step 415 / No), the process moves to the next pixel position in raster order (step S416). If the processing at the final pixel position is completed (step S415 / Yes), the process ends. To do.

FIG. 17 shows a flow of operations in the correction process. First, an image pImage after shading is acquired (step S501). Next, the image tmpimg after color discrimination is acquired (step S502). If the image tmpimg after color discrimination is not blue or red (step S503 / No), it is corrected to tmpimg at the current position (step S513). When the image tmpimg after color discrimination is blue or red (step S503 / Yes), first, totalcount and count are initialized to 0 (step S504). Next, if tmpimg is not white (step S505 / No), totalcount is incremented (step S506). Next, if tmpimg is black (step S507 / Yes), count is incremented (step 508). If the correction range at the current position has not ended (step S509 / No), tmpimg is moved to the next position within the correction range (step S510), and this process is repeated within the correction range centered on the target pixel. When the target pixel reaches the end of the correction range (step S509 / Yes), the correction loop for the target pixel is terminated.
Next, the ratio rate of black pixels in the marker color within the correction range in the target pixel is calculated (step S511). The calculation formula is rate = count / totalcount. If rate is larger than the arbitrary threshold value thr (step S512 / Yes), the pImage at the current position is corrected to black (step S514). If rate is equal to or less than thr (step S512 / No), the pImage at the current position is corrected to tmpimg (step S513). Although thr is an arbitrary threshold value, when this value is experimentally changed, the correction result is the best when 0.15 ≦ thr ≦ 0.25, more preferably thr = 0.2. there were. Next, tmpimg is returned to the current position, and the correction for the pixel of interest is completed (step S515). If the processing at the final pixel position is not completed (step 516 / No), the process moves to the next pixel position in raster order (step S517), and the processing at the final pixel position is completed (step S516 / Yes). To do.

<Minimum area extraction processing: Step S106>
Since the image obtained by the correction is the size of the image from the camera, the minimum area of only the part with writing is cut out. For example, in the processing up to the previous stage, since the parts other than the writing unit are converted to white, the scanning from the left to the right is repeated starting from the upper left of the image from left to right, and the first hit with a pixel other than white is x1. Let x2 be the last location that hits a pixel other than white. Similarly, scanning is repeated from the top to the bottom starting from the top left of the image from the top to the bottom, and the first hit with a pixel other than white is y1, and the last hit with a pixel other than white is y2. As shown in FIG. 18, an image corresponding to the written area is obtained by cutting out a rectangular area having (x1, y1) and (x2, y2) as the upper left and lower right vertex coordinates, respectively. The operability at the time of performing the synthesis process later is improved.

As described above, the projection display apparatus according to the present embodiment can accurately determine the color of the written image by correcting the variation due to the diffusion of the image transmitted through the screen by the averaging process.
In addition, in order to eliminate erroneous determination, pixels that cannot be determined by the determination formula are used as they are before averaging, so that a color-converted image that does not feel uncomfortable without being erroneously determined can be obtained.

  The process of combining the extracted image with the originally projected image and the process of projecting can be applied with the same method as in the prior art, and thus description thereof is omitted.

In addition, the said embodiment is an example of suitable implementation of this invention, and this invention is not limited to this.
For example, the flowchart shown in the above embodiment is merely an example of the operation, and the present invention is not limited to this. For example, the numerical values indicated by the branch condition may be different from those in the above embodiment.
As described above, the present invention can be variously modified.

It is a figure which shows the external appearance of the projection type display apparatus concerning suitable embodiment of this invention. It is a figure which shows schematic structure of the projection type display apparatus concerning suitable embodiment of this invention. It is a figure which shows the function structure of a projection type display apparatus. It is a flowchart which shows the flow of operation | movement of the extraction part of operation | movement of a projection type display apparatus. It is a figure which shows an example of the image which image | photographed the screen in which writing was made. It is a figure which shows an example of the image in which the shadow was reflected. It is a figure which shows an example of a local area | region. It is a figure which shows an example of the image which removed the background part and extracted only the writing part. It is a figure which shows an example of the image which image | photographed the screen which has not been written. It is a figure which shows the brightness | luminance gradient on a certain line in a plain image. It is a figure which shows the state in which the luminance value of the writing part was not fixed by the influence of a luminance gradient, and the luminance value differed according to the place. It is a figure which shows the state which correct | amended the brightness | luminance gradient and eliminated the brightness | luminance difference within the same color. It is a figure which shows an example of the writing picked-up image image | photographed with the clear hue. It is a flowchart which shows the flow of operation | movement in a color conversion process. It is a flowchart which shows the flow of operation | movement by the averaging filter process. It is a flowchart which shows the flow of operation | movement in a color discrimination process. It is a flowchart which shows the flow of operation | movement in a correction process. It is a figure which shows an example of the method of cutting out the minimum area | region only of a part with writing.

Explanation of symbols

1 Screen 2 Imaging device (CCD camera)
3 Projector (projector)
4. Control device (computer)
41 Image capturing unit 42 Extraction unit 43 Composition unit 44 Projection unit

Claims (18)

A projection surface on which a projection image is formed, a writing surface on which a writing image can be formed overlapping with the projection image, an imaging means for photographing the writing image formed on the writing surface, and photographing by the imaging means A projection display device comprising: extraction means for extracting information on a portion corresponding to the written image from the image obtained; and color conversion means for replacing the extracted color of the written image with a representative color,
The color conversion means includes averaging means for averaging by replacing the pixel value of the target pixel of the written image with the average value of the pixel values of the write pixel in a square area centered on the target pixel. A projection display device characterized by that.   The projection display apparatus according to claim 1, wherein the extraction unit removes a background portion from the image captured by the imaging unit.   The extraction unit causes the imaging unit to capture a plain image of the projection plane when the apparatus is activated, stores the luminance distribution in the image, and corrects the luminance distribution of the writing image captured by the imaging unit based on the image. The projection display device according to claim 1, wherein the projection display device is a projection display device.   The projection display apparatus according to claim 1, wherein the representative colors are black, blue, and red.   5. The projection display device according to claim 4, wherein the color conversion means includes means for determining which representative color the color of the written image is to be converted in order of black, blue, and red.   The difference between the maximum value and the minimum value of the color components of the RGB colors after the averaging at the target pixel is calculated, and when the difference is equal to or smaller than a predetermined value, the color conversion unit calculates the color of the target pixel. 6. The projection display device according to claim 5, wherein the projection display device converts the color to black.   The color conversion means converts the color of the pixel of interest into blue when the luminance value of blue is the maximum among the luminance values of the RGB color components after the averaging in the pixel of interest. The projection display device according to claim 5.   The color conversion means converts the color of the pixel of interest into red when the luminance value of red is the maximum among the luminance values of the RGB color components after the averaging in the pixel of interest. The projection display device according to claim 5. From projection processing for forming a projection image, writing processing for forming a writing image overlapping with the projection image, imaging processing for photographing the writing image formed in the writing processing, and an image photographed by the imaging processing A writing image extraction method comprising: an extraction process for extracting information of a portion corresponding to the writing image; and a color conversion process for replacing the extracted color of the writing image with a representative color,
In the color conversion process, the pixel value of the target pixel of the written image is averaged by replacing the pixel value with the average value of the write pixel in a square area centered on the target pixel. Write image extraction method.   The written image extraction method according to claim 9, wherein in the extraction process, a background portion is removed from the image captured in the imaging process.   11. The extraction process corrects the luminance distribution of the writing image captured by the imaging process based on a luminance distribution in a plain image of the projection plane recorded in advance. Writing image extraction method.   12. The written image extraction method according to claim 9, wherein the representative colors are black, blue, and red.   13. The written image extraction method according to claim 12, wherein in the color conversion process, it is determined in order of black, blue, and red which color the written image is to be converted into.   The difference between the maximum value and the minimum value of the RGB color components after the averaging at the target pixel is calculated, and when the difference is equal to or smaller than a predetermined value, the color of the target pixel is calculated in the color conversion process. 14. The written image extracting method according to claim 13, wherein the written image is converted to black.   Of the luminance values of the RGB color components after averaging at the pixel of interest, when the luminance value of blue is the maximum, the color of the pixel of interest is converted to blue in the color conversion process. The written image extracting method according to claim 13.   Of the luminance values of the RGB color components after the averaging in the pixel of interest, when the luminance value of red is the maximum, the color of the pixel of interest is converted to red in the color conversion process. The written image extracting method according to claim 13.   An image extraction method program for causing a computer to execute the written image extraction method according to any one of claims 9 to 16.   18. A computer-readable information recording medium on which a program for an image extraction method according to claim 17 is recorded.

Images