JP2010273289A - Electronic information board system, computer terminal, and calibration method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an electronic board information system which reduces an error in projection transformation parameters to perform more correct calibration. <P>SOLUTION: A projector projects a calibration image including an indicator indicating a coordinate position in a projection image. A camera picks up an image of a region where the calibration image is projected by the projector. A computer terminal divides the pickup image picked up by the camera into a plurality of images, calculates the projection transformation parameters for projecting and transforming the pickup image into a projection image based on the indicator included in each of the plurality of division images, and projects and transforms the pickup image picked up by the camera into a projection image, based on the calculated projection transformation parameters. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an electronic information board system, a computer terminal, and a calibration method for moving a cursor on a computer terminal according to infrared light on a screen area.

  In recent years, an image displayed on a display unit of a computer terminal is projected onto a screen area such as a whiteboard by a projector, a pen moving on the screen area in accordance with a user operation is detected by a camera, and the locus of the detected pen There is provided an electronic information board system that draws an image on a computer terminal according to the operation. As the pen for moving the screen area, for example, a light emitting pen that emits infrared light by contacting a whiteboard is used. Then, the cursor on the computer terminal is moved according to the operation of the light-emitting pen captured by the camera that captures infrared light, and drawing is performed on the image on the computer terminal in accordance with the movement of the cursor. According to such an electronic information board system, a plurality of people can simultaneously view the image projected on the screen area by the projector, and the user can write on the projected image by moving the pen on the screen area. Can provide an electronic blackboard. Patent Document 1 proposes a remote lecture system using such an electronic information board system.

  In such an electronic information board system, when the cursor on the computer terminal is moved in accordance with the movement of infrared light on the screen area, it is projected by the projector from the coordinate system of the captured image of the screen area captured by the camera. It is necessary to convert the projected image into a coordinate system. Here, a projective transformation parameter from the coordinate system of the captured image to the coordinate system of the projected image is calculated, and the projective transformation is performed based on the calculated projective transformation parameter. The projective transformation parameter is calculated by substituting the four vertices of the quadrilateral of the image into the general formula of the projective transformation.

JP 2005-234368 A

  However, in order to calculate the projective transformation parameters, it is necessary to extract the quadrilateral of the image as described above, but the captured image captured by the camera is distorted due to distortion or slack of the screen area itself. Or due to aberration distortion caused by the camera lens, it may be a quadrilateral bulging into an ellipse. For example, when the projection image projected by the projector is a rectangle along the X axis and the Y axis as shown in FIG. 9, the captured image taken by the camera is an ellipse as shown in FIG. The image is swollen. If there is a deviation in the calculated projective transformation parameter due to such distortion, an error occurs when performing the projective transformation, and the infrared light coordinate position included in the captured image and the cursor coordinate included in the projected image The position will shift. That is, there is a difference between the operation on the screen area by the user using the light emitting pen and the drawing on the image on the computer terminal. Here, when performing projective transformation, it is desirable to correct such a shift due to distortion, reduce the error, and perform more accurate calibration.

  The present invention has been made in view of such a situation, and provides an electronic board information system, a computer terminal, and a calibration method which reduce errors in projective transformation parameters and perform more accurate calibration.

  In order to solve the above-described problems, the present invention includes a projector that projects a predetermined projection image, a camera that captures an area projected by the projector, and a computer terminal that receives an image captured by the camera. The electronic information board system, wherein the projector projects a calibration image including an index indicating a coordinate position in the projected image, and the camera captures an area where the calibration image is projected by the projector, and the computer terminal The image dividing unit that divides the captured image captured by the camera into a plurality of images and the plurality of images divided by the image dividing unit are converted into projected images based on indices included in the images. Projection to calculate projective transformation parameters for projective transformation A conversion parameter calculation unit, on the basis of the projective transformation parameter calculated by the projective transformation parameter calculating unit, characterized in that it and a projective transformation unit for projective transformation on the projection image an image captured by the camera.

  Further, according to the present invention, the camera described above captures visible light, and the projective transformation parameter calculation unit calculates the coordinate position of the index in the image, and calculates the projective transformation parameter based on the calculated coordinate position. And

  The present invention also provides a projection projector that includes a calibration image including an index indicating a coordinate position in an image to be projected, a camera that captures an area on which the calibration image is projected by the projector, and an image captured by the camera. A computer terminal in an electronic information board system including a computer terminal for receiving an image, and an image dividing unit that divides a captured image captured by a camera into a plurality of images, and a plurality of images divided by the image dividing unit For each, based on the projection conversion parameter calculated by the projection conversion parameter calculation unit, a projection conversion parameter calculation unit that calculates a projection conversion parameter for projectively converting the captured image into a projection image based on an index included in the image, Imaged by the camera A projective transformation unit for projective transformation of the captured image to the projection image, characterized in that it comprises a.

  According to another aspect of the present invention, there is provided a calibration of an electronic information board system including a projector that projects a predetermined projection image, a camera that captures an area projected by the projector, and a computer terminal that receives an image captured by the camera. A method in which a projector projects a calibration image including an index indicating a coordinate position in a projection image, a camera images an area in which the calibration image is projected by the projector, and a computer The image dividing unit of the terminal includes a step of dividing the captured image captured by the camera into a plurality of images, and the projective transformation parameter calculating unit includes each of the plurality of images divided by the image dividing unit. Based on the index A projective transformation parameter for projectively transforming the image into a projected image, and the projection transforming unit converts the captured image captured by the camera into a projected image based on the projective transformation parameter calculated by the projective transformation parameter calculating unit. And a projective transformation step.

  As described above, according to the present invention, the projector projects the calibration image including the index indicating the coordinate position in the projection image, and the camera captures the area where the calibration image is projected by the projector. The computer terminal divides the captured image captured by the camera into a plurality of images, and performs projective conversion of the captured image into a projected image based on an index included in each of the divided images. Projection conversion parameters are calculated, and based on the calculated projection conversion parameters, the captured image captured by the camera is projectively converted into a projected image. Projective transformation parameters can be calculated for each of the displayed screens. . Thereby, it is possible to perform the projective transformation with the projective transformation parameter with less error.

It is a figure which shows the concept of the electronic information board system by one Embodiment of this invention. It is a block diagram which shows the structural example of the electronic information board system by one Embodiment of this invention. It is a figure which shows the example of the calibration image by one Embodiment of this invention. It is a figure which shows the example of the captured image divided | segmented by one Embodiment of this invention. It is a figure which shows the example of the projection image divided | segmented by one Embodiment of this invention. It is a figure which shows the example of the drawing toolbox by one Embodiment of this invention. It is a flowchart which shows the operation example of the electronic information board system by one Embodiment of this invention. It is a flowchart which shows the operation example of the electronic information board system by one Embodiment of this invention. It is a figure which shows the example of the projection image by a prior art. It is a figure which shows the example of the captured image by a prior art.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram showing the concept of the electronic information board system 1 according to the present embodiment. Here, an example in which a user makes a presentation using the electronic information board system 1 is shown. For example, a whiteboard that is a screen area 200 for projecting a screen is installed in a room where a presentation is performed. The projector 100 is installed so as to project an image toward the screen area 200. The user stands in front of the screen area 200 and moves the light-emitting pen 210 on the image projected on the screen area 200 so as to write on the blackboard with chalk. A light emitting unit that emits infrared light is provided at the tip of the light emitting pen 210 and emits infrared light when it is detected that it is in contact with the screen region 200. The camera 300 is installed so as to capture the entire screen area 200. The computer terminal 400 is connected to the projector 100 by, for example, a VGA (Video Graphics Array) cable, and is connected to the camera 300 by a USB (Universal Serial Bus) cable.

FIG. 2 is a block diagram showing the configuration of the electronic information board system 1 according to the present embodiment. The projector 100 is a display device that receives an image displayed on the display unit 410 of the computer terminal 400 and projects the received image on the screen region 200 as a projection image.
The screen area 200 is a projection surface of an image projected by the projector 100. In the present embodiment, the screen area 200 is described as a whiteboard. However, the screen area 200 may be anything as long as an image is projected from the projector 100, and may be a flat part of an indoor wall, cloth, vinyl sheet, or the like. A screen may be used.

  The camera 300 is an imaging device that captures an image projected on the screen area 200. The camera 300 has a function of imaging visible light and infrared light, and an infrared image is obtained by installing a visible light cut filter (infrared light transmission filter) in front of a lens included in the imaging camera 300 to perform imaging. It is possible to image only light. On the other hand, if imaging is performed without installing a visible light cut filter, both visible light and infrared light are imaged.

The computer terminal 400 includes a display unit 410, a display control unit 420, a calibration image storage unit 430, an image reception unit 440, a screen division unit 450, a projection conversion parameter calculation unit 460, and a projection conversion parameter storage unit 470. A projection conversion unit 480, a drawing processing unit 490, and an overlay processing unit 495, each of which is operated by an operation command from a control unit such as an OS (Operating System).
The display unit 410 is a display that outputs an image, and is, for example, an LCD (Liquid Crystal Display).

The display control unit 420 causes the display unit 410 to output image information input from the OS or the like. For example, the display control unit 420 causes the display unit 410 to output a presentation image stored in advance in the storage area of the computer terminal 400.
The calibration image storage unit 430 indicates coordinate positions for calculating projective transformation parameters for calibrating (calibrating) the distortion between the projection image projected by the projector 100 onto the screen region 200 and the captured image captured by the camera 300. It is an image including an index. For example, FIG. 3 is a diagram illustrating an example of a calibration image stored in the calibration image storage unit 430. The calibration image has a rectangular shape corresponding to the aspect ratio of the screen of the display unit 410, and indexes are arranged at predetermined intervals in the image. In this example, a black circle is arranged as an index. As for the index, four vertices of a rectangle and four points in total are arranged at the center of each side.

  The image receiving unit 440 receives a captured image of the screen area 200 captured by the camera 300 from the camera 300. Here, the captured image received by the image receiving unit 440 from the camera 300 is an image in which the image output to the display unit 410 is distorted due to distortion of the screen region 200 itself, aberration distortion of the camera 300, or the like. Here, the captured image received by the image receiving unit 440 is an infrared light captured image in which visible light is cut and only infrared light is captured in a state where a visible light cut filter is installed in front of the lens of the camera 300. In the state where the visible light cut filter is not installed, the visible light captured image includes both visible light and infrared light.

The screen dividing unit 450 divides the visible light captured image received by the image receiving unit 440 into a plurality of images. For example, as illustrated in FIG. 4, the screen division unit 450 divides the visible light captured image, and includes a first divided image indicated by the symbol a, a second divided image indicated by the symbol b, and a third indicated by the symbol c. 4 divided images and a fourth divided image indicated by the symbol d are generated.
Projection conversion parameter calculation unit 460 calculates a projection conversion parameter for each of the divided images generated by screen division unit 450 based on the visible light captured image. The projective transformation parameter is obtained by a general formula of projective transformation represented by the following formula (1).

In the above formula (1), x and y indicate coordinate positions in the visible light captured image before projective transformation, and X and Y indicate coordinate positions in the projected image after projective transformation. That is, if the nine coefficients (a ₀ , a ₁ , a ₂ , b ₀ , b ₁ , b ₂ , c ₀ , c ₁ , c ₂ ) in the above formula (1) are given, The coordinate position (X, Y) of the projection image corresponding to the coordinate position (x, y) can be calculated. These nine coefficients serve as projective transformation parameters for performing projective transformation from a visible light captured image to a projected image. However, since the above formula (1) can share the numerator denominator with a scalar, there are eight independent unknowns. Therefore, the projective transformation parameter calculation unit 460 substitutes the four vertices of the quadrilateral of the visible light captured image and the four vertices of the projection image into the above equation (1), and solves the quaternary simultaneous equations to thereby compute the projective transformation parameters. Is calculated. The projective transformation parameter calculation unit 460 calculates the coordinate positions of the four vertices of the quadrilateral by detecting an index (black circle) included in the calibration image received as the visible light captured image. Projection conversion parameter calculation unit 460 calculates a projection conversion parameter for each of the divided images divided by screen division unit 450. For example, when a calibration image including an index at a position as illustrated in FIG. 3 is divided into four and the coordinate positions of four vertices are calculated for each of the divided divided images, each index included in the divided image is calculated. The vertex is calculated from the positional relationship. For example, the intersection of the line connecting the upper right index and the lower left index of the image before division and the line connecting the upper left index and the lower right index of the image before division is calculated as one vertex of the divided image. To do. Further, the projection conversion parameter calculation unit 460 stores the calculated projection conversion parameter in the projection conversion parameter storage unit 470.

  The projective transformation unit 480 performs projective transformation on the coordinate position of the infrared light contained in the infrared light captured image received by the image receiving unit 440 based on the projective transformation parameter stored in the projective transformation parameter storage unit 470, and A coordinate position in the projection image corresponding to the coordinate position of the infrared light imaged in the external light imaging is calculated. For example, the projective transformation unit 480 calculates a point (X, Y) in the projection image shown in FIG. 5 corresponding to the point (x, y) in the captured image shown in FIG.

The drawing processing unit 490 performs a drawing process according to the coordinate position calculated by the projective conversion unit 480 based on a predetermined setting. For example, the drawing processing unit 490 performs the same drawing process as that operated by the mouse at the coordinate position calculated by the projective transformation unit 480. Here, the drawing processing unit 490 displays a drawing tool box as shown in FIG. 6 on the display unit 410 in advance, receives input of setting information related to drawing processing, and stores the drawing setting information in its own storage area. You may make it leave. For example, when “pen” is selected in the drawing toolbox, the drawing processing unit 490 draws a drawing image in which a line having a thickness corresponding to the setting is drawn at the coordinate position of the infrared light after the projective transformation. Generate.
The overlay processing unit 495 performs overlay processing in which the drawing image generated by the drawing processing unit 490 is combined with the image output to the display unit 410 by the display control unit 420. Further, the image synthesized by the overlay processing unit 495 may be stored in the storage area of the computer terminal 400.

Next, an example of calibration operation by the electronic information board system 1 will be described with reference to the drawings.
FIG. 7 is a flowchart showing an operation example of the projective transformation parameter calculation process performed by the electronic information board system 1.
Here, it is assumed that a visible light cut filter is not installed in the lens of the camera 300 and a visible light captured image is captured. On the screen area 200, a calibration image received from the computer terminal 400 by the projector 100 is projected. The camera 300 captures the calibration image projected on the screen area 200, generates a visible light captured image, and outputs it (step S1). The image receiving unit 440 of the computer terminal 400 receives the visible light captured image output from the camera 300 (step S2). The screen dividing unit 450 generates a divided image obtained by dividing the visible light captured image received by the image receiving unit 440 into four (step S3).

  The projective transformation parameter calculation unit 460 performs projective transformation according to the above formula (1) based on the divided image of the visible light captured image generated by the screen division unit 450 and the calibration image stored in the calibration image storage unit 430. A parameter is calculated (step S4). The projection conversion parameter calculation unit 460 stores the calculated projection conversion parameter in the projection conversion parameter storage unit 470 (step S5).

FIG. 8 is a flowchart showing an operation example of projecting a projection image calibrated by the electronic information board system 1.
Here, it is assumed that a visible light cut filter is installed in the lens of the camera 300 and an infrared light captured image is captured. On the screen area 200, a presentation image received from the computer terminal 400 by the projector 100 is projected. The camera 300 captures the screen area 200, generates an infrared light captured image, and outputs it (step S11). The image receiving unit 440 of the computer terminal 400 receives the infrared light captured image output from the camera 300 (step S12). The projective transformation unit 480 reads the projective transformation parameters stored in the projective transformation parameter storage unit 470 (step S13). The projective conversion unit 480 calculates the coordinate position of the projection image corresponding to the coordinate position of the infrared light included in the infrared light captured image based on the read projective conversion parameter, and performs the projective conversion of the coordinate position. S14).

  The drawing processing unit 490 performs a predetermined drawing process according to the coordinate position calculated by the projective transformation unit 480, and generates a drawing image (step S15). The overlay processing unit 495 performs overlay processing to superimpose the drawing image generated by the drawing processing unit 490 on the image output from the display control unit 420 (step S16). The display control unit 420 outputs an image on which the drawing image is overlaid to the projector 100 (step S17). The projector 100 projects the image transmitted from the display control unit 420 of the computer terminal 400 onto the screen area 200 (step S18).

  In the present embodiment, the white board is applied as the screen area 200 on which an image is projected by the projector 100. However, for example, a human body model or a map may be used as the screen area 200. Even in this case, the overlay processing unit 495 can perform drawing by moving the light emitting pen 210 while emitting light between the screen region 200 and the camera 300. For this reason, even when the user uses the electronic information board system 1 as an object of the screen area 200 even if the object that cannot be erased by writing by applying ink using a pen or the like is used as the object of the screen area 200, Explanations can be given while drawing electronically.

  Further, in the present embodiment, the screen dividing unit 450 has been described as dividing the captured image into four. However, depending on the aspect ratio of the screen, the accuracy of calibration required, and the like, a plurality of images such as 6 and 8 are divided. And projective transformation parameters may be calculated for each of the divided images.

  In the present embodiment, the light-emitting pen 210 detects that the screen pen 200 has been touched and emits infrared light. For example, a button is provided on the light-emitting pen 210 and the button is pressed. Alternatively, infrared light may be emitted. That is, since drawing is performed by detecting infrared light emitted from the light emitting pen 210 existing between the screen area 200 and the camera 300, the light emitting pen 210 can perform drawing even if the light emitting pen 210 is not grounded. is there. In this way, when the object in the screen region 200 is easily deformed or easily damaged, it is possible to draw without touching the object. At this time, the drawing processing unit 490 can be used as a pointer by drawing a point at the coordinate position of the infrared light.

In addition, when accurate projection conversion parameters cannot be calculated by calibration using visible light due to the influence of the shape of the screen area 200 or other light irradiated on the screen area 200, calibration using infrared light is performed. May be performed.
In the present embodiment, the computer terminal 400 includes each function unit. However, a dedicated computer device may include a function unit necessary for drawing processing and calibration processing.

  As described above, according to the present embodiment, a captured image distorted due to aberration distortion or the like is divided into a plurality of parts, a projection conversion parameter is calculated for each of the divided images, and a projection of the captured image is calculated using the calculated projection conversion parameter. Since the conversion is performed, it is possible to improve the accuracy of the projective conversion as compared to performing the projective conversion using one projection change parameter for the entire captured image. Thereby, the coordinate position of the light emitting pen 210 with respect to the image projected on the screen area 200 can be more accurately reflected in the projected image. For this reason, the user can more accurately perform writing with the light-emitting pen 210 to the intended place, and can perform a presentation smoothly. On the other hand, presentation viewers can view presentations that are easier to understand without stress.

  In the present embodiment, only the infrared light is imaged by installing a visible light filter with respect to the camera 300, and the visible light is imaged by removing the visible light filter. It is not necessary to install two cameras, a camera and a camera for infrared light imaging, and the electronic information board system 1 can be configured more easily. Further, since the camera 300 can capture visible light, it is not necessary to perform calibration by causing the light-emitting pen 210 to emit light at a predetermined position of the screen area 200, and automatic calibration is possible. . It is also possible to take a snapshot of the screen area 200 and save it. In this case, when the screen area 200 is a whiteboard or the like, the information written on the whiteboard by the ink pen can be captured and stored.

  The program for realizing the function of the processing unit in the present invention is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into the computer system and executed for calibration. May be. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a storage device such as a flexible disk, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” is a volatile memory (RAM) inside a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

DESCRIPTION OF SYMBOLS 1 Electronic information board system 100 Projector 200 Screen area 210 Light-emitting pen 300 Camera 400 Computer terminal 410 Display part 420 Display control part 430 Calibration image memory | storage part 440 Image receiving part 450 Screen division part 460 Projection conversion parameter calculation part 470 Projection conversion parameter memory | storage Unit 480 projection conversion unit 490 drawing processing unit 495 overlay processing unit

Claims (4)

A projector that projects a predetermined projection image; a camera that captures an area in which the projection image is projected by the projector to generate a captured image; and a computer terminal that receives the captured image generated by the camera. An electronic information board system comprising:
The projector is
Projecting a calibration image including an index indicating the coordinate position in the region,
The camera
Imaging the area where the calibration image is projected by the projector to generate the captured image;
The computer terminal is
An image dividing unit that divides the captured image generated by the camera into a plurality of images;
For each of the plurality of images divided by the image dividing unit, a projective conversion parameter calculating unit that calculates a projective conversion parameter for projectively converting the captured image to the projected image based on the index included in the image. When,
A projective conversion unit for projectively converting the captured image captured by the camera into the projected image based on the projective conversion parameter calculated by the projective conversion parameter calculating unit;
An electronic information board system comprising: The camera captures visible light;
The electronic information board according to claim 1, wherein the projective transformation parameter calculation unit calculates a coordinate position of the index in the image, and calculates the projective transformation parameter based on the calculated coordinate position. system. A projector that projects a calibration image including an index indicating a coordinate position in the image to be projected, a camera that captures an area where the calibration image is projected by the projector and generates a captured image, and a camera that generates the captured image A computer terminal in an electronic information board system comprising a computer terminal for receiving the captured image,
An image dividing unit that divides the captured image generated by the camera into a plurality of images;
For each of the plurality of images divided by the image dividing unit, a projective conversion parameter calculating unit that calculates a projective conversion parameter for projectively converting the captured image to the projected image based on the index included in the image. When,
A projective conversion unit for projectively converting the captured image captured by the camera into the projected image based on the projective conversion parameter calculated by the projective conversion parameter calculating unit;
A computer terminal comprising: Electronic information comprising: a projector that projects a predetermined projection image; a camera that captures an area projected by the projector to generate a captured image; and a computer terminal that receives the captured image generated by the camera A board system calibration method comprising:
The projector is
Projecting a calibration image including an index indicating a coordinate position in the region;
The camera is
Imaging the area where the calibration image is projected by the projector to generate the captured image;
Of the computer terminal,
An image dividing unit dividing the captured image generated by the camera into a plurality of images;
A projective transformation parameter calculating unit calculates a projective transformation parameter for projectively transforming the captured image into the projected image based on the index included in the image for each of the plurality of images divided by the image dividing unit. A calculating step;
A projective transformation unit, based on the projective transformation parameter calculated by the projective transformation parameter calculating unit, projectively transforms the captured image captured by the camera into the projected image;
A calibration method comprising:

Images