JP2015220503A - Image processing system, projection and imaging method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To acquire a screen image with a resolution reduction suppressed more than conventional systems.SOLUTION: A projector 1 for projecting on a screen a target image being a target of output comprises: a projection image processor 106 which generates a corrected image by correcting the target image so as to be displayed in a rectangular shape on the screen; a projection unit 12 which selectively projects the target image and the corrected image on the screen; and a first imaging unit 13 which acquires an imaged image by imaging the screen when the target image is displayed on the screen.

Description

  The present invention relates to a technique for projecting an image by a projector or the like and photographing the image.

  In recent years, projectors that project an image displayed on a personal computer or a smartphone onto a screen have become widespread. The projector is sometimes called an “image projector”.

  In addition to image data such as JPEG (Joint Photographic Experts Group), GIF (Graphics Interchange Format), or TIFF (Tagged Image File Format), images can be output using image data in the format of a presentation application. Projectors are also commercially available.

  A whiteboard can be used as a screen. Alternatively, white vinyl can be pasted on the wall and used as a screen. In these cases, the user can hand-write a memo or the like with a pen when the image is reflected on the screen. Then, the user can take a picture with a digital camera and record it.

  The following techniques have been proposed as a technique for projecting an image on a screen and a technique for photographing the screen.

  According to the technique described in Patent Document 1, the four sides of the board and the four sides of the projection area are detected from an input image obtained by photographing the board on which the projection object is projected by the projection device and the handwritten object is handwritten by the user. Then, the first and second frames are extracted based on the detected four sides, and the first distortion correction value is obtained from the first frame, and the second distortion correction value is obtained from the second frame. An object is extracted from the input image, the first distortion correction value is used for a handwritten object, and the second distortion correction value is used for a projection object. Corrected image data is generated.

  According to the technique described in Patent Document 2, an image distorted in a direction opposite to the direction of distortion generated on a screen from a square is displayed on a light valve, and this is projected through a projection lens. As a result, the distortion of the image displayed on the screen is corrected, and a corrected projected image having a rectangular outline is visually recognized. At this time, a captured image is obtained as an output from the image sensor that has captured the corrected projected image. Since this captured image is distorted, a corrected captured image is obtained by using the captured image distortion correcting means in order to correct the captured image to a rectangular image.

  According to the technique described in Patent Document 3, an image correction unit that performs correction processing such as color correction, luminance correction, and trapezoidal distortion correction on video information to be projected input to the projection image input unit, and the image correction In a video display device comprising: a projector that projects video on the screen surface based on video information corrected by the unit; and a camera that captures video on the screen surface projected by the projector, the projector includes an image of a test chart And the camera takes an image of the projected test chart. Then, the control unit obtains a correction amount by comparing the video information of the test chart photographed by the camera and the image information of the test chart projected on the projector, and the image correction unit determines the correction amount according to the correction amount. Correction processing is performed on video information to be projected.

  According to the technique described in Patent Document 4, a fixed distance is formed between a projection system including a micromirror element and a projection lens that form an optical image corresponding to an input image signal and project it toward a projection surface. A pair of imaging systems including a first imaging lens and a CCD, a second imaging lens and a CCD, and the positions of images captured by the pair of imaging systems. A control unit that calculates a distance and an angle from the phase difference to the projection surface and corrects an image projected by the projection system based on the obtained distance and angle to the projection surface;

  According to the technique described in Patent Document 5, a search process for searching for an image corresponding to the predetermined image from a captured image obtained by capturing an image obtained by projecting the predetermined image onto a projection plane, and the search based on the searched image. A parameter calculation process for calculating a perspective distortion correction parameter for the captured image, and a captured image captured after calculating the parameter based on the calculated parameter and a correction process for correcting the captured image are performed. To do.

JP, 2011-141600, A JP 2005-64667 A JP 2003-283964 A JP 2007-94036 A JP 2013-257463 A

  According to the prior art, when an image subjected to trapezoidal correction is projected onto a screen, it is displayed on the screen in the shape of the original image, that is, a rectangle. However, the resolution is lower than the original image.

  Moreover, when the image displayed on the screen is photographed and keystone correction is performed, the original image shape, that is, a rectangle can be adjusted, but the resolution is lowered. Similarly, an image handwritten on the screen (handwritten image) has a lower resolution than before the keystone correction.

  Then, even if the screen is shot and recorded, it may be difficult for the user to recognize what is written thereafter.

  In view of such a problem, an object of the present invention is to obtain a screen image while suppressing a reduction in resolution as compared with the prior art.

  An image projection apparatus according to an aspect of the present invention is an image projection apparatus that projects a target image, which is a target to be output, onto a screen so that the target image is displayed in a rectangular shape on the screen. Correction means for generating a correction image by correction, projection means for selectively projecting the target image and the correction image onto the screen, and the screen when the target image is displayed on the screen Photographing means for obtaining a photographed image by photographing the image.

  Preferably, the image forming apparatus includes detection means for detecting a written image, which is an image written on the screen, by comparing the captured image with the target image.

  Alternatively, the correction unit calculates the corrected image based on the relationship between the position of the photographing unit and the position of the screen.

  Or it has the 2nd correction means which produces | generates a 2nd correction image by correct | amending the said written image based on the said positional relationship so that a shape may become a rectangle. Furthermore, it is preferable to have recording image generation means for generating a recording image by superimposing the second correction image on the target image.

  Or it has relation data generation means for generating data representing the relation by projecting a calibration pattern onto the screen and photographing the screen, and the correction means converts the correction image based on the data. The second correction unit generates the second corrected image based on the data. Or an angle unit that detects an angle of an optical axis of the photographing unit with respect to the screen, the correction unit generates the correction image based on the angle, and the second correction unit includes the angle Based on the above, the second corrected image is generated.

  Alternatively, the projection unit and the photographing unit are provided so as to be adjacent in the horizontal direction. Alternatively, the projection unit and the photographing unit use a common lens.

  According to the present invention, it is possible to acquire a screen image while suppressing the reduction in resolution as compared with the prior art.

It is a figure which shows the example of the whole structure of a conference system. It is a figure which shows the example of the external appearance of a 1st projector and a 2nd projector. It is a figure which shows the example of the hardware constitutions of a 1st projector. It is a figure for demonstrating the principle of distortion and correction | amendment of an image. It is a flowchart explaining the example of the flow of the data acquisition process for correction | amendment. It is a figure for demonstrating the process which acquires the data for correction | amendment. It is a flowchart explaining the example of the flow of a projection process. It is a figure for demonstrating the process which projects a target image. It is a figure which shows the example of the form of each image in the case of an imaging | photography process. It is a flowchart explaining the example of the flow of an imaging process. It is a flowchart explaining the example of the flow of the whole process in the case of a projection and imaging | photography. It is a figure which shows the example of the hardware constitutions of a 2nd projector.

  FIG. 1 is a diagram illustrating an example of the overall configuration of the conference system 4. FIG. 2 is a diagram showing an example of the external appearance of the first projector 1A and the second projector 1B.

  A conference system 4 shown in FIG. 1 is a system for displaying and photographing an image of a conference material. The conference system 4 includes a projector 1, a terminal device 2, a screen 3, and the like. The projector 1 and the terminal device 2 are connected by wire or wireless.

  An image output from the projector 1 is displayed on the screen 3. A dedicated board is used as the screen 3, but a white board may be used. Alternatively, the wall surface of the room may be used.

  Hereinafter, a case where a white board is used as the screen 3 will be described as an example. Therefore, the user can write a memo on the screen 3 with the water-based pen when the image is displayed.

  The projector 1 has a projection unit and projects an image on the screen 3. The projector 1 further includes a photographing unit, and can record a memo together with this image by photographing the screen 3.

  The first projector 1A that uses separate lenses as shown in FIG. 2A as the photographing lens of the photographing unit and the projection lens of the projection unit, and the second that uses a single lens as shown in FIG. 2B. Projector 1B.

  The terminal device 2 provides image data of an image to be projected on the screen 3 to the projector 1. As the terminal device 2, a personal computer, a smartphone, a tablet computer, or the like is used.

  Hereinafter, each of the first projector 1A and the second projector 1B will be sequentially described.

[First projector]
FIG. 3 is a diagram illustrating an example of a hardware configuration of the first projector 1A. FIG. 4 is a diagram for explaining the principle of image distortion and correction. FIG. 5 is a flowchart for explaining an example of the flow of correction data acquisition processing. FIG. 6 is a diagram for explaining processing for acquiring the correction data 61. FIG. 7 is a flowchart for explaining an example of the flow of projection processing. FIG. 8 is a diagram for explaining the process of projecting the target image 52a. FIG. 9 is a diagram illustrating an example of the form of each image at the time of shooting processing. FIG. 10 is a flowchart illustrating an example of the flow of shooting processing.

  As shown in FIG. 3, the projector 1A includes a control unit 101, an input / output interface 102, an operation panel 103, a projection unit 12, a first imaging unit 13, a second imaging unit 104, a correction amount calculation unit 105, a projection image. The processing unit 106, the storage unit 107, the captured image processing unit 108, the handwritten object detection unit 109, and the recorded image generation unit 110 are configured.

  The correction amount calculation unit 105, the projection image processing unit 106, the captured image processing unit 108, the handwriting object detection unit 109, and the recorded image generation unit 110 are an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Circuit. The functions will be sequentially described later.

  The control unit 101 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, and performs overall control of the projector 1A. The RAM stores computer programs such as a main program 1P0, a correction data acquisition program 1P1, a projection program 1P2, and an imaging program 1P3. These computer programs are loaded into the RAM as necessary and executed by the CPU.

  The input / output interface 102 is an interface for exchanging data with the terminal device 2. As the input / output interface 102, a standard interface such as D-SUB (D-Subminiature), DVI (Digital Visual Interface), HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), Ethernet, Bluetooth, or wireless LAN. Is used. HDMI, Ethernet, and Bluetooth are registered trademarks.

  The operation panel 103 has buttons for the user to input information and commands, and notifies the control unit 101 of the input content.

  The projection unit 12 is a transmissive liquid crystal type projection device, and includes a light source 121, a projection element 122, a projection lens 123, and the like, and projects an image input from the terminal device 2 onto the screen 3. Note that this image may be corrected by a projection image processing unit 106 described later.

  The light source 121 includes a lamp and a mirror, and irradiates the projection element 122 with light corresponding to an image.

  The projection element 122 is a liquid crystal panel. The amount of light emitted from the lamp is adjusted by the projection element 122 and proceeds to the projection lens 123.

  The projection lens 123 diverges the light from the projection element 122 and irradiates the screen 3. As a result, an image is displayed on the screen 3.

  The first photographing unit 13 is a photographing device for photographing the screen 3, and includes a photographing lens 131, a photographing element 132, and the like.

  The photographing lens 131 converges the light from the screen 3 on the photographing element 132. The imaging element 132 converts the received light into an electrical signal. Thereby, the screen 3 is photographed. As the imaging element 132, an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) is used.

  Note that the projection lens 123 and the photographing lens 131 are arranged adjacent to each other in the horizontal direction as shown in FIG. 2A, and have a certain parallax α.

  By the way, if the image is projected onto the screen 3 without correcting the shape of the image, the image may be distorted and displayed. This is because the optical axis of the projection lens 123 of the first projector 1A is not orthogonal to the projection surface of the screen 3 as shown in FIG.

  For example, as shown in FIG. 4B, when a rectangular image 50a is projected without correcting the shape, a trapezoidal image 50b is projected.

  Therefore, if the image 50a is corrected in advance to an image 50c as shown in FIG. 4C and projected, a rectangular image 50d is displayed.

  Using such a known principle, the control unit 101, the second photographing unit 104, the correction amount calculation unit 105, and the projection image processing unit 106 are as follows so that the image is not distorted and displayed on the screen 3. To process.

  When the user installs the first projector 1A in front of the screen 3 and performs a predetermined operation on the operation panel 103, the control unit 101 performs correction data (hereinafter referred to as correction data (hereinafter referred to as correction data) based on the correction data acquisition program 1P1). , Described as “correction data 61”) is executed according to the procedure shown in the flowchart of FIG.

  The control unit 101 controls the projection unit 12 to project the calibration pattern image 51a without correcting the shape (# 701 in FIG. 5). As shown in FIG. 6A, the calibration pattern image 51a is an image of a lattice pattern that is entirely rectangular and has a predetermined size.

  Then, the calibration pattern image 51 a is projected on the screen 3. However, as shown in FIG. 6B, the calibration pattern image 51a is often distorted. Hereinafter, the calibration pattern image 51a displayed on the screen 3 is referred to as a “calibration pattern image 51b”.

  The control unit 101 controls the second photographing unit 104 so as to photograph the calibration pattern image 51b (# 702). However, the optical axis of the lens of the second photographing unit 104 often does not intersect the screen 3 perpendicularly. Therefore, when the calibration pattern image 51b is photographed by the second photographing unit 104, a shape corresponding to the positional relationship among the projection unit 12, the screen 3, and the second photographing unit 104 as shown in FIG. Image data of the calibration pattern image 51c is obtained.

  The control unit 101 calculates the difference between the calibration pattern image 51a and the calibration pattern image 51c (# 703), and based on the calculated difference and the positional relationship between the projection unit 12 and the second imaging unit 104, and the like. Based on the known principle of keystone correction, the angle of the optical axis of the projection lens 123 with respect to the screen 3 (projection angle θ) is calculated (# 704). The projection angle θ thus obtained is stored in the storage unit 107 as correction data 61 (# 705).

  The control unit 101 performs a process shown in the flowchart of FIG. 7 on the basis of the projection program 1P2 for projecting an image to be projected onto the screen 3 for presentation or the like (hereinafter referred to as “target image 52a”). Run with.

  When the target image 52a as shown in FIG. 8A is input from the terminal device 2, the control unit 101 calculates how much each pixel of the target image 52a should be reduced (that is, the correction amount). The calculation is performed by the unit 105 (# 711 in FIG. 7). The correction amount calculation unit 105 calculates the correction amount by the keystone method based on the correction data 61 (projection angle θ).

  The control unit 101 controls the projection image processing unit 106 so as to correct the target image 52a (# 712). Then, the projection image processing unit 106 calculates a trapezoidal correction image 52b as shown in FIG. 8B by performing correction on the target image 52a based on the calculated correction amount.

  Then, the control unit 101 controls the projection unit 12 to project the corrected image 52b onto the screen 3 (# 713). Then, a rectangular target image 52c as shown in FIG. 8C is displayed.

  Thus, the projection image processing unit 106 corrects the corrected image 52b so that the rectangular target image 52c is displayed. That is, so-called reverse trapezoidal correction is performed. At this time, a known principle of keystone correction is used. Since the target image 52c is obtained by reducing the target image 52a, the resolution is lower than that of the target image 52a.

  The control unit 101 stores the image data of the target image 52a in the storage unit 107 (# 714).

  When the target image 52c is displayed on the screen 3, the user can write an object such as a memo on the screen 3 with an aqueous pen as shown in FIG. Then, the first photographing unit 13 can record this object together with the target image 52c. Hereinafter, the handwritten object is referred to as “handwritten object 53c”.

  The target image 52 c and the handwriting object 53 c are photographed by the first photographing unit 13. However, the captured target image 52c and handwritten object 53c are distorted. Therefore, the captured image processing unit 108 uses a known keystone correction principle to perform a perspective method on the captured target image 52c and handwritten object 53c based on the parallax α and the projection angle θ. ) To correct the trapezoidal distortion correction (hereinafter referred to as “trapezoidal correction”), and adjust to a rectangle.

  Incidentally, as shown in FIGS. 8A and 8C, the target image 52c has a lower resolution than the original image, that is, the target image 52a. Therefore, when the target image 52a includes a small character, it may be difficult for the user to recognize the character in the target image 52c. Moreover, if the target image 52c is captured by the first imaging unit 13 and the keystone is corrected, the target image 52c is further reduced and the resolution is lowered, so that it becomes difficult to recognize.

  Therefore, when a predetermined command is input via the operation panel 103, the control unit 101 performs processing for capturing the target image 52c and the handwriting object 53c based on the capturing program 1P3 in the flowchart illustrated in FIG. Follow the steps.

  The control unit 101 controls the projection unit 12 to project the target image 52a without performing keystone correction (reverse keystone correction) (that is, canceling the keystone correction) (# 721 in FIG. 10).

  Then, the target image 52a is distorted and displayed on the screen 3 as shown in FIG. Hereinafter, the displayed target image 52a is referred to as “target image 52d”. On the other hand, since the handwritten object 53c is drawn by the user with the water-based pen on the screen 3, it is in the same position as before releasing the trapezoidal correction and remains in the same shape.

  The control unit 101 controls the first photographing unit 13 to photograph the screen 3 (# 722). Then, the target image 52d and the handwriting object 53c are photographed as one image by the first photographing unit 13. Hereinafter, the captured image is referred to as “captured image 52e”.

  As shown in FIG. 9C, the captured image 52e is changed in position and shape as compared with the image on the screen 3 (the target image 52d and the handwriting object 53c). However, the shape of the portion corresponding to the target image 52d in the captured image 52e is substantially the same as the original image, that is, the target image 52a.

  The control unit 101 causes the handwriting object detection unit 109 to execute processing for detecting a portion corresponding to the handwriting object 53c (hereinafter referred to as “handwriting object 53e”) from the target image 52d (# 723). .

  Then, the handwriting object detection unit 109 detects the image included in the target image 52d but not included in the target image 52a as the handwriting object 53e by comparing the target image 52d and the target image 52a ( Extract.

  Further, as shown in FIG. 9D, the handwriting object detection unit 109 generates the handwriting object 53f by correcting the handwriting object 53e with a trapezoid based on the projection angle θ or the like (# 724). Specifically, when the handwriting object 53e is projected onto the screen 3, the image displayed on the screen 3 is calculated as the handwriting object 53f based on the projection angle θ or the like.

  Then, the control unit 101 causes the recording image generation unit 110 to execute a process of generating the recording image 52f as illustrated in FIG. 9E by superimposing (merging) the handwriting object 53f with the target image 52a (see FIG. 9E). # 725).

  For example, the recorded image generation unit 110 can determine which position in the target image 52a should be overlaid with the handwriting object 53f as follows. A point P2 in the photographed image 52e corresponding to an arbitrary point P1 (for example, the upper left corner point) of the handwriting object 53f is detected. A point P3 on the screen 3 corresponding to the point P2 is calculated based on the projection angle θ and the like. Then, a point P4 in the target image 52a corresponding to the point P3 is calculated based on the projection angle θ or the like. The recorded image generating unit 110 superimposes the handwriting object 53f on the target image 52a so that the point P1 coincides with the point P4.

  The recorded image 52f is transmitted as image data to an external device by the input / output interface 102 or stored in the storage unit 107 (# 726).

  FIG. 11 is a flowchart for explaining an example of the overall processing flow during projection and photographing.

  Next, the overall processing flow in the first projector 1A will be described with reference to the flowchart shown in FIG.

  When the first projector 1A is turned on, the main program 1P0 is activated. Then, every time an event occurs, the process of FIG. 11 is executed based on the main program 1P0.

  When a user installs an adjustment command in front of the screen 3 (Yes in # 11 of FIG. 11), the first projector 1A performs a process of generating correction data 61 (# 12). The procedure of this process is as described in FIG. When the correction data 61 is generated, the presentation mode is set.

  When the target image 52a is input from the terminal device 2 in the presentation mode (Yes in # 13), the first projector 1A performs a process of projecting the target image 52a onto the screen 3 (# 14). The procedure of this process is as described in FIG. In the case of a plurality of slides, first, the first slide is projected as the target image 52a.

  When the handwriting object 53c is written on the screen 3 and a shooting command is input by the user (Yes in # 15), the mode is switched to the shooting mode, and the first projector 1A performs a process of shooting the screen 3 (#). 16). The procedure of this process is as described above with reference to FIG. Therefore, since the keystone correction (reverse keystone correction) is canceled and the target image 52a is projected, the target image 52a is distorted and displayed. After shooting, the mode is switched to the presentation mode, the process returns to step # 14, and the keystone correction is performed again to project again.

  When a command to advance to the next page is input by the user (Yes in # 17), a process of projecting the next slide as the target image 52a is executed instead of the most recently displayed slide (# 14).

[Second projector 1B]
FIG. 12 is a diagram illustrating an example of a hardware configuration of the second projector 1B.

  As shown in FIG. 2A, the projection lens 123 and the photographing lens 131 of the first projector 1A have a parallax α in the horizontal direction. Therefore, if the line connecting the projection lens 123 and the photographing lens 131 is not parallel to the display surface of the screen 3, the shapes of the target image 52d and the target image 52a are different, and the two images may not be compared.

  On the other hand, in the second projector 1B, as shown in FIG. 2B, since the projection unit and the photographing unit share one lens, the front of the first projector 1A is parallel to the display surface of the screen 3. Even if not, the handwriting object 53e can be detected more reliably than the first projector 1A.

  Hereinafter, the mechanism of the second projector 1B will be described. The description overlapping with the first projector 1A is omitted.

  As shown in FIG. 12, the second projector 1B includes a control unit 151, an input / output interface 152, an operation panel 153, a projection unit 16, a first imaging unit 17, a shared optical system 18, a second imaging unit 154, The correction amount calculation unit 155, the projection image processing unit 156, the storage unit 157, the captured image processing unit 158, the handwriting object detection unit 159, the recorded image generation unit 150, and the like are included.

  The functions of the control unit 151, the input / output interface 152, and the operation panel 153 are the same as those of the control unit 101, the input / output interface 102, and the operation panel 103 of the first projector 1A.

  The projection unit 16 is a transmissive liquid crystal type projection device, like the projection unit 12 of the first projector 1A, and includes a light source 161, a projection element 162, and the like. However, the optical system is shared with the first photographing unit 17. The functions of the light source 161 and the projection element 162 are the same as the functions of the light source 121 and the projection element 122 of the first projector 1A.

  The first photographing unit 17 is a photographing device for photographing the screen 3 like the first photographing unit 13 of the first projector 1A, and includes a photographing element 172 and the like. However, the optical system is shared with the projection unit 16. The function of the imaging element 172 is the same as the function of the imaging element 132 of the first projector 1A.

  The shared optical system 18 is a shared optical system for the projection unit 16 and the first photographing unit 17, and includes mirrors 181, 182, 183, a half mirror 184, a lens 185, and the like.

  Light emitted from the light source 161 passes through the projection element 162. Then, the light is reflected by the mirrors 181 and 182, transmitted through the half mirror 184, diffused by the lens 185, and irradiated on the screen 3. As a result, an image is displayed on the screen 3.

  At the time of photographing, the light from the screen 3 is converged by the lens 185, passes through the half mirror 184, reflects off the mirror 183, and reaches the photographing element 132. The photographing element 172 converts the received light into an electrical signal. Thereby, the screen 3 is photographed.

  The transmittance of the half mirror 184 is 50%. The distance of the optical path from the projection element 162 to the lens 185 is equal to the consideration distance from the lens 185 to the imaging element 172.

  The procedure of processing for generating correction data, the processing of projecting a target image for presentation, and the processing of photographing the target image and handwriting object for recording are the procedures of the respective processes in the first projector 1A. The same is true (see FIGS. 5, 7, and 10). Therefore, in each process, an image similar to that when the first projector 1A is used is displayed on the screen 3, and a similar image is taken. The overall processing flow of the second projector 1B is the same as that when the first projector 1A is used, as described in FIG. In other words, the correction amount calculation unit 155, the projection image processing unit 156, the storage unit 157, the captured image processing unit 158, the handwriting object detection unit 159, and the recorded image generation unit 150 are the correction amount calculation unit 105, the projection image processing unit, respectively. Functions in the same manner as the unit 106, the storage unit 107, the captured image processing unit 108, the handwritten object detection unit 109, and the recorded image generation unit 110. However, unlike the first projector 1A, since the projection unit 16 and the first photographing unit 17 have no parallax, the parallax is not considered when performing the keystone correction.

  According to the present embodiment, it is possible to take an image displayed on the screen 3 while suppressing reduction in resolution as compared with the conventional case. Furthermore, a handwritten object can be detected from the photographed image and merged with the original image and recorded.

  In the present embodiment, the calibration pattern image is captured by the second imaging unit 104 or 154 in order to generate the correction data 61, but may be captured by the first imaging unit 13 or 17. Alternatively, if the positional relationship between the screen 3 and the first projector 1A or 1B is fixed and the distance between the two can be grasped in advance, the projection angle θ may be measured by an inclination angle sensor.

  Alternatively, the user may manually adjust the calibration pattern image so as to be displayed as a rectangle while viewing the calibration pattern image displayed on the screen 3. Then, the projector 1 may store and use data indicating the adjustment amount as the correction data 61.

  The range of projection by the projection unit 12 or 16 may be the same as the range of imaging by the first imaging unit 13 or 17, or the imaging range is widened so that the projection range is included in the imaging range. May be.

  In the present embodiment, the correction amount calculation units 105 and 155, the projection image processing units 106 and 156, the photographed image processing units 108 and 158, the handwriting object detection units 109 and 159, and the recorded image generation units 110 and 150 are connected to the ASIC or FPGA. However, all or some of them may be software modules. That is, each unit may be realized by preparing a computer program describing the processing of each unit and causing the control unit 101 to execute the computer program.

  In the present embodiment, the case where only one handwriting object 53c is written on the screen 3 has been described as an example. However, the present invention can also be applied to the case where a plurality of handwriting objects 53c are written. In this case, keystone correction may be performed for each handwriting object 53c and merged into the target image 52a.

  Not only the projection angle θ of the first projector 1A but also the shooting angle γ is calculated, and each image (calibration pattern image, target image, handwriting object) is calculated based on the projection angle θ, the shooting angle γ, and the parallax α. Keystone correction may be performed.

  In addition, the configuration, processing contents, processing order, images, and the like of the entire conference system 4, the first projector 1A, and the second projector 1B, or each unit can be appropriately changed in accordance with the spirit of the present invention.

1 Projector 12 Projector (Projector)
13 First imaging unit (imaging means)
101 control unit (related data generation means)
106 Projected image processing unit (correction means)
109 Handwritten object detection unit (detection means, second correction means)
110 Recorded image generating unit (recording image generating means)
185 Lens 52a Target image 52b Correction image 52d Target image (captured image)
52f Recorded image (recording image)
Handwritten object 53e (written image)

Claims (11)

An image projection device that projects a target image to be output onto a screen,
Correcting means for generating a corrected image by correcting the target image so that the shape is displayed in a rectangular shape on the screen;
Projection means for selectively projecting the target image and the corrected image onto the screen;
Photographing means for obtaining a photographed image by photographing the screen when the target image is displayed on the screen;
An image processing apparatus comprising: Detecting means for detecting a written image, which is an image written on the screen, by comparing the captured image with the target image;
The image processing apparatus according to claim 1. The correction means calculates the correction image based on the relationship between the position of the photographing means and the position of the screen;
The image processing apparatus according to claim 1. Second correction means for generating a second corrected image by correcting the written image based on the positional relationship so that the shape is rectangular.
The image processing apparatus according to claim 3. Recording image generating means for generating a recording image by superimposing the second correction image on the target image,
The image processing apparatus according to claim 4. Relationship data generating means for generating data representing the relationship by projecting a calibration pattern onto the screen and photographing the screen;
The correction means generates the corrected image based on the data,
The second correction unit generates the second correction image based on the data.
The image processing apparatus according to claim 4 or 5. Angle means for detecting the angle of the optical axis of the photographing means with respect to the screen,
The correction means generates the corrected image based on the angle,
The second correction unit generates the second correction image based on the angle.
The image processing apparatus according to claim 4 or 5. The projection means and the photographing means are provided so as to be adjacent in the horizontal direction.
The image processing apparatus according to claim 1. The projection unit and the photographing unit use a common lens.
The image processing apparatus according to claim 1. A projection and photographing method in which a target image that is a target to be output is projected by a projection unit and photographed by a photographing unit,
By correcting the target image so that the shape is displayed in a rectangular shape on the screen, a corrected image is generated,
Selectively projecting the target image and the corrected image onto the screen by the projection means;
When the target image is displayed on the screen, a captured image is acquired by capturing the screen with a capturing unit.
Projection and photographing method characterized by the above. A computer program used in a computer for controlling a projection unit to project a target image to be output onto a screen and controlling the photographing unit to shoot the screen,
In the computer,
By correcting the target image so that the shape is displayed in a rectangular shape on the screen, a process for generating a corrected image is executed,
When the first mode is set, the control unit controls the projection unit so that the corrected image is projected onto the screen.
When a second mode different from the first mode is set, a process of controlling the projection unit is executed so that the target image is projected onto the screen,
Executing a process of acquiring a captured image by controlling the imaging unit so that the screen is captured when the target image is displayed on the screen;
A computer program characterized by the above.

Images