JP2000200149A - Display system, pointing control method and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an interactive presentation environment just by adding an image pickup device by fetching the image of a correction pattern projected from a projector onto a screen to a computer and executing an image processing to the fetched image. SOLUTION: This system is provided with a means for projecting and displaying the output image of the computer onto the screen, a fetch means for fetching a projected image displayed on the screen to the computer by the image pickup device and the means for fetching a pattern for correction projected on the screen to the computer by the fetch means and performing correction work by using the fetched image. The system is provided with a computer main body 20, the image pickup device 14, the projector 16 and the screen 10, etc. Then, by an image processing routine operated inside the computer main body 20 to image data transferred to a RAM, the image processing for performing a tip detection processing by a pointing address calculation program is executed.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a computer,
Projector, display system using screen and imaging device, and pointing control method, and
The present invention relates to a recording medium on which a computer program for performing setting of capture conditions of an imaging device, image processing, and calculation of a pointing address for that purpose is recorded.

[0002]

2. Description of the Related Art Generally, when a projector is connected to a computer and data created by the computer is projected on a whiteboard or a screen using the projector and a presentation is made, a projected image is projected using a stick or the like. When the computer moves the mouse pointer to the pointed position, or the character information entered by handwriting on the whiteboard, or the computer captures the information indicating the gist of the projection information from the projector, such as underlining, the area around the screen This is done by installing some kind of optical system on the screen, or by using special hardware such as a screen having a built-in resistive film or inductance on a screen or a white board.

[0003]

However, in the above-mentioned conventional technique, pointing and handwriting input information can be captured. For example, an optical system is provided near the pen tip of a pen used for handwriting input. Since a pen printed with a pattern that can determine the type is used, or a screen or whiteboard with a built-in resistive film or inductance is used, a normal screen, whiteboard, or a normal whiteboard writing pen could not be used . Therefore, in the prior art,
It was virtually impossible to make an interactive presentation using a normal screen or a normal whiteboard.

An object of the present invention is to solve the above-mentioned problems in the prior art, and to realize an interactive presentation environment simply by adding an imaging device to a conventional screen, whiteboard, computer, and projector. And

[0005]

According to a first aspect of the present invention, there is provided a display device for projecting an image output from a computer onto a screen and displaying the image on the screen, and displaying a projected image displayed on the screen. Capturing means for capturing into the computer by an imaging device; and calibrating means for capturing the calibration pattern projected on the screen by the capturing means into the computer, and performing calibration using the captured image. It is characterized by.

According to this configuration, the image of the calibration pattern projected on the screen from the projector is taken into the computer, and the taken image is subjected to image processing such as binarization and pattern extraction, thereby automatically performing calibration work. Is performed.

In the present invention, when the calibration means binarizes the captured image, the calibration means divides the image into a plurality of regions, sets a threshold value for each region, and performs binarization. It is characterized by including.

According to this configuration, when the captured image is binarized, the captured image is divided into a plurality of regions and binarized rather than performing binarization using the same threshold value for the entire image. By doing so, the displayed pattern can be accurately binarized, and correct calibration can be performed.

Further, the calibration means includes a process for extracting a projection area of the display means from the captured image.

[0010] According to this configuration, the work area is reduced by extracting the area where the work is actually performed from the entire capture, and the processing can be speeded up.

Further, the image pickup apparatus is characterized in that after completion of the calibration work, the focus, aperture and gain of the image pickup apparatus are fixed.

According to this configuration, the position of the projected image in the captured image after the calibration is completed can be obtained stably. For example, if the image pickup apparatus has an automatic focus control function and an automatic aperture control function, after the calibration operation is completed, if these functions are enabled and continued to be used, if a person or any object passes in front of the screen, The focus and aperture of the imaging device are controlled by these functions. As a result, a problem occurs in that the image on the screen is blurred, and a projected image cannot be stably obtained. According to the present invention, as described above,
The position of the projection image in the captured image after the calibration is completed can be obtained stably.

Further, according to the present invention, the calibration means includes:
When the calibration pattern is extracted from the captured image, a region is divided into a plurality of regions, and the calibration pattern is extracted for each region.

According to this configuration, the method of extracting the calibration pattern from the calibration image taken into the computer is simplified, and the calibration operation can be performed quickly and accurately.

According to a second aspect of the present invention, there is provided a display means for projecting an output image of a computer on a screen by using a projector and displaying the image, and an imaging device for projecting an image from the projector displayed on the screen by the imaging device. Calculating a difference image between frames for the image captured by the capturing means,
Further, an image generating means for binarizing the image to generate a binarized difference image, and if there is a point indicated by some instruction means in the binarized difference image, using the binarized difference image And a detecting means for detecting the location.

According to this configuration, an image pointed by a human finger, a stick, a laser pointer, or the like is captured by the image pickup device with respect to the projected image on the screen, so that the information on the screen is converted into image information by the computer. The captured image information is processed by a computer, so that the pointed location can be specified. Therefore, it is possible to specify a portion pointed by a human finger, a stick, a laser pointer, or the like with respect to a projection image projected from the projector onto the screen without incorporating a resistive film or an inductance in the screen.

Further, according to the second aspect of the present invention, when the image pickup device captures an image, the image pickup device fixes the focus, the aperture, and the gain of the image pickup device.

According to this configuration, since the focus and the aperture are always in a state for capturing an image on the screen, an image on the screen can be stably obtained.

Further, according to the present invention, the image generating means includes:
When binarizing the difference image, the method includes a process of setting a threshold value for each pixel from the two original images used for calculating the difference image and binarizing the image.

According to this configuration, normally, when binarization is performed on a differential image using one threshold value, a stable binary differential image cannot be obtained. A difference image can be obtained.

Further, according to the present invention, the detecting means sets a scan area on a rectangle, compares the number of points near the inside of the four sides of the set scan area, and determines the most The side with the largest number is set as the scan start position, scanning is performed sequentially from the scan start position toward the inside of the area, the distance between the scanned line and the point existing on the previously scanned line is compared, and it is determined that it is a valid point. Extracting only the detected points and detecting the tip, comparing the result of the detected tip with the detection result detected at the time of the preceding tip detection, and determining the detection result. .

According to this configuration, since unnecessary point information is removed, the detection accuracy and the detection speed of the leading end are improved.

Further, according to the present invention, when the tip is detected at the time of the previous tip detection, the detecting means sets the tip detection area at the time of the tip detection only to the vicinity of the location detected at the time of the previous tip detection. It is characterized by including a treatment to perform.

According to this configuration, normally, when an object on the screen is pointed by some pointing means, the pointed position is rarely greatly moved.
When the leading edge is detected, by limiting the scan area for the next leading edge detection to only around the detected leading edge, the leading edge detection accuracy and the detection speed are improved.

Further, in the present invention, the detecting means has a processing method after the detection of the front end which differs depending on whether the front end is detected or not, and whether the front end is detected. The determination is characterized by including a process performed based on the result of the previous tip detection.

According to this configuration, the entire projected image area of the projector is always set as the scan area when the leading end is not detected, and when the leading end is detected, the leading end detection fails at some point. Also, if the tip is detected as a result of the previous detection, a scan area is set from the result, or if detection failures occur continuously, it is determined that the tip does not exist on the screen, and By shifting to a non-detected state and setting an appropriate tip detection area depending on the state, such as setting a scan area on the entire screen, the tip detection accuracy is improved.

A third invention is a pointing control method in a display system including a computer, a projector, a screen, and an imaging device,
(A) a step of setting an image capturing condition of the image capturing apparatus; and (b) a step of performing image processing on an image captured from the image capturing apparatus that has been set in the step (a).
(C) calculating a pointing address for the image subjected to the image processing in the step (b).

According to the third aspect of the invention, similarly to the first and second aspects of the invention, a calibration image projected from a projector onto a screen is taken into a computer, and the taken image is subjected to binarization and pattern extraction. By performing the image processing, the calibration work is automatically performed, and a portion pointed by some instruction means can be specified with respect to the projection image projected from the projector onto the screen.

Further, a fourth invention is a projector,
A recording medium recording a computer program for pointing control in a display system using a computer, a screen, and an imaging device, comprising:
Setting an image capture condition of the imaging device;
(B) performing image processing on an image captured from the imaging device whose settings have been completed in step (a);
And (c) a step of calculating a pointing address for the image that has been subjected to the image processing in the step (b).

According to the fourth invention, as in the first invention and the second invention, a calibration image projected from a projector onto a screen is taken into a computer, and the taken image is subjected to binarization and pattern extraction. By performing the image processing, the calibration work is automatically performed, and a portion pointed by some instruction means can be specified with respect to the projection image projected from the projector onto the screen.

[0031]

DESCRIPTION OF THE PREFERRED EMBODIMENTS In order to further clarify the configuration and operation of the present invention described above, embodiments of the present invention will be described below based on examples.

FIG. 1 is an external view of a system to which one embodiment of the present invention is applied. This system comprises a computer body 20 and a CRT display 18 as a peripheral device.
And a keyboard 22, a mouse 24, an imaging device 14, a liquid crystal projector 16, and a screen 10. The computer is equipped with a floppy disk drive 28 for reading the contents of a floppy disk.

FIG. 2 is a block diagram showing a schematic configuration of hardware of the computer system. As shown in the figure, the computer main body 20 comprises a RAM 42, a ROM 44, and a mouse interface 5 interconnected by a bus around a CPU 40 as a central processing unit.
6, keyboard interface 58, FDC60, H
DC 62, CRTC 64, IEEE 1394 interface 66, video capture device 46, parallel interface 48, USB interface 50,
A network interface 52 and a serial interface 54 are provided.

The ROM 44 is a read-only memory for storing various built-in programs and the like. RAM4
Reference numeral 2 denotes a memory that can read and write various data. The mouse interface 56 is an interface that controls data exchange with the mouse 24. F
The DC 60 is a floppy disk controller that controls the floppy disk drive 28. HDC62
Is a hard disk controller that controls the hard disk drive 30 (HDD). The CRTC is a controller that controls a CRT that displays necessary data and the like and a signal output to a liquid crystal projector. IEEE
The 1394 interface 66 is an interface that controls output signals to the CRT 18 and the liquid crystal projector 16 and input signals from the imaging device 14. The video capture device 46 is a device that transfers image data sent from the imaging device 14 to a predetermined memory area of the RAM 42 inside the computer main body 20. Parallel interface 48 and USB
The interface 50 is an interface that controls input signals from the imaging device 14 and controls data output to a printer. The network interface 52 is connected to the external network 34 and can be connected to the external computer 36. The serial interface 54 is connected to the modem 32, and the computer main body 20 communicates through the modem 32
It is connected to an external network 34 and can be connected to an external computer 36.

In this computer system, the operating system is stored in the HDD 30,
When the power of the computer main body 20 is turned on in accordance with the loader written in the boot block of D30, the RAM
42 is loaded into a predetermined area. In addition, the imaging device capturing condition setting program and the pointing program are stored in the floppy disk in advance, and are installed from the floppy disk drive 28 to the computer main body 20 by activating a predetermined installation program. The installed program is stored in the HDD 30 and is stored in the computer main unit 2.
When the power is turned on, the memory is loaded into a predetermined area of the RAM together with the operating system.

The various constituent elements of the present invention are realized by the CPU 40 executing the imaging device fetching condition setting program, the image processing program, and the pointing program. Note that these software programs are stored on the floppy disk 26 as described above, but may be replaced by another portable recording medium (such as a CD-ROM, a magneto-optical disk, an IC card). (A portable recording medium). Also,
The above-described software program can also be obtained by downloading program data provided via the external network 34 from an external computer 36 connected to the external network 34 and transferring the program data to the RAM 42 or the HDD 30.

FIG. 3 is a configuration diagram showing a schematic configuration of the hardware of the image pickup apparatus 14. As illustrated, the imaging device 14 includes a focus control device 100, an imaging element 102, an aperture control device 104, an aperture 106, a gain control device 108, and an amplifier 110.

In the focus control device 100, switching between automatic control, manual control, and fixed focus can be performed by a switch of the imaging device main body, but can also be switched by a command from the computer main body 20. Similarly, the aperture control device 1
In the case of the camera 04 as well, switching between automatic control, manual control, and fixed aperture can be performed by a switch of the imaging apparatus main body. Similarly, the gain control device 110 also performs automatic control,
The switching between the manual control and the fixed gain can be performed by a switch of the imaging apparatus main body, but can also be performed by a command from the computer main body 20.

Further, the imaging device 14 is
Of the video capture device 4 built in the computer body 20 depending on the form of the connection interface of
6, the parallel interface 48, the USB interface 50, and the IEEE 1394 interface 66.

A description will be given of how the computer system having the above-described hardware configuration performs from imaging to calculation of a pointing address.

FIG. 4 is a diagram for explaining the state of processing from capturing an image from the imaging device 14, performing image processing and pointing processing in the computer main body 20, and projecting an image by the liquid crystal projector 16. .

As shown in the figure, the image data sent from the image sensor 14 is converted into a video signal by the video capture driver 120 for controlling the video capture device 46 operating inside the computer shown in FIG. The data is transferred to a predetermined area of the RAM 42 shown. This RA
An image processing routine 122 operating inside the computer main body 20 for the image data transferred to M42
The pointing address calculation program 12
The image processing for performing the leading edge detection processing by step 4 is performed. The image data that has been subjected to the image processing by the image processing routine 122 is sent to the pointing address calculation program 124, and the coordinates indicated by the finger or the stick are specified. The coordinate data calculated and output by the pointing address calculation program 124 is passed to a mouse driver 126 operating inside the computer main body 20, and the mouse pointer on the screen is rewritten. Subsequent processing is performed by the application 128
The data is finally sent from the video driver 130 to the liquid crystal projector 18, the video is projected from the liquid crystal projector 18 onto the screen 12, and the result of the pointing is displayed.

FIG. 5 is a diagram for explaining the state of processing from capturing an image from the imaging device 14 to projecting an image by the liquid crystal projector 16. The processing mode shown in FIG. 4 can also be realized by the processing mode shown in FIG. As shown in the figure, image data sent from the image pickup device 14 is transferred to a predetermined area of the RAM 42 inside the computer main body 20 by a video capture device 46 built in the computer main body. The image processing and the pointing processing chip 150 perform image processing and leading end detection processing on the transferred image data, and output coordinate data. The coordinate data calculated and output by the image processing and pointing processing chip 150 is passed to a mouse driver 126 operating inside the computer main body 20, and the mouse pointer on the screen is rewritten. Subsequent processing is entrusted to the application 128 side, and finally data is sent from the video driver 130 to the liquid crystal projector 18, and the data is sent from the liquid crystal projector 18 to the screen 12
Is projected on the screen, and the result of the pointing is displayed.

FIG. 6 is a flowchart showing a flow of a series of processing from the screen setting executed by the CPU 40 to the execution of the pointing processing. As shown, CP
When the process is started, the U40 first starts a process of a screen setting subroutine (step S10). At the time of the pointing processing performed later, information at the time of performing the preceding pointing processing is required, so it is necessary to initialize information required at the time of the pointing processing before performing the pointing processing ( Step S12). After the information has been initialized, the pointing process subroutine is executed (step S14), and the process ends.

FIG. 7 is a flowchart of the screen setting subroutine shown in FIG. As shown, when the processing is started, the CPU 40 executes a white pattern display subroutine, and the pattern pic1 shown in FIG.
Is displayed (step S40).

Subsequently, an image capturing subroutine is executed and transferred to a predetermined area of the RAM 42 inside the computer main body 20 (step S42). Subsequently, a projector projection image frame extraction subroutine is executed to detect a region of the liquid crystal projector projection image in the captured image (step S44). When the process returns from the projector projection image frame extraction subroutine, information indicating whether or not the process has been normally completed is also sent from the projector projection image frame extraction subroutine. Therefore, the information is determined (step S46). If the processing has not been correctly performed, the error processing routine is called to perform error processing (step S56), and after the error processing is completed, the process returns to the end and ends.

On the other hand, if it is determined that the projector projected image frame extraction subroutine has been normally processed, the position calibration pattern display subroutine is called and the pattern pic2 shown in FIG. 11 is displayed. (Step S48). Subsequently, an image capturing subroutine is executed, and the RAM 4 in the computer main body 20 is read.
2 is transferred to a predetermined area (step S50).

Subsequently, a calibration dot coordinate extraction subroutine is executed to calculate the center coordinates of four calibration dots in the image captured in step S50 (step S52). When the process returns from the calibration dot coordinate extraction subroutine, information as to whether or not the process has been completed normally is also sent from the calibration dot coordinate extraction subroutine, and the information is determined (step S5).
4) If the processing has not been performed correctly, the error processing routine is called to perform error processing (step S56), and after the error processing is completed, the process returns to the end and ends the processing. On the other hand, when it is determined that the calibration dot coordinate extraction subroutine has been normally performed, the information of the liquid crystal projector projection image area in the captured image and the coordinates of the calibration dot in the captured image are output ( In step S58), the process ends with returning to the end.

FIG. 8 is an explanatory diagram showing a pattern displayed when the white pattern pic1 display subroutine shown in FIG. 7 is executed.

FIG. 9 is a flowchart of the image capturing subroutine shown in FIGS. 7 and 14. As shown, when the processing is started, the CPU 40 sets to transfer the specified image format (step S60), and then sets to transfer data to the specified transfer destination address (step S60). S6
2) Subsequently, transfer of the image data is started in order from the designated address (step S64). When the transfer is completed, the process exits from the return and ends the process.

FIG. 10 is a flowchart of the projector image frame extraction subroutine shown in FIG. First, the captured image is binarized in order to separate the projection area and the peripheral area of the liquid crystal projector from the captured image. As shown, when the processing is started, the CPU 40 calls a designated area binarization subroutine,
For the image data that has already been transferred to the RAM 42 of the computer body 20, a binarization process is performed on the upper third area (upper area) of the captured image (step S
70) Then, a binarization process is performed on a region (middle region) of the middle of the captured image (step S7).
2) Then, a binarization process is performed on a lower third region (lower region) of the captured image. When the process returns from the designated area binarization subroutine, information as to whether or not the designated area has been properly binarized is also sent. Therefore, binarization is normally performed based on this information. It is determined whether or not it has occurred (step S78). If the binarization processing has not been performed normally, an error processing routine is called to perform error processing (step S82), and after the error processing ends, the process returns to the end and ends.

On the other hand, if it is determined that the processing has been performed normally, it is calculated how much the liquid crystal projector projection image area occupies the entire captured image (step S80).

Next, it is determined whether or not the projected image area of the liquid crystal projector is too small for all the captured images (step S84). An arbitrary ratio having a ratio of the liquid crystal projector projection image area to the entire captured image (the arbitrary ratio is a value for determining whether the liquid crystal projector projection image region is too small for the entire captured image. In the embodiment, the number is set to 45%, but the present invention is not limited to this number.) When it is determined that the projection area of the liquid crystal projector is small, the error processing routine is executed. The error processing is performed by calling (step S98). After the error processing is completed, the process returns to the return and ends.

On the other hand, if it is determined that the ratio of the projected image area of the liquid crystal projector to all the captured images is 45% or more, it is determined that the projected area of the liquid crystal projector is large enough to perform the coordinate calculation. An approximate value of the vertical and horizontal resolution of the liquid crystal projector projection image area in the captured image is calculated from the resolution of the computer main unit 20, its aspect ratio, and the ratio of the liquid crystal projector projection image area to the entire captured image (step S8).
8) The vertical and horizontal resolutions are set as determination values (step S90), and the line is scanned from the end of the captured image line by line from the end of the captured image in the order of left, right, up and down, and exists on the first scanned line. A line whose number of dots is sufficiently close to the determination value is set as a frame of the liquid crystal projector projected image, and its x coordinate and y coordinate are output (steps S90, S92, S94, S96). After these processes are completed, the process returns to the end and the process ends.

FIG. 11 is an explanatory diagram showing a pattern displayed when the position calibration pattern display subroutine shown in FIG. 7 is executed.

FIG. 12 is a flowchart of the calibration dot coordinate extraction subroutine shown in FIG. As shown in the figure, when the processing is started, the CPU 40 adds one third of the image data already transferred to the RAM 42 of the computer body 20 to the upper part of the captured image.
(Step S70), and then a one-third (middle) region of the captured image is processed (step S70).
72) Then, a binarization process is performed on the lower third region (lower region) of the captured image.

Next, a projector projection image frame obtained when the CPU 40 executes the projector projection image frame extraction subroutine is set as an area for removing an isolated point (step S116), and an isolated point is set for the set area. A removal process is performed (step S118). Subsequently, the area from which the isolated points have been removed is divided into four areas of upper left, upper right, lower left, and lower right (step S120), and the number of black points and their coordinates are recorded in each area (step S122).

Subsequently, from the number of black points in each area,
It is determined whether or not the position calibration dots have been correctly extracted (step S124). If the number of black dots is clearly large, it is determined that the projection condition and the capture condition are bad, and that the calibration dot has not been correctly detected, and the process proceeds to an error processing subroutine (step S128). To end the processing.

On the other hand, when it is determined that the number of black points is appropriate, the average value of the x coordinate and the average value of the y coordinate of the detected coordinates are calculated in each region, and the coordinates are calculated. The coordinates are output as the coordinates of the calibration dots on the captured image (step S126), and the process exits from the return.

FIG. 13 is a flowchart of the designated area binarization subroutine. As shown, when the processing is started, the CPU 40 sets an image area designated when this subroutine is called as a work area (step S350). Subsequently, a histogram of the work area is calculated (step S352). Subsequently, the differential value of the obtained histogram is calculated (step S354), and the area ratio of the cumulative value of the histogram from the lowest luminance value to the highest luminance value from the number of pixels in the histogram and the work area is calculated (step S356).

Subsequently, it is determined whether or not there is an area that meets the conditions (step S360). If there is no area that meets the conditions, the screen or whiteboard 10 and the liquid crystal projector It is determined that the contrast ratio of the projection image 12 is low and the threshold value could not be set, and the error processing subroutine is called (step S366).

On the other hand, if there is an area that satisfies the condition, it is determined that the threshold value has been correctly set, and step S35 is performed.
8 is determined (step S362), and the work area is binarized from the obtained threshold (step S36).
4). After the binarization process is completed, the process returns to the end and the process ends.

FIG. 14 is a flowchart of the pointing processing subroutine shown in FIG.
This routine is repeatedly executed at predetermined time intervals. As shown, when the processing is started, the CPU 40 first captures an image (step S20), and determines whether the captured background includes a still image or a moving image (step S2).
2).

If it is determined that the background is a still image,
Thereafter, the image processing routine for a still image is executed to generate an image suitable for pointing (step S24). Subsequently, a coordinate output subroutine is executed, and the coordinates of the position pointed by some instruction means are output (step S2).
8), the process ends with return.

On the other hand, if it is determined that the background is a moving image or that a part of the background contains a moving image, a moving image processing routine is executed to remove an image unnecessary for pointing (step S26). Subsequently, a coordinate output subroutine is executed to output the coordinates of the location pointed by some instruction means (step S28), and after the processing is completed, the process returns to the end and the processing ends.

FIG. 15 is a flowchart of the still image processing subroutine shown in FIG.
As shown, when the processing is started, the CPU 40 calls a difference image calculation subroutine and calculates a difference image between frames (step S140).

Subsequently, a difference image binarization subroutine is called to binarize the difference image (step S142). After the processing in the differential image binarization subroutine is completed, image data obtained as a result of the processing is output (step S14).
4), the process ends with return.

FIG. 16 is a flowchart of a moving image processing subroutine shown in FIG. As shown, when the processing is started, the CPU 40 calls a difference image calculation subroutine and calculates a difference image between frames (step S240).

Subsequently, a difference image binarization subroutine is called to binarize the difference image (step S242). After the processing in the difference image binarization subroutine is completed, a moving image area existing in the difference image is detected (step S24).
4) Yes. When the image difference processing is performed on an image having a moving image area, an unnecessary difference image appears in the moving image area. Therefore, it is necessary to remove the unnecessary difference image. In order to perform the processing, a moving image area unnecessary point removal subroutine is called to remove unnecessary difference images in the moving image area (step S
246). The resulting image is output (step S
248), the process ends with return.

FIG. 17 is a flowchart of the difference image binarization subroutine shown in FIGS. 15 and 16. In the present embodiment, the case where the image has 256 gray scales will be described. However, the present invention is not limited to color or gray scale and the number of gray scales. As shown, CP
When the processing is started, U40 selects the value of the pixel having the larger luminance value among the pixels of the two captured images used when generating the difference image (step S380), and sets the selected value to 2 Is set as the threshold value of each pixel (step S382).

Subsequently, the threshold value set for each pixel is 25 or more (this threshold value is a value for determining whether or not a certain pixel is originally a dark display area, and in this embodiment, 25, but is not limited to this number in the present invention) (step S384).

If it is determined that the threshold value is 25 or more, the threshold value remains unchanged.

On the other hand, if it is determined that the threshold value is 25 or less, the threshold value is set to 25. (Step S38
6) Thereafter, each pixel is binarized using the set threshold value (step S388), and the process exits from the return.

FIG. 18 is a flowchart of the tip coordinate output subroutine shown in FIG. As shown in the figure, when the processing is started, the CPU 40 calls and executes a tip detection subroutine (step S16).
0).

Subsequently, referring to the previous tip detection result,
It is determined whether the current state is the leading end detection state or the standby state (step S161). The leading edge detection state is a state in which the leading edge has been detected at the time of the previous scan, and the waiting state is a state in which the leading edge has not yet been detected since the start of leading edge detection, or the leading edge has been detected. However, many failures in tip detection continued,
This indicates a state in which the leading end is determined to be out of the area and the state has returned to the initial state.

If it is determined in step S161 that the leading edge is detected, it is determined whether or not the leading edge has been detected as a result of this leading edge detection (step S162).

If it is determined in step S162 that the leading end has been detected, then the distance between the previous coordinate detection result and the current coordinate detection result is calculated, and this distance is determined (step S163). The calculated distance is within an arbitrary number of pixels (this arbitrary number of pixels is a value for determining whether the distance between the previous coordinate detection result and the current coordinate detection result is sufficiently large and invalid. In the present embodiment, the number is 30 pixels, but the present invention is not limited to this number.) If it is determined that the number of pixels is 30 pixels, it is determined that the current detection result is valid. The result is output as coordinates (step S164), the number of detection failures is set to 0, and the periphery of the coordinates detected this time is set as the scan area for the next leading edge detection (step S16).
5) The tip detection state is held (step S167).

On the other hand, if the tip is not detected in step S162, or if it is determined in step S163 that the distance between the previously detected coordinates and the currently detected coordinates is 30 pixels or more, the current address detection is performed. As the result, the same coordinates as the previous scan result are output (step S17).
0), 1 is added to the number of consecutive failures, and the area around the coordinates detected this time is set as the scan area for the next leading edge detection (step S172).

Subsequently, the number of detection failures is determined (step S174). The number of detection failures is equal to or less than an arbitrary number (this arbitrary number is a value for determining whether or not the currently output scan result is appropriate in the case where the failure of the tip detection occurs continuously. In the present embodiment, the number is set to 20, but the number is not limited to this number in the present invention. When it is determined that the number is 20), the state of detecting the leading end is maintained (step S1).
67).

On the other hand, if it is determined in step S174 that the number of detection failures is more than 20, the state is shifted from the leading edge detection state to the standby state (step S176), the number of detection failures is reduced to zero, and the next leading edge detection is performed. The entire projected image frame of the liquid crystal projector is set as the scan area at this time (step 178).

On the other hand, when it is determined in step S161 that the apparatus is in the standby state, it is determined whether or not the leading end is detected as a result of the present leading end detection (step S180).

If it is determined in step S180 that the leading end has been detected, it is determined that the current detection result is valid, and the current detection result is output as coordinates (step S18).
2) The number of detection failures is set to 0, the entire liquid crystal projector projection image frame is set as a scan area for the next leading edge detection (step S184), and the state shifts to the leading edge detection state (step S186).

On the other hand, if it is determined in step S180 that the leading end has not been detected, the current address detection result outputs the same coordinates as the previous scan result (step S180).
190), the number of detection failures is set to 0, and the entire liquid crystal projector projection image frame is set as a scan area for the next leading edge detection (step 192), and the standby state is maintained (step S194).

Since the coordinates output in step S164, step S170, step S182 or step S190 are the coordinates in the captured image, it is necessary to convert the coordinates in the captured image into the coordinates in the computer main body 20. Is performed to convert the coordinates (step S200), the converted coordinates are output (step S202), and the process exits from the return.

FIG. 19 is a flowchart of the leading edge detection subroutine shown in FIG. As shown, when the processing is started, the CPU 40 initializes information for scanning a designated area (step S210). Subsequently, the number and coordinates of points existing on the upper, lower, left, and right end lines inside the designated area are recorded (step S212).

Subsequently, it is determined whether or not a point exists as a result of the scanning (step S214).

If the point does not exist, the difference image does not exist, that is, it is determined that the object for which the leading edge detection does not exist in the scan area, or that the object exists but is stationary. It is determined that scanning is not possible (S2
36), return to the end of the process.

On the other hand, if there are points on the outer circumference, the number of points existing on each end line is compared, and the end line having the largest number of points is set as the scan start line (step S216). Subsequently, the next adjacent line is scanned, and the number of existing points and the coordinates of the points are recorded (step S218).

Subsequently, for all the points of the line scanned this time, the distances from all the valid dots existing on the line from the line immediately before the line currently being scanned to several lines before are calculated, The distance is equal to or less than an arbitrary number of pixels (the arbitrary number of pixels is a value for determining whether or not a dot present on the line currently being scanned is a valid dot, and is set to 5 in this embodiment. However, the present invention is not limited to this number. The points that do not exist are removed as invalid points, and only valid points are extracted (step S220). Then, the dot information of the line five lines before is discarded, and the dot information is replaced (step S222).

Subsequently, as a result of extracting valid points, it is determined whether or not point information of the scanned line exists (step S224). If it is determined that a valid point exists, the invalid distance is set to 0 (step S22).
8). On the other hand, when it is determined that there is no valid point, it is determined that the currently scanned line is invalid, and 1 is added to the value of the invalid distance (step S226).

Next, it is determined whether or not the leading end has been detected (step S230).

The value of the invalid distance is equal to or more than an arbitrary number of lines (the number of arbitrary lines is a value for determining whether or not an effective point exists before the line currently being scanned. In the present invention, the number is set to 5, but the present invention is not limited to this number at all). The average coordinates of the detected line are output as a detection result (step S236), and the process ends with returning.

On the other hand, if it is determined that the value of the invalid distance is less than 5, it is determined whether or not the currently scanned line is the end opposite to the start end (step S232). If it is determined that the scanned line is the end opposite to the start end, the average coordinates of the line on which the last valid point was present are output as a detection result (step S236), and the process returns to the return. The process ends.

On the other hand, if it is determined that the currently scanned line is not the end opposite to the start end, the process returns to step S218 to start scanning the next inner line.

In the above embodiment, a liquid crystal projector is used as the projector, but a projector using a reflection type light modulation element that performs modulation by controlling the angle of a minute mirror in accordance with image data. May be applied. Although a desktop computer is used for the computer main body 20 in FIG. 1, the configuration may be applied to a notebook computer instead. Also, in FIG.
Although the projection image from the liquid crystal projector 16 is projected above, a configuration in which the projection image is applied to a whiteboard may be used instead.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and can be carried out in various modes without departing from the gist of the present invention. Of course it is.

[0097]

As described above, the pointing device, the pointing control method, and the recording medium of the present invention can automate a calibration work by adding an image pickup means to a computer, a projector, a normal screen or a whiteboard. There is an effect that the pointing operation can be performed without using a special screen or a special pen.

[Brief description of the drawings]

FIG. 1 is a schematic view of a computer system, a liquid crystal projector, an image sensor, a screen or whiteboard, and a presenter to which an embodiment of the present invention is applied.

FIG. 2 is a block diagram illustrating a schematic configuration of hardware of a computer system.

FIG. 3 is a configuration diagram illustrating a schematic configuration of hardware of the imaging device.

FIG. 4 is a block diagram illustrating a state of processing performed by a computer main body 20 on an image captured by an imaging device 14.

FIG. 5 is a block diagram illustrating a state of processing performed by a computer main body 20 on an image captured by an imaging device 14.

FIG. 6 is a flowchart showing a flow of a series of processes from a screen setting to a pointing process executed by a CPU 40.

FIG. 7 is a flowchart of a screen setting subroutine executed by a CPU 40;

FIG. 8 is an explanatory diagram showing a pattern displayed when a white pattern display subroutine executed by a CPU 40 is executed.

FIG. 9 is a flowchart of an image capturing subroutine executed by the CPU 40;

FIG. 10 is a flowchart of a projector projection image frame extraction subroutine executed by a CPU 40;

FIG. 11 is an explanatory diagram showing a pattern displayed when a position calibration pattern display subroutine executed by the CPU 40 is executed.

FIG. 12 is a flowchart of a calibration dot coordinate extraction subroutine executed by the CPU 40;

FIG. 13 is a flowchart of a designated area binarization subroutine executed by the CPU 40;

FIG. 14 is a flowchart of a pointing processing subroutine executed by the CPU 40;

FIG. 15 is a flowchart of a still image processing subroutine executed by the CPU 40;

16 is a flowchart of a moving image processing subroutine executed by the CPU 40. FIG.

FIG. 17 is a flowchart of a differential image binarization subroutine executed by the CPU 40;

FIG. 18 is a flowchart of a tip coordinate output subroutine executed by the CPU 40.

FIG. 19 is a flowchart of a leading edge detection subroutine executed by the CPU 40.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 ... Screen or white board 12 ... Liquid crystal projector projection image 14 ... Imaging device 16 ... Liquid crystal projector 18 ... CRT display 20 ... Computer body 22 ... Keyboard 24 ... Mouse 26 ... Floppy disk 28 ... Floppy disk drive 30 ... HDD 32 ... Modem 34 ... External network line 36 ... External computer 40 ... CPU 42 ... RAM 44 ... ROM 46 ... Video capture device 48 ... Parallel interface 50 ... USB interface 52 ... Network interface 54 ... Serial interface 56 ... Mouse interface 58 ... Keyboard interface 60 ... FDC 62 ... HDC 64 ... CRTC 66 ... IEEE 1394 interface 80 ... Presenter 82 Stick 100 Focus control device 102 Image sensor 104 Aperture control device 106 Aperture 108 Gain control device 110 Amplifier 120 Video capture driver 122 Image processing routine 124 Pointing address calculation routine 126 Mouse driver 128 Application 130: video driver 150: image processing and pointing processing chip

Claims (13)

[Claims] A display means for projecting and displaying an output image of a computer on a screen; a capturing means for capturing a projection image displayed on the screen into the computer by an imaging device; and a projection means for projecting the image on the screen. A display system comprising: a calibration unit that captures a calibration pattern into the computer by the capturing unit, and performs a calibration operation using the captured image. 2. The display system according to claim 1, wherein said calibration means divides the captured image into a plurality of regions and sets a threshold value for each region when binarizing the captured image. A display system including a process of performing binarization. 3. The display system according to claim 1, wherein said calibration means includes a process of extracting a projection area of said display means from a captured image. 4. The display system according to claim 1, wherein the image pickup device fixes a focus, an aperture, and a gain of the image pickup device after completion of a calibration operation. And display system. 5. The display system according to claim 1, wherein the calibration unit divides an area into a plurality of areas when extracting the calibration pattern from the captured image. A display system for extracting the calibration pattern for each area. 6. A display means for projecting and displaying an output image of a computer on a screen by using a projector, and a capture means for capturing a projection image displayed on the screen from the projector by the imaging device into the computer. An image generating unit that calculates a difference image between frames for the image captured by the capturing unit, and further binarizes the image to generate a binary difference image; A display system comprising: a detection unit that detects a location indicated by the instruction unit using the binarized difference image if the location is indicated by the instruction unit. 7. The display system according to claim 6, wherein, when capturing an image, the imaging device fixes a focus, an aperture, and a gain of the imaging device. 8. The display system according to claim 6, wherein said image generating means is configured to convert the difference image from two original images used to calculate the difference image when binarizing the difference image. A display system comprising a process of setting a threshold value for each pixel and binarizing an image. 9. The display system according to claim 6, wherein said detecting means sets a scan area on a rectangle, and sets a scan area near four sides of the set scan area. Compare the number of points present in the, the scan start position is the side with the largest number of points, scan sequentially from the scan start position toward the inside of the area,
Compare the distance between the scanned line and the point on the previously scanned line, extract only the points that are determined to be valid points and perform tip detection.The result of the tip detection and the previous tip A display system comprising a process of comparing a detection result detected at the time of detection and determining a detection result. 10. The display system according to claim 6, wherein the detecting means detects a leading end when the leading end is detected at a previous leading end detection. A display system characterized by including a process of setting an area only around a location detected at the time of a previous tip detection. 11. The display system according to claim 6, wherein said detecting means is configured to detect a state in which the tip is detected and a state in which the tip is not detected after detecting the tip. Wherein the determination as to whether or not the tip is detected includes a process performed based on the results of the previous tip detection. 12. A pointing control method in a display system including a computer, a projector, a screen, and an imaging device, comprising: (a) a step of setting an image capturing condition of the imaging device; and (b) the step (a). A) a step of performing image processing on an image captured from the imaging device whose settings have been completed according to step (b); and (c) a step of calculating a pointing address for the image subjected to image processing in step (b). Control method. 13. A recording medium recording a computer program for pointing control in a display system using a projector, a computer, a screen, and an imaging device, wherein: (a) a step of setting an image capturing condition of the imaging device (B) performing image processing on the image captured from the imaging device that has been set in step (a); and (c) pointing the image that has been subjected to image processing in step (b). A recording medium recording a computer program for causing a computer to execute the step of calculating an address.