JP2003036142A - System and method for presentation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide presentation technology for indicating a reference place and obtaining natural operability even when a large-sized screen is used. SOLUTION: A presentation system displays an image on a screen and indicates a point in the image with a light spot by using a pointing device. An image recognizing means extracts the light spot indicated by the pointing device from the image picked up by an imaging means and detects the position indicated by the pointing device. An image generating means generates an image having a symbol added at the position. The image with the added symbol is displayed on the screen by an image display means.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a presentation technique for displaying an image on a screen and pointing to the image displayed on the screen to proceed with the explanation.

[0002]

2. Description of the Related Art Conventionally, when giving a presentation, a large bulletin board having a brief description is presented in front of an audience, and a position is shown corresponding to a context or a figure to be referred in a lecture. It has been customary to proceed with.

With the widespread use of personal computers in recent years and the generalization of electronic presentations,
The discrepancy between the method of indicating the reference position, which has been used in conventional presentations, and the way in which electronic presentations proceed has become a problem.

At present, there are mainly two types of methods for instructing a reference point on a screen displayed by using presentation software (application). One is to operate a position indicating icon by using a position indicating device such as a mouse, which is unique to a personal computer, to give an instruction on the screen of the personal computer. The other is to directly specify the projection screen of the presentation by using a physical pointer or a light spot of a laser pointer. A method of selecting one of these or using both is general.

However, in the former case, it is not suitable for the conventional presentation method, and it is difficult to obtain a natural operation feeling while giving an interactive impression with the audience. On the other hand, in the latter case, the operation of the presentation application needs to be performed by another means. In the case of a presentation using a large screen, the latter method may make the reference point unclear or may make the instruction physically difficult. As described above, a method for coexisting a more natural operation feeling and utilization of the interactive function of the computer is being sought.
Various inventions have been devised for this problem.

In Japanese Patent Laid-Open No. 6-308879, information on light emitted from a display screen (or the vicinity thereof) is taken into an indicator, the position and axial direction of the indicator are calculated, and a point mark is displayed on the screen. Has been proposed.

Japanese Patent Laid-Open No. 8-95707 proposes a method of photographing the screen with an infrared camera, detecting the tip position of the shadow cast by the pointer, and calculating the position indicated by the pointer.

Japanese Unexamined Patent Publication No. 10-39994 discloses a method of calculating the instructed position by measuring the position, inclination, and gazing point of the explainer at any time with an instrument worn by the explainer. Has been done.

In Japanese Patent Laid-Open No. 9-120340, a user adjusts the focusing at a position pointed on the image with a laser beam pointer to increase the spot diameter while keeping the spot center fixed and incident on the sensor. Then, a method has been proposed in which the timing of incidence on each sensor is detected and obtained.

Further, in recent years, in Japanese Patent Laid-Open No. 2000-81950, there is a technique in which a camera captures an image of a bright spot pointed by a pointing rod and the position of this bright spot and a blinking pattern are detected. Proposed.

[0011]

DISCLOSURE OF THE INVENTION Problems to be Solved by the Invention
In the technique proposed in Japanese Patent Laid-Open Publication, it is necessary to directly point on the screen with a pointer, and in a presentation using a large screen, the instruction may be physically difficult.

Further, in the technique proposed in the above-mentioned Japanese Patent Laid-Open No. 2000-81950, the camera constantly captures an image of the screen, and the bright points (reference points) reflected on the screen are detected by image processing. This operation requires a large calculation load. To perform this load in real-time processing, high-speed calculation processing is required. Furthermore, when a large screen is used and an optical pointing device such as a laser pointer is used to project a light spot on the screen to indicate a position, the output performance of the pointing device and the distance to the instructor are also affected. Depending on the situation, it is assumed that the reference point cannot often be detected unless the screen is imaged at a higher resolution. In this case,
Higher performance is required for the camera, and the computational load increases accordingly.

[0013] Therefore, an object of the present invention is to provide a presentation technique capable of indicating a reference point even in a presentation using a large-screen screen and obtaining natural operability.

Another object of the present invention is to enable high-resolution image capturing by a camera and to carry out a large calculation load in real-time processing, so that image recognition processing can be performed at high speed, efficiency, and high accuracy. It is to provide a presentation technology that can be done.

[0015]

To solve the first problem, the present invention uses an optical indicator typified by a laser pointer to project a light spot at a reference position on a screen. To do. From the image of the screen captured by the camera, a light spot is extracted using image processing by an image recognition device, the shape of this light spot is recognized by the device to calculate the reference position, and the reference position can be easily understood by a computer. Combine symbols. Then, the image combined with this symbol is projected on the screen.

In order to solve the second problem,
A high resolution image can be obtained by capturing an image of the screen using a plurality of cameras. Further, the images of the respective cameras are subjected to image processing and recognition processing by using different image recognition devices, so that all steps of these processing are efficiently distributed.

[0017]

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of the present invention will be described below.

FIG. 1 is a block diagram showing an embodiment of a presentation system using the present invention.

The screen 100 is a screen for displaying the content of the presentation as an image. The screen surface onto which this screen is projected can be either flat or non-planar. Even if this screen is of a special type, the present invention can be applied.

The instructor 105 is a screen displaying the contents to be referred during the lecture of the presentation.
Optical pointing device towards 100 spots 1
Projects the 04 projection light. Reference numeral 102 denotes a light spot projected by the pointing device 104 held by the instructor 105. Further, 103 represents a visual image (symbol) for indicating a designated position on the screen screen designated by the pointing device 104. This image 104 is drawn in order to emphasize the pointing position of the optical pointing device, is created by a presentation application running on the computer 114 or its auxiliary application, and is projected by the projector 106 in the image display device 108. Is displayed together with the presentation content. Below, this is described as an instruction icon. Reference numeral 106 in FIG. 1 denotes a projector that projects a screen on a screen. This projector 106
Projects the screen-corrected image data sent by the processing of the distortion correction device 115 in the image display device 108 onto the screen 100. There may be a plurality of projectors 106.

When projecting an image on a non-flat screen, the distortion correcting device 115 uses the image of the presentation output by the computer 114 in which the presentation tension application is operating (hereinafter, this image is the original image). Is input as an input, and an image corrected in accordance with the distortion of the screen 100 is projected by the projector 106.
Output to. For this distortion correction device 115, an existing product having the above functions is used. There are measurement cameras 107-1 and 107-2 that keep observing the screen 100 during the operation of the presentation. Although only two such cameras are shown in FIG. 1, this does not limit the number of cameras to only two. As described below,
This camera can be installed more than one, and any number can be installed.

FIG. 2 shows a situation in which a flat screen is imaged by four cameras. Reference numeral 301 denotes a screen, and 107-1 to 107-4 denote cameras capturing 301. These cameras are installed on the surface of the screen where the display screen can be imaged. Alternatively, in the case of a transmissive screen, the back side of the screen may be a place where an image can be taken. As shown in FIG. 2, when there are a plurality of cameras, each camera takes an image of a part of the screen. However, when all the imaging areas are integrated, the position where the entire image display area can be taken can be taken. is set up. Of course, the areas captured by these cameras may overlap each other.

Returning to FIG. 1, image recognition devices 110-1 and 110-2
Reads the position and shape of the light spot projected on the screen 100 by the pointing device 104 from the images sent by the cameras 107-1 and 107-2. Image recognition device
For 110-1 and 110-2, basically, a plurality of devices with the same type of implementation are used to perform parallel processing. Although only two image recognition devices are shown in FIG. 1, this does not limit the number of image recognition devices to only two.

In the present embodiment, the same number of cameras and image recognition devices are used. However, in the present invention, the number of image recognition devices is not limited by the number of cameras. It is also assumed that images for several cameras are processed by a single image recognition device, or images for one camera are processed by several image recognition devices.

In the information integration device 113, the image recognition device 110
The most probable candidate is selected from the data that -1,110-2 sent that may conflict with each other. Also, the adopted position information is sent to the computer 114 on which the application for presentation operates.

The presentation application is operated by the computer 114 to generate a display image. These applications display a pointing icon, that is, a visual image 103 that emphasizes the pointing position, on the information integration device 113.
Create in the position adopted in. This instruction icon is displayed by the projector 106 in the image display device 108 together with the content of the presentation.

FIG. 3 shows the connection status of the camera 107, the image recognition device 110, the information integration device 113, the computer 114, and the image display device 108. The image recognition devices 110-1, 110-2, ..., 110-n shown in FIG.
The image of the presentation screen captured by 7-1, 107-2, ..., 107-n is received as an input, and the feature amount of the image determined to be the light emitting point of the reference position indication is sent to the information integration device 113 as an output. The feature amount sent at this time is not always one set. When a plurality of light emitting point candidates for reference position indication are found within the image range captured by the cameras 107-1, 107-2, ..., 107-n, the image recognition devices 110-1, 110-2,
..., 110-n send information about multiple locations. if,
If no candidate is found within the range, information that the candidate cannot be found is transmitted. In addition, it is possible that these pieces of information include erroneous information determined by noise.

In the information integration device 113, the image recognition device 110
The most probable candidate is selected from the data sent by -1,110-2, ..., 110-n that may conflict with each other. Also, the adopted position information is sent to the computer 114 on which the application for presentation operates.

The presentation application is operated by the computer 114 to generate a display image. The device can be replaced by dynamically connecting a general purpose computer to the system. Alternatively, a dedicated computer unique to the system may be used. Hereinafter, this is referred to as a presentation computer. Information integration device 11
The information sent from 3 is received by the presentation application running on the computer 114 or its auxiliary application. These applications create the pointing icon, that is, the visual image 103 that emphasizes the pointing position, based on the information sent from the information integration device 113. Then, the display image data including the visual image 103 is sent to the image display device 108.

The image display device 108, based on the display image data including the visual image 103 sent from the computer 114,
Display an image on screen 100.

The operation of this embodiment will be described below.

FIG. 4 shows a processing flow of an embodiment of the presentation system using the present invention. It is assumed that the output screen of the presentation computer 114 is displayed on the screen 100. Further, a light emitting point is projected on the screen by the optical pointing device 104 held by the instructor 105.

In the process 210 in the camera 107, the screen 100 is captured by the camera 107. As the camera 107 used at this time, an existing camera having a digital image output function can be used. In the present embodiment, there is one or more cameras, and an arbitrary number of cameras. In FIG. 2, reference numerals 107-1 to 107-4 denote cameras that capture the screen 301, respectively. These cameras are installed in a place where the front surface of the screen or, in the case of a transmissive screen, the back surface of the screen can capture images. As shown in FIG. 2, it is assumed that there are a plurality of cameras. Each of the cameras 107-1 to 107-4 captures an image of a part of the screen 301, but it is necessary that the cameras 107-1 to 107-4 be installed at a position where the entire screen display area can be fully captured when the entire imaging areas are integrated. The areas captured by these cameras may overlap each other. Each camera takes an image of the screen on which the presentation is being carried out, every predetermined period. These shall continue to be imaged for the period of time the system continues to operate.

As shown in FIGS. 5 and 6, the above conditions do not change when the screen is not flat. FIG. 5 is an example in which the present invention is applied to a device that provides an immersive virtual space. Three large screens 401, 402, 403 are installed so as to surround the user 400. At this time, the cameras 107-1 and 107-2 are installed at appropriate positions capable of capturing images of all the screens 401, 402, and 403. In this case, it is not necessary to match the number of cameras to the number of screens. Further, FIG. 6 is a diagram when the present invention is applied to a spherical screen. It is necessary to install an appropriate number of cameras in order to capture the entire area of the spherical screen 501.

As described above, high-resolution imaging can be performed by imaging the screen using a plurality of cameras. Further, by performing the image processing and the recognition processing on the images of the respective cameras by using different devices, all the steps of these processings can be efficiently distributed and processed. Also,
In this way, by using a plurality of cameras, even when the screen is a curved surface, it is possible to image each part of the screen from an angle at which the recognition performance is improved.

The relationship among the cameras in the entire system has a connection structure as shown in FIG. Processing of FIG.
In 211, the images captured by the cameras 107-1 to 107-n are transmitted in parallel to the image recognition devices 110-1 to 110-n. Image recognition device 110-1 from cameras 107-1 to 107-n
The output data sent to ~ 110-n contains the color information of each pixel.
It is assumed to be a two-dimensional array stored in RGB format, that is, a digital image. However, when data is transferred between devices, it is not limited to output and transmission in an array of RGB values. It is also assumed that appropriate compression processing is appropriately performed using the existing method and then the data is transferred.

In the present embodiment, each camera has only one corresponding image recognition device. However, in the present invention, the number of image recognition devices is not limited by the number of cameras. It is also assumed that images for several cameras are processed by a single image recognition device, or images for one camera are processed by several image recognition devices.

In the image recognition devices 110-1 to 110-n, the digital images periodically sent from the cameras 107-1 to 107-n are received as inputs in the process 212 of FIG. Next, by processing 213 and processing 214, a position estimated to be the light emitting point 102 projected on the screen 100 is detected from the image, and a list of structures including coordinates, shape information, etc. regarding the position is detected. create. Then, in processing 215, the information integration device 11 outputs the list of the structures as an output.
Send to 3. The format of this output data (list of structures) will be described later.

Processing 213 and processing 214 are performed by the image recognition devices 110-1 to 110-1.
110-n executed in parallel. Image recognition device 110-1
The same device is used for ~ 110-n.

The information integration device 113 is an image recognition device 110-
The data sent from 1-110-n is received by the process 216. In the process 217, the image recognition devices 110-1 to 110-1
Although the data sent from 10-n is divided into two images, the image originally having one light emitting point is searched for and a list of structures of such two images is synthesized. Next, in process 218, a list of image structure structures having a high probability of being a reference position by the pointing device is selected from the data subjected to process 217. Next, in process 219, the list of structures selected in process 218 is transmitted to the computer 114.

The computer 114 receives the data sent from the information integration device 113 by the process 221. next,
By the process 222, an image (arrow or the like) for emphasizing the reference position is arranged at a position with high probability that is the reference position by the pointing device, based on the data sent from the information integration device 113. Next, by the process 223,
Image data in which an image for emphasizing the reference position is arranged is transmitted to the image display device 108.

The image display device 108 receives the image data transmitted from the computer 114 by the process 224. Next, in process 225, the image data is processed by the distortion correction device 115,
Adds distortion according to the screen distortion. Finally,
By the process 226, the projector 106 displays the image data on the screen 100.

Processing 21 by the image recognition devices 110-1 to 110-n
A more detailed flow of the processing of 3 and processing 214 is shown in FIG. The processing performed by the image recognition devices 110-1 to 110-n will be described below with reference to FIG.

Prior to this, FIG. 8 is a block diagram showing a structural example of an apparatus capable of realizing this processing. In the device shown in FIG. 8, the CPU 707 is connected to the bus 707.
01, the main storage device 702, the input / output I / F unit 703, and the external storage device 704 are connected. An output device 705 and an input device 706 are connected to the input / output I / F unit 703. CPU
The 701 executes various processes according to the program recorded in the main storage device 702. The main storage device 702 and the external storage device 704 store programs and data necessary for executing processing. As the external storage device 704, in addition to a hard disk device or the like, a floppy (registered trademark) disk or the like which is a portable recording medium can be used. The input / output interface unit (input / output I / F unit) 703 is provided with a data transfer means necessary for exchanging input data from the camera, output data to the information integration device, and the like. In the present embodiment,
The following processing is executed by the CPU 701. However, this does not limit the present processing to be operated by a general-purpose CPU. It is also possible to use dedicated hardware having functions equivalent to the various processes described below.

In the process 212 of FIG. 4, the image information sent from the above-mentioned camera is received via the input / output I / F unit 703, and if necessary, the expansion process of the compressed image is performed by the existing method. To do. This expansion processing is performed by the CPU 701, and the restored image is stored in the main storage device 702.

In process 213, geometric transformation is performed on the image received from the camera by the above method. In the present embodiment, this processing 213 is executed by the CPU 701. However, this means that the process 213 is performed by a general-purpose CP.
It is not limited to operate by U. processing
It is also assumed that dedicated hardware having the same function as 213 is used.

Processing 213 is processing 801 and processing 802 shown in FIG.
Consists of. In process 801, geometric transformation for the purpose of correcting the distortion of the image captured by the camera is performed. The input and output of this process are both digital image information. The output image is an image in which the input image is geometrically distorted. The state of this geometric conversion will be described with reference to FIGS. 9 and 10. Reference numeral 901 in FIG. 9 shows the entire image of the screen 100 taken from a certain angle. The range 902 imaged by a camera contains only a portion of the image on this screen. In process 801, an image corresponding to the pixel position in the original image (planar screen data generated by the computer 114) as shown in FIG. 10 is reconstructed from this image. Reference numeral 1002 in FIG. 10 indicates the image range of the original image. Pixel position 905 and pixel position 1005, pixel position 906 and pixel position 1006, pixel position 907 and pixel position 1007,
The pixel position 908 and the pixel position 1008 are corresponding pixels.

FIG. 11 is a flowchart showing the processing steps for performing this geometric transformation. The process of geometric conversion will be described below with reference to FIG. In this process, the above-mentioned image is used for input. The output image is two-dimensional array data of color information expressed in a digital format such as RGB system. This output image is a digital image having the same size and image range as the original image displayed on the screen. Both input data and output data are stored in the main storage device 702. When the original image is divided into a plurality of images, there are the same number of output images as those images.

Each pixel of the output image has a 1-bit flag as auxiliary information in addition to color information. This flag includes
The determination as to whether or not the captured image obtained as an input includes image information corresponding to the pixel is written.
Hereinafter, this flag is called an undefined flag. The state of the undefined flag will be described again with reference to the reference diagram shown in FIG. FIG. 9 shows the entire screen and the image capturing state of the camera that captures the state. Of the entire screen 901 used for the presentation, the image information of the portion 902 imaged by this camera is the main processing 8
It is defined as the input image of 01. Meanwhile, processing
In the output image 1001 of 801, the entire surface of the screen 901 is the area of the image range. This range is shown in FIG. Ten
The area surrounded by 05, 1006, 1007, and 1008 is a place in the entire screen, which is imaged by the camera associated with the image recognition device. For the pixels corresponding to this area, the above-mentioned undefined flag is defined as true. For other areas, the undefined flag is defined as false.

Prior to the processing 801, two types of tables are prepared which store the correspondence between input pixels and output pixels. A two-dimensional position coordinate value information table and a true / false value (undefined flag) table. In these tables,
The two-dimensional position coordinate value of the pixel on the screen is used as an input, and the corresponding two-dimensional position coordinate value information on the original image and 1-bit data (undefined flag) are obtained as an output. An example of implementation of these tables is given in FIG. In this implementation example, addresses N to N + nm-1 of the main memory are used as a table of position coordinate value information, and addresses M to M + nm-1 are used as a table of 1-bit flag data.
Output x coordinate corresponding to input coordinate (a, b), output y coordinate,
Each data of undefined flag is N + 2nb + 2a address, N + 2nb + 2a + 1
It can be obtained by reading the memory at address M + nb + a. A method of creating data to be incorporated in the table of the two-dimensional position coordinate value information will be described later. The true / false value (undefined flag) table is true for the addresses corresponding to the position coordinates of the pixels imaged by the camera, and false for the addresses corresponding to the position coordinates of the pixels not imaged by the camera. Is stored.

FIG. 13 shows an explanatory view of the table of the two-dimensional position coordinate value information shown in FIG. As shown in FIG. 13, the table of the two-dimensional position coordinate information shown in FIG. 12 shows which coordinate position on the distorted screen corresponds to which coordinate position on the flat screen.

Now, the process of the geometric conversion shown in FIG. 11 will be described.

First, the color information data and flag of each pixel of the output image from the image recognition device are initialized to 0 (step 1
101).

Next, one of the output pixels from the image recognition device is selected, and the value in the true / false value table is read using the position information of that pixel as a key (step 1102). At this time, what is read from the true / false value table is 1-bit data representing the presence of an input pixel to the image recognition device corresponding to the output pixel. This data represents whether or not the corresponding output pixel was included in the imaging range of the camera.

Next, it is determined whether the value of the true / false value table read in step 1102 is true or false (step 1103).

If it is determined to be false in step 1103 and the pixel corresponding to the output image is not included in the range of the input image, the 1-bit information returned by the true / false value table is false. In that case, false is written in the undefined flag of the output pixel, and the position coordinates read from the two-dimensional position coordinate value information table are not used. And
Proceed to step 1107 (step 1104).

When it is determined to be true in step 1103 and the pixel corresponding to the output image from the image recognition device is included in the range of the input image to the image recognition device, 1 bit returned by the true / false value table. Information is true. In this case, the corresponding input pixel is referred to based on the coordinate value read from the table of the two-dimensional position coordinate value information (step 1105).

Next, true is also written in the undefined flag of the output pixel from the image recognition device. Further, the color information of the input pixel referred to based on the coordinate value read from the table of the two-dimensional position coordinate value information is copied to the color information of the output pixel (step 1106). At this time, it is also possible to adopt a method in which color information of several pixels around the input image is read from the input image and interpolation is performed between the color information to obtain the color information of the output pixel.

Next, in step 1107, it is determined whether or not the processing of all output pixels has been completed. If there is an output pixel that has not been processed, the process returns to step 1102, the pixel is selected, and the above process is performed again. When the above process is completed for all output pixels, the process 801 for that frame is completed (step 110).
8). This operation is performed in real time according to the image cycle of the camera.

The table of the two-dimensional position coordinate value information for the above-mentioned geometric transformation is created as follows. A test pattern image prepared in advance is used for creating the table. The state of this image is shown in FIG. A test pattern image is projected from the projector 106 onto the screen 100, and the screen is photographed by a fixed camera. The image of the test pattern used here is an image in which the positions of the pixels on the image captured on the screen screen and the pixel positions on the original image sent from the computer 114 can be easily associated with each other. As an example of such an image, a black and white mosaic image can be used in this embodiment. The size of the image mosaic is 1,2,4,8,16,32,
It is assumed that a plurality of images each having a side length twice as long as pixels are prepared and stored in the virtual storage device or the main storage device. A state in which the test pattern image is displayed on the screen screen is shown at 1401. Also,
The state of the original image of the test pattern image is shown at 1402.

FIG. 15 is a flowchart showing the flow of processing for creating a table of two-dimensional position coordinate value information for geometric transformation using the test pattern image.

First, the largest mosaic test pattern image is read (step 1300). Project this mosaic image onto a distorted screen and capture it with a camera
(Step 1301). Then, the end points where the white and black of each mosaic intersect are detected (step 1302). Next, the detected end points are associated with the end point positions of the original image (step 130).
3). At this time, if the number of mosaics included in the image is small, it can be associated with the end points of other mosaics without mistake. Save this correspondence in a table
(Step 1304). Subsequently, it is determined whether or not all mosaic images have been used (step 1305). If there is an image of a mosaic one step smaller than the previous mosaic, that image is read (step 1307) and the process returns to step 1301. At this time, by performing the work based on the recognition point of the previous image, it becomes unnecessary to associate different coordinate values. By the above method, the combination of the pixel position in the image 1401 captured by the camera and the pixel position in the image 1402 before output is found, and the information is stored in the table. Correspondence information of pixel positions that could not be matched only by the test pattern image is obtained by interpolating between the image coordinate values matched by the above method, and the corresponding positions of all output pixels are stored in the table (step 1306). The table of the two-dimensional position coordinate value information includes the position and distortion of the screen,
It shall be required again each time the position and angle of the camera and other setting conditions are changed.

In the conventional techniques such as the technique proposed in the above-mentioned Japanese Patent Laid-Open No. 2000-81950, it is assumed that the screen is flat. However, in reality, the screen may be distorted or the presentation system may be designed on the assumption that the screen is a curved surface. In such a system, consideration is given to display a more natural image by correcting the distortion of the image projected on the screen. In this case, it is necessary to consider the influence of the above-mentioned distortion correction also when acquiring the position information of the light spot designated by the image on the screen. Also, a screen designed as a flat surface may be distorted due to aging. In such a case, the screen itself has to be physically repaired in order to maintain the accuracy of detecting the position of the bright spot indicated by the image on the screen.

As in the present embodiment, the geometrical information is calculated using a table of two-dimensional position coordinate information in which the relationship between the pixel positions of the digitized picked-up image and the pixel positions of the original image projected on the projector is recorded. By performing the conversion, it is possible to correct the non-planar screen or the projection distortion of the screen in the calculation of the designated position. By this distortion correction, it becomes possible to more accurately read the designated position and recognize the designated content. With this, not only the flat screen,
It is possible to flexibly cope with the distortion of the curved screen and the projector that occurs during display. Further, even when the screen is distorted due to aging, it can be corrected and corrected by software. This allows
It is possible to provide a presentation technique capable of accurately detecting the position of a reference point even when the screen has a distortion or a curved surface.

Subsequently, color correction processing 802 is performed. This process 802 is intended to reduce the difference in color components generated between the image obtained by the process 801 and the original image. The procedure of this process is shown as a flowchart in FIG. The operation of the color correction processing 802 will be described below based on this flowchart.

In the color correction processing 802, the input image is the original image transmitted from the computer 114 in synchronization with the period in which the image is transmitted from the camera. This image is
It is received through the input / output I / F unit 703 and stored in the main storage device 702. It is defined as a two-dimensional array data structure of color information, that is, a digital image. In the color correction processing 802, the output image is formed as a digital image in which color information is stored in a two-dimensional array. It is assumed that the input image and the output image have the same size image data. In addition, a color correction table that stores the correspondence between input colors and output colors is prepared. The method of creating the information registered in this color correction table will be described later.

FIG. 17 shows an example of the mounting method of this color correction table. The RGB value of the color information has 8-bit data for each component. The table has a red component, a green component,
256 array data 1701-1703 for each blue component
Is prepared, and each array element contains the converted numerical value. This number is an integer between 0 and 255.

In the flowchart shown in FIG. 16,
After initializing the memory of the output image of the color correction processing 802 (step 1501), each input pixel is selected (step 1502),
The following processing is performed for each selected input pixel. The color information of a pixel is divided into three components, a red component, a green component, and a blue component. Each component is integer data with a value from 0 to 255. The data corresponding to each component is read from the color correction table (step 1503 to step 1505).
The read components are combined again to create output color information (step 1506). This color information is written as the color information of the corresponding pixel of the output image (step 1509). And
It is determined whether the above processing has been completed for all pixels (step 1510). If it is determined that some pixels have not yet been processed, the next pixel is selected (step
1512) and returns to step 1503. When the above process is completed for all the pixels, the output image is stored in the main storage device (step 1511), and the process 802 ends.

The color correction table used in the process 802 is created as follows by the procedure of the flowchart shown in FIG. First, a table for the red component is created. A color with 50% of three color components (50% gray)
Is created (step 1601). The red component of them is set to 0% (step 1602). This color is projected on the screen sent to the projector, and the color is photographed by the measurement camera (step 1603). The color information obtained by this processing is recorded in the table in association with the color information of the original image (step 160
Four). Increase the red component by one unit (step 160
7) returns to step 1603 and performs the same operation. It is determined whether or not the processing has been completed for all the red color components (step 1605), and if the processing has been completed for all the red color components, the table regarding the red color components is completed. Then, it is determined whether or not all color components have been selected (step 16
06) If there is any color component that has not been selected yet, return to step 1601. In this way, the color correction table for the blue component and the green component is created. It is assumed that the color correction table is required again each time conditions such as the brightness and the light amount of the room are changed. In addition, since it is possible that the color changes depending on the location within the screen, the accuracy can be improved by dividing the screen into some areas and using different color correction tables for each area.

Next, an example of a procedure for extracting the characteristic amount of the pointing device will be described. Here, as an example, a method of registering a feature amount using color information will be described.

An average position where the presenter who points the screen stands is defined as a basic position. The projector displays 50% gray on the screen and directs the light from the pointing device from the basic position toward the center of the screen to image the screen at that time. In this image,
The geometric transformation process 801 is performed to obtain the difference from the original image when only 50% gray is captured. The average of the amounts of change is calculated for all the pixels whose color information has changed by a certain threshold value or more. This average value is held as color information of the pointing device. During the image recognition processing 214, which will be described in detail later,
A comparison is made with this color information. Hereinafter, this color information is referred to as a registered color. Also, the radius of the figure is obtained and registered.

In this way, the feature quantity can be registered for each pointing device. As a result, it is possible to perform highly versatile discrimination in the image recognition processing.

Next, the process 803 shown in FIG. 7 will be described. The process of generating a binary image for facilitating image recognition is process 803. FIG. 19 is a flowchart showing an example of the processing procedure of this step. The procedure of the process 803 will be described below according to this flowchart. The input of this processing 803 is two types of image data. The output image created by processing 801 is the input image A
And Further, the output image created by the process 802 is referred to as the input image B. The output of this process 803 is assumed to be a binary image having the same size as the images A and B. Registered colors when the pointing device is projected on the screen are registered in the main storage device 702 in advance. If it is compatible with a plurality of pointing devices, a plurality of these registration colors are also registered. The registration method of this registered color has been described above. Here, the following operation is performed for each registered color.

First, one registered color is selected (step
1801). Next, one pixel coordinate P is selected (step
1802). In the color information of image A and image B, each component is
Assume that it takes an integer value from 0 to 255. Its coordinates P
The difference is calculated between the red component of the color of the pixel of image A in the image and the red component of the color of the pixel of image B in the same coordinate,
Add% component, ie 128. When the result is 0 or less, the missing component is truncated to 0, and when the result is the maximum value, that is, 255 or more, the excess component is truncated to give the maximum value. In this case, it is set to 255. Do the same for the blue and green components. The color information C obtained as a result of this is compared with the currently selected registered color. The absolute value of the difference between the red component of the color information C and the red component of the registered color is taken. Do the same for the blue and green components and take the sum. It is determined whether or not this total sum is larger than a predetermined reference value (threshold value) (step 1803). When it is determined in step 1803 that the value is larger than the reference value (threshold value), 0 is written in the position of the coordinate P of the output image (step 1805) and the process proceeds to step 1806. If it is determined that the value is smaller than the reference value (threshold value), 1 is written in the position of the coordinate P of the output image (step 1
804), and proceeds to step 1806. The value written at the position of the coordinate P of the output image indicates whether or not the value is similar to the light emitting point projected by the pointing device.

In step 1806, it is determined whether the processing has been completed for all output pixels. Step
If it is determined in 1806 that there is a pixel that has not been processed, the process proceeds to step 1807. Step 18
In 07, the next pixel is selected, and the process 1803 is returned to.
If it is determined in step 1806 that the processing has been completed for all output pixels, the process advances to step 1808. When the above operation is performed for all the pixels, a binary image in which 1s are written only at the points where the color components are close to the light emitting points projected by the pointing device is obtained. In step 1808, the obtained binary image is paired with the selected registered color and stored in the main storage device as the input image of the process 804. Next, in step 1809, it is determined whether all registered colors have been selected. If it is determined that there is a registered color that has not been selected, the next registered color is selected in step 1810, and the process returns to step 1802. When it is determined that all the registered colors have been selected, that is, when the above binary image is created for all the registered colors, the work of the process 803 ends.

The process 804 shown in FIG. 7 is a process for recognizing the projection view of the pointing device and its position with respect to the binary image information created in the process 803. Various techniques have been proposed and studied for such recognition technology.
The input of this process 804 is a list of combinations of binary images created in process 803 and registered colors. The output is a list whose elements are the structure 1900 shown in FIG. In this processing 804, with respect to the continuous image extracted from the binary image, the center point, the angle formed by the long axis and the length thereof, and the length in the direction orthogonal to the angle (short axis Length) and the number of pixels are measured and written to the data structure 1900 including the corresponding registered color. Since we can get 0 or more of these structures and an infinite number of them, we will create a list of structures and use it as the output.

Here, the process 804 for obtaining the structure 1900
Will be described. Note that the following description is a process performed assuming that the image of the part in which 1 is written in the binary image information created in process 803 is an ellipse.

First, in the binary image information created in the process 803, the image of the portion in which 1 is written is divided according to the connected portion. This processing can be realized by assigning an ID to each of the connected images. Figure 2
An example of binary image information is shown in 1 (a). FIG. 21 (b)
An example of the binary image information assigned an ID is shown in FIG. The data structure required for this processing is shown in FIGS.
It looks like (d).

FIG. 21C shows structure data having an ID for each pixel. In FIG. 21C, the ID is an integer number sequentially assigned from 1. In addition, 0 is fixed as a value when it does not belong to anywhere, that is, when it is not an image.

Further, FIG. 21 (d) stores the coordinate value of the pixel having this ID for the ID number. The pixel coordinate value for one ID number may reach several hundreds.

Here, FIG. 21 (c) and FIG.
A method of generating the data shown in (d) will be described. One pixel is selected from the binary image information as shown in FIG. 21A, and the following process is performed. (1) If the pixel value of the selected pixel is 0, 0 is given to the structure data having the ID for each pixel as shown in FIG. 21C to select the next pixel. . (2) If the pixel value of the selected pixel is 1, it is searched whether or not there is a pixel to which the ID is already attached in the 8 pixels around the pixel. If IDs have already been assigned to the 8 pixels around the pixel, the same ID number as that ID is given, and
Registration is performed on the two types of data shown in 1 (c) and FIG. 21 (d). If there is no pixel to which the ID is already assigned in the 8 pixels around the pixel, the next ID (in the structure data having the ID for each pixel as shown in FIG. Write N). Further, the coordinates of the pixel are given to the Nth structure data held for the connected image as shown in FIG. 21D, and 1 is added to the number of pixels. The next pixel is selected and the above process is repeated. In this way, the above processing is performed for all pixels.

As described above, the image in which the binary image is captured is divided as a unit. That is, with respect to the binary image information shown in FIG. 21A, the ID image-assigned binary image information shown in FIG. 21B was obtained.

Next, for each of these images, the center position, the angle formed by the long axis, the length of the long axis, and the length of the short axis are obtained. The following processing is performed on the structure data of each image.

First, the center position of the structure data held for the connected image is obtained. For this, the average value of the coordinate values registered in the structure data held for the connected image is obtained, and this average value is adopted as the center position. For this average value, the registered coordinate values are (x _i , y _i ) i = 0,
, N-1 is represented by the following mathematical expression (Equation 1).

[0085]

[Equation 1] An example of the center position obtained by the above processing is shown in FIG. 22 (a), for example.

Next, the inclination of the long axis is obtained. The inclination of the major axis is a, the coordinate values registered in the structure data held for the connected image are (x _i , y _i ) i = 0, ..., N−1, and the center position is (x _c , y _c ), the slope a is calculated by the following mathematical expression (Equation 2).

[0087]

[Equation 2] An example of the tilt obtained by the above process is, for example,
It is shown in FIG.

Next, the angle θ formed by the major axis is obtained. Angle θ
Is calculated by the following mathematical expression (Equation 3) from the inclination a.

[0089]

[Equation 3] Then, as shown in FIG. 22C, the image is rotated. FIG. 22C shows a state in which the image is rotated by the angle θ when the inclination is a.

Next, the major axis length and the minor axis length of the image rotated as shown in FIG. 22C are obtained. Of the coordinate values registered in the structure data for this image, the difference between the one with the largest y coordinate value and the one with the smallest y coordinate value is taken, and this length is taken as the length of the long axis. Establish. Also, the difference between the one with the largest x-coordinate value and the one with the smallest x-coordinate value is taken, and this length is defined as the length of the minor axis. The appearance of the long axis and the short axis of the image is shown in FIG.

As described above, the list having the structure 1900 shown in FIG. 20 as an element is generated.

Next, the operation of the information integration device 113 will be described. The information integration device 113 synthesizes the data sent from the image recognition device 110, infers the pointing portion indicated by the pointing device, and outputs the result to the presentation computer 114.

FIG. 23 is a block diagram showing a structural example of an apparatus capable of realizing this processing. FIG. 23
In the device shown in FIG. 1, a bus 2007, a CPU 2001, a main storage device 2002, an input / output I / F unit 2003, and an external storage device 200 are provided.
4 and are connected. The input / output I / F unit 2003 has an output device.
2005 and the input device 2006 are connected. CPU2001
Executes various processes according to the program recorded in the main storage device 2002. Main memory 2002 and external memory
In 2004, programs and data necessary for executing the processing are stored. As the external storage device 2004, a floppy disk or the like which is a portable recording medium can be used in addition to the hard disk device. In the input / output interface unit (input / output I / F unit) 2003, the image recognition device 110
It is assumed that a means for transferring data necessary for exchanging input data from the computer, output data to the presentation computer 114, and the like is provided. In the present embodiment, the following processing is executed by CPU 2001. However, this is not limited to the case where the present apparatus operates by a general-purpose CPU. It is also possible to use dedicated hardware having functions equivalent to the various processes described below.

The information integration device 113 receives the data sent from the plurality of image recognition devices 110. When all the data transferred from all image recognition devices are available,
The following work shall be started. The input of this process is
List of structures sent from multiple image recognition devices 19
00.

FIG. 24 shows a processing flow of the information integration device 113. In the information integration device 113, the lists of all the structures sent from the plurality of image recognition devices 202 are merged into one list. This is used as the input of the process shown in FIG. In the information integration device 113, first, in process 216, a plurality of image recognition devices 1
Receive a list of all structures sent from 10.
Next, in process 217, although the data from the image recognition device is divided into two figures, an object originally having an image of one light emitting point is searched for and combined. Next, in process 218, the degree of matching with the target image is checked, and the image estimated to be optimal is selected. Finally, in process 219,
The processing result is transmitted to the computer 114. Each process shown in FIG. 24 is the same process as the process of the same code shown in FIG.
FIG. 25 is a flowchart showing the operation of the process 217. The process 217 is a work for finding and combining an object that was originally an image with one light emitting point, which is divided into two figures in the data from the image recognition device.

First, in step 2201, it is determined whether or not there is an unselected registered color. If there is no registered color that has not been selected, the process ends. If it is determined that there is an unselected registered color, the registered color is selected (step 2202).
Next, in step 2203, it is determined whether or not there is an unselected image pair. If it is determined that there is no unselected image pair, the process returns to step 2201. If it is determined that there is an unselected image pair, two different elements are selected from the input list (step 2204).

It is examined whether these two elements satisfy all the following conditions. Condition 1, the registered colors of the two images are equal. Condition 2, the difference between the angles of the two images is less than a certain value. Condition 3, the value obtained by subtracting half the sum of the lengths of the major axes from the distance between the centers of the two images is less than or equal to a certain value. Above 3
If two conditions are met, the two structures are combined into one structure. The length of the new major axis is then the sum of half the length of the two major axes and the distance between the centers. The new minor axis length shall be the greater of the two minor axis lengths. Also, the new center position is obtained by subtracting a quarter of the long major axis from a quarter of the long major axis in the direction of the center point of the image having the long major axis from the midpoint of the two centers. It is defined as moving only the length.

The above work is performed for all combinations of elements.

In the flowchart shown in FIG. 25,
In step 2202, the above-mentioned condition 1 is taken into consideration. Then, in step 2205, it is determined whether or not the angles of the two images (structures) are similar, and the above-mentioned condition 2 is added. If it is determined that they are not similar, the process returns to step 2203. If determined to be similar,
Proceed to step 2206. In step 2206, it is determined whether or not the distance between the two images is short, and the above-mentioned condition 3 is added. If it is determined that they are not close, step 2203
Return to. If it is determined that they are close to each other, the two image pairs (structure pairs) are combined into one image (structure) (step 2207). This way of collecting is performed as described above.

Next, the process 218 will be described. The process 218 is from the list of image feature amounts given in the process 217,
This is a process of checking the degree of compatibility with a target image and selecting an image that is estimated to be optimal. In this process 218, the structure of the element selected as the reference position with high probability for each registered color is returned from the list changed in the above process 217 as an output. The details of this process 218 are shown in the flowchart of FIG.

In the flowchart of FIG. 26, first, a registered color is selected (step 2301). Next, a new image (structure) is selected (step 2302). Next, the number of pixels of the selected image is read, and this number of pixels is used as an evaluation value (step 2303). Next, in step 2304, it is determined whether or not the processing has been completed for all images (structures). If the processing has not been completed for all images (structures), the process returns to step 2302. When the processing is completed for all the images (structures), the process proceeds to step 2305. In step 2305, the image (structure) having the largest evaluation value is selected. Then step 23
In 06, it is determined whether or not the processing has been completed for all registered colors. If there is a registered color that has not been processed, the process returns to step 2301. When the processing is completed for all the registered colors, the processing proceeds to the transmission processing 219.

The following test criteria may be calculated for each element (structure). However, this test standard is an example for realizing the present invention, and the present invention is not limited to this method. Select an element from the list of inputs. The radius associated with the registered color of the element is read from the main storage device. The registration method of this information is described above. Takes the absolute value of the difference between the minor axis length and the radius value registered in the list. Let this absolute value be L. For each registered color, in process 2303, the absolute value of the distance between the position coordinate of the corresponding element selected last time and the position coordinate of the element being processed is calculated and stored. This number is called P. The linear sum aL + bP is calculated for the numerical values L and P of each element. Where a and b
Is a constant given when the device is created, and depends on the unit system and mounting method. The one with the smallest value is judged to have a high probability of being the light emitting point of the registered color and is selected.
At this time, if all of the above-mentioned selected test quantities are larger than a certain fixed value, it is determined that the image corresponding to the reference point was not obtained.

Returning to FIG. 24, in process 219, process 2
The data of the element (structure) selected in 18 is transmitted to the computer 114. This transmission is performed via the input / output I / F section.

With that, the processing of the information integration apparatus 113 is completed. The information integration device 113 performs the above processing in real time according to the transmission cycle from the image recognition device 110. After the processing is completed, the information integration device 113 waits for the next transmission.

Next, the processing of the presentation computer 114 will be described. Here, the information transmitted from the information integration device 113 is transferred to the presentation computer.
It is reflected in the contents of the screen drawing of 114.

In the computer 114 in which the presentation application operates, an application using the position reference information sent from the information integration device 113 is operating. This application can be the presentation application itself,
It may be operating as the auxiliary application. This application is a device for information integration 113
It receives the data input from and creates an image for emphasizing the reference position.

An example of creating an emphasized image will be given. Calculator 114
Has a two-dimensional image in advance in the main storage device or the external storage device, which is an image in which the reference position imitating an arrow or a finger can be intuitively understood. When there are a plurality of pointing devices, this image exists for each pointing device and is stored in association with each other. For each piece of information sent from the information integration device 113, a corresponding pointing device is selected from the registered color information. Then, the image associated with the pointing device is arranged at the coordinates written in the structure.

When there are a plurality of speakers, the above information is
The position of the instructor can also be represented by a metaphorical icon. As a result, a third party can also determine which pointing device the speaker holds, and thus the presentation power of the presentation can be increased. FIG. 27 shows an example of a presentation using this method. Speaker 2606 and speaker 2607 individually point on the screen with different pointing devices 2604 and 2605, but they are not confused because the instruction icons 2602 and 2603 are different. Moreover, since the direction of the icon implies the physical presence position of the instructor, the third party 2608 does not need any prior knowledge as to which speaker uses which instruction icon.

As another example of creating a reference image, an example of a reference image that can be used to distinguish a plurality of speakers as described above will be given.

An image that allows the user to intuitively understand the reference position of an arrow, a finger or the like is stored in advance in the main storage device or the external storage device as an image image. This image exists for each pointing device and is stored in association with each other. For each pointing device, this image is rotated according to the angular direction of the long-axis component sent from the information integration device 113. In the coordinates of the reference position (the coordinates of the center position sent from the information integration device 113),
The reference image is arranged on the screen so that the positions of the gaze points (arrows, tips of fingers, etc.) in this image overlap.

Another example of creating a reference image will be described. Hereinafter, in the example of the reference image, it is assumed that a pointing device that projects a circular light-emitting point is used on a screen installed in a front direction with respect to the projection direction. The image projected on the screen generally draws an ellipse as the position of the instructor (speaker) and the angle at which it is directed change. In the case of a flat screen, it is possible to estimate the three-dimensional position of the indicator projecting the light emitting point from information such as the angle of the ellipse, the ratio of the long axis to the short axis, and the size. .

28 and 29 are reference diagrams relating to the specific processing method. FIG. 28 is a diagram when the relationship between the screen and the designated position (reference position) is viewed from above. FIG. 29 is a diagram of the relationship between the screen and the designated position (reference position) when viewed from the front. It is assumed that the large screen 100 is located higher than the instructor 105. The pointing device 104 of the instructor is a pointing device that draws a perfect circle of radius r1 on the screen when projected vertically from the front of the screen at a distance d to the screen. This relationship is measured and registered before the start of the presentation. The flow of processing for obtaining the above-mentioned information from the above-mentioned binary image is shown.
First, the center of gravity of the diagram of the light spot of the pointing device is obtained. Then, for all the selected pixels of the figure, the direction in which the pixel distribution is the largest is calculated. This direction is defined as the angle 2402 of the figure. The figure is rotated clockwise by this angle, and the lengths of the rotated figure in the x-axis direction and the y-axis direction are the long axis length a (2405) and short axis length b (2403) of the figure. To decide. Shape radius length r (240
1) is defined as half the length of the short axis. Further, the angle θ (2400) formed by the straight line extended from the instructor to the designated point with respect to the screen can be expressed as θ = arcsin (b / a). From the angle θ and the angle of the long axis, two types of candidates are obtained as the three-dimensional direction from the designated point toward the instructor.
In that case, one of them can be selected by defining an appropriate constraint condition (in this case, the instructor is positioned below the screen). Further, rd / r1 is the distance from the instructor.

The three-dimensional position of the instructor obtained by this method is made to be a metaphor by the shape of the icon. FIG. 30 shows two types of instruction icons used for this operation. Figure 30
(A) shows that the two-dimensional image indicating the position is
This is a method used as a texture of a three-dimensional polygon. 27
01 indicates the point of instruction, and the direction indicated by 2702 indicates the direction in which the instructor is located. FIG. 30B shows a method of indicating the designated position and the presence position of the indicator using a three-dimensional polygon structure. The direction indicated by 2703 is
The pointing point is indicated, and the direction indicated by 2704 indicates the direction in which the instructor is located.

The computer 114 synthesizes the image (icon) created as described above on the presentation screen. The implementation method of this composition depends on the operating system used by the presentation computer 114.

Thus, the process 222 (see FIG. 4) is completed. In the next cycle, the image data (original image) accompanied by the created image (icon) is transmitted to the image display device 108 (process 223).

The image display device 108 receives the image data in process 224. Next, in process 225, the image data is distorted according to the screen distortion. Then, in process 226, the screen based on the distorted image data is projected on the screen 100. After that, again, in step 210, the screen projected on the screen 100 is captured by the camera 107.

Next, FIG. 31 shows a block diagram of another embodiment of the present invention. In FIG. 31, components given the same reference numerals as those in FIG. 1 have the same functions as the components shown in FIG. In FIG. 31, the indicator 105 points on the screen 100 with the optical pointing device 2804. A plurality of cameras 107-1 and 107-2 installed periodically capture the screen 100. Each image is sent to the image recognition devices 2810-1 and 2810-2. There, the light emitting points 28 projected in each area on the screen
02 information is extracted and sent to the information integration device 113.
In this information integration device 113, a plurality of image recognition devices 2810
The most probable candidate is selected based on the information that has been sent from and that may conflict with each other. This information is sent to the presentation computer 114, where the icon image 103 is synthesized and the presentation application is operated.

FIG. 32 shows a structural example of the optical pointing device 2804 used in this embodiment. The optical pointing device 2804 projects six light spots. (However, the feature of this pointing device in the present invention is not limited to the number of light spots. The same effect can be obtained as long as it projects a figure asymmetric with respect to rotation). This device can be considered to be divided into a light emitting unit 2906 and a holding unit 2907. Light emitting part
The 2906 is different from the holding section 2907 in the power section 2905 and the motor 29.
It rotates by the action of 04. The light emitting unit 2906 includes a light emitting component 2901 that emits a laser or visible light toward the front.
Installed (2901-1 to 2901-6). When the screen is pointed by the light emitting component 2901, an image of a light emitting point that periodically rotates is displayed on the screen. FIG. 33 shows a luminescent image projected by the pointing device 2804 shown in FIG. Screen 100
Six light spots 3301-1 to 3301-6 are reflected on the top. The emission images (light spots) 3301-1 to 3301-6 rotate with the rotation of the light emitting unit 2906 of the device 2804.

FIG. 34 shows the flow of processing performed in this embodiment. In FIG. 34, the process given the same reference numeral as the process shown in FIG. 4 is assumed to be the same process as the process shown in FIG. In FIG. 34, the image recognition device
Only the process 3410 (image recognition process) by 2810 is different from the process shown in FIG.

In FIG. 34, the camera 107 periodically picks up the screen. However, it is assumed that the imaging cycle and the rotation cycle of the pointing device 2804 are not a constant multiple. Further, it is assumed that the number of cameras 107 is arbitrary. The conditions regarding the arrangement of the camera 107 are assumed to be the same as those in the previous embodiment. The digital image captured by the camera 107 is sent to the image recognition device 2810. The image recognition device 2810 receives an image transferred from the camera 107 as an input, and outputs an array (list) of structures having position coordinate information.

FIG. 35 shows an example of the configuration of an image recognition device 2810 that can perform this image recognition process. In the device shown in FIG. 8, a bus 3307 is connected to a CPU 3301, a main storage device 3302, an input / output I / F unit 3303, and an external storage device 3304. Input / output I / F section 3303 has output device 330
5 and the input device 3306 are connected. CPU3301
Executes various processes according to the program recorded in the main storage device 3302. Main memory device 3302 and external memory device
The 3304 stores programs and data necessary for executing the processing. As the external storage device 3304, in addition to a hard disk device or the like, a floppy disk or the like which is a portable recording medium can be used. The input / output interface unit (input / output I / F unit) 3303 is provided with a data transfer means necessary for exchanging input data from the camera, output data to the information integration device, and the like. In the present embodiment, the following processing is C
It shall be executed by the PU3301. However, this is not limited to the case where the present apparatus operates by a general-purpose CPU. It is also possible to use dedicated hardware having functions equivalent to the various processes described below.

FIG. 36 is a diagram showing the process 3401 performed by the image recognition device 2810. The image transmitted from the camera at the time of the previous processing is stored in the main storage device 3302 (processing 3202). When a new image is input from the camera, the difference between the previous image and this new image is calculated (process 3203). The output of this process 3203 is also an image of the same size as the input image. The difference between the image taken in the previous frame (previous time) and the image sent from the camera in the current frame (this time) is calculated, and a 50% gray image is added. For the details of this difference processing, the same method as in the previous embodiment is taken. As for the area outside the screen in the image pickup range of the camera, the undefined flag is set as in the above-described embodiment. This process 32
The image generated as a result of 03 is sent to the process 3204 as an output.

A process 3204 receives the image output in the process 3203 as an input and outputs a binary image. The characteristic color of the pointing device is registered in advance. This registration method uses the same method as that described in the previous embodiment.

In process 3204, the following process is performed on the pixel at each coordinate of the image output in process 3203. The color component of a pixel is divided into a red component, a blue component, and a green component. The difference between the above red component and the registered red component is calculated.
The difference between the above blue component and the registered green component is calculated. The difference between the above green component and the registered green component is calculated. Take the sum of the above three differences, and if this is below a certain reference value,
Write 1 to the corresponding pixel position in the output image. Otherwise, write 0. When the above process is completed for all pixels, the process 3204 ends.

Next, in process 3205, distortion correction process using the geometric conversion table is performed on the binary image obtained in process 3204. This processing uses the same method as that described in the previous embodiment. The output from this process 3205 is a binary image that has undergone distortion correction processing.

The process 3206 of FIG. 36 shows the recognition process for the image input from the process 3205. Input the above process 3205
It is a binary image that has been subjected to the distortion correction processing created by. If the total number of pixels that are 1 in the input image is larger than the threshold value B, it is considered that the image itself has changed significantly for some reason, and information is not captured. In this case, the position information of the pointing device uses the information obtained by the previous frame. If it is less than a certain threshold value C, it is determined that the irradiation by the pointing device is not performed, and the information is not captured. In this case, the position information of the pointing device is determined to be undiscoverable. In other cases, the recognized image is determined to be position information based on the light spot of the pointing device.

In process 3206, the image recognition device 2810
The center of gravity of the pixel position determined to be the position information based on the light point of the pointing device is obtained, and the point is determined to be the designated point. In addition, the direction with the largest variance is obtained for the positions of all pixels. The direction is defined as the major axis direction. The length of the long axis and the length of the short axis are obtained in the same manner as described above. These pieces of information are written in the data structure shown in FIG. After that, the data structure is transmitted to the information integration device 113 by the process 215 of FIG. Subsequent processing,
The same as in the previous embodiment. When the screen other than the light spot designated by the pointing device changes due to the change of the screen or some other reason, it may fail to obtain the update information regarding the position of the pointing device from the difference information. In that case, the icon is presented based on the previous position information.

The contents of the present invention are not limited to presentations, but can be applied to amusement systems and communication systems using large-screen projector systems.

[0129]

According to the present invention, it is possible to provide a presentation technique capable of indicating a reference point even in a presentation or the like using a large-screen screen and obtaining natural operability.

Further, according to the present invention, high-resolution image pickup by a camera is possible, and a large calculation load is performed in real-time processing, so that image recognition processing can be performed at high speed, efficiency, and high accuracy. Can provide presentation technology that can be done.

[Brief description of drawings]

FIG. 1 is a configuration diagram showing an embodiment of the present invention.

FIG. 2 is an explanatory diagram when a screen is captured by four cameras.

FIG. 3 is an explanatory diagram showing a connection relationship of components according to the embodiment of the present invention.

FIG. 4 is an explanatory diagram showing a flow of processing according to the embodiment of the present invention.

FIG. 5 is an explanatory view when the screen is not flat.

FIG. 6 is an explanatory view when the screen is not flat.

FIG. 7 is a flowchart showing processing of the image recognition device.

FIG. 8 is a block diagram showing a configuration example of an image recognition device.

FIG. 9 is an explanatory diagram of geometric conversion.

FIG. 10 is an explanatory diagram of geometric conversion.

FIG. 11 is a flowchart showing processing of geometric conversion.

FIG. 12 is an explanatory diagram of a table used in geometric conversion.

FIG. 13 is an explanatory diagram of a table used in geometric conversion.

FIG. 14 is an explanatory diagram of a method of creating a table used in geometric transformation.

FIG. 15 is a flowchart showing a process of a method of creating a table used in geometric transformation.

FIG. 16 is a flowchart showing color correction processing.

FIG. 17 is an explanatory diagram of a table used in color correction processing.

FIG. 18 is a flowchart showing processing of a method of creating a table used in color correction processing.

FIG. 19 is a flowchart showing a process of generating a binary image.

FIG. 20 is an explanatory diagram showing an example of a format of information sent from the image recognition device to the information integration device.

FIG. 21 is an explanatory diagram of a method of creating information sent from the image recognition device to the information integration device.

FIG. 22 is an explanatory diagram of a method of creating information sent from the image recognition device to the information integration device.

FIG. 23 is a block diagram showing a configuration example of an information integration device.

FIG. 24 is a flowchart showing the processing of the information integration device.

FIG. 25 is a flowchart showing an identity determination process.

FIG. 26 is a flowchart showing a selection process.

FIG. 27 is an explanatory diagram of an example in which different icons are given to a plurality of instructors.

FIG. 28 is an explanatory diagram of a method of estimating the position of the instructor.

FIG. 29 is an explanatory diagram of a method of estimating the position of the instructor.

FIG. 30 is an explanatory diagram of an example of an icon for indicating the position of the instructor.

FIG. 31 is a configuration diagram showing another embodiment of the present invention.

FIG. 32 is an explanatory diagram of a pointing device according to another embodiment of the present invention.

FIG. 33 is an explanatory diagram of a light spot by a pointing device according to another embodiment of the present invention.

FIG. 34 is an explanatory diagram showing a flow of processing in another embodiment of the present invention.

FIG. 35 is a block diagram showing a configuration example of an image recognition device according to another embodiment of the present invention.

FIG. 36 is a flowchart showing the processing of the image recognition device in another embodiment of the present invention.

[Explanation of symbols]

100 screen Light spot illuminated by 102 104 103 symbol 104 Indicator (optical pointing device) 105 Instructor 106 Projector 107-1, 107-2 cameras 108 image display device 110-1, 110-2 image recognition device 113 Information integration device 114 Presentation Calculator 115 Distortion correction device

Front page continuation (51) Int.Cl. ⁷ Identification code FI theme code (reference) G09G 5/08 G09G 5/08 L (72) Inventor Haruo Takeda 1099, Ozenji Temple, Aso-ku, Kawasaki-shi, Kanagawa Hitachi, Ltd. System Development Laboratory F-term (reference) 5B087 AA09 AB02 CC09 CC12 CC26 CC33 DD05 DE03 DE07 5C082 AA03 AA24 AA27 BB42 CA02 CA59 CB05 DA42 DA87 MM09 MM10

Claims (10)

[Claims] 1. A presentation system in which an image is displayed on a screen and a pointing device indicates the light spot on the image, and image generation means for generating an image to be displayed on the screen, and the image generation means. Image display means for displaying the image generated by the screen on the screen, image pickup means for picking up an image projection surface of the screen, and light selected by the pointing device from among the images picked up by the image pickup means. Image recognition means for extracting a point, recognizing the shape of the extracted light spot, and detecting a position designated by the pointing device, wherein the image generation means is detected by the image recognition means. An image is generated by combining a symbol indicating that the position is indicated at the position, and the image display unit is configured to display the symbol. The synthesized image,
A presentation system for displaying on the screen. 2. A presentation system in which an image is displayed on a screen and a pointing device indicates the light spot on the image, and image generation means for generating an image to be displayed on the screen, and the image generation means. Image display means for displaying the image generated by the above on the screen, image pickup means for picking up the image projection surface of the screen, and light selected by the pointing device from among the images picked up by the image pickup means. An image recognition unit that extracts a point, recognizes the shape of the extracted light spot, and detects the position indicated by the pointing device and the orientation of the shape of the light spot; and Pointing to the position detected by the image recognition means from the direction detected by the image recognition means Generating a combined image symbol indicating the image display unit, an image synthesized with the symbols,
A presentation system for displaying on the screen. 3. A presentation system in which an image is displayed on a screen and a pointing device indicates the light spot on the image. Image generating means for generating an image to be displayed on the screen, and the image generating means. Image display means for displaying the image generated by the above on the screen, a plurality of image pickup means for dividing the image projection surface of the screen into a plurality of areas, and image pickup for each of the areas, and image pickup by the plurality of image pickup means A plurality of light sources designated by the pointing device, recognizing the shape of the extracted light spot, and detecting the position designated by the pointing device from among the displayed images, which are operable in parallel. Image recognizing means and whether the light spots extracted by the plurality of image recognizing means are the same light spot on the screen , An information integration unit that estimates the position where the light spot should be from the plurality of light spots when it is determined that they are the same, and the image generation unit is located at the position estimated by the information integration unit. , An image in which a symbol indicating that the position is designated is combined, and the image display unit displays the image in which the symbol is combined,
A presentation system for displaying on the screen. 4. A presentation system in which an image is displayed on a screen and a pointing device indicates the light spot on the image, and image generation means for generating an image to be displayed on the screen, and the image generation means. Image display means for displaying the image generated by the above on the screen, a plurality of image pickup means for dividing the image projection surface of the screen into a plurality of areas, and image pickup for each of the areas, and image pickup by the plurality of image pickup means From the image that has been extracted, the light spot designated by the pointing device is extracted, the shape of the extracted light spot is recognized, and the position designated by the pointing device and the direction of the shape of the light spot are determined. A plurality of image recognizing means that can be operated in parallel and a plurality of light spots extracted by the plurality of image recognizing means are detected on the screen. It is determined whether or not one light spot, and when it is determined that they are the same, information integration means for estimating the position of the light spot from the plurality of light spots and the direction of the shape of the light spot. The image generation means generates an image in which a symbol indicating that the position is indicated from the direction estimated by the information integration means is combined with the position estimated by the information integration means, The display means displays the image combined with the symbol,
A presentation system for displaying on the screen. 5. Accepting first image information, which is information about an image to be displayed on a screen, and second image information, which is information about an image in which the image is illuminated with a light spot, and An image recognition apparatus, which extracts a light spot, recognizes a shape of the extracted light spot, and detects a position indicated by the light spot and a direction of the shape of the light spot. 6. The screen according to claim 1, claim 2, claim 3, or claim 4, wherein the screen is a flat surface with distortion or a non-planar surface, and the image recognition means captures an image by the image capturing means. A presentation system comprising: a coordinate conversion unit for converting the formed image into an image on a flat screen, and recognizing the shape of the light spot using the image converted by the coordinate conversion unit. 7. The image according to claim 1, claim 2, claim 3, or claim 4, wherein the image recognition means displays an image from the image generated by the image generation means on the screen. In order to estimate, the image generated by the image generation means is provided with color conversion means for performing color conversion suitable for the environment condition, and the image converted by the color conversion means is used to A presentation system that recognizes the shape of a light spot. 8. The screen according to claim 1, claim 2, claim 3, or claim 4, wherein the screen is a flat surface or a non-planar surface with distortion, and the image recognition means captures an image by the image capturing means. Coordinate conversion means for converting the generated image into an image on a flat screen, and the image recognition means estimates an image displayed on the screen from the image generated by the image generation means. In order to do so, the image generated by the image generation means is provided with a color conversion means for performing color conversion suitable for the implementation environment state, and the image converted by the coordinate conversion means and the color converted by the color conversion means. A presentation system characterized in that the shape of the light spot is recognized by comparing the converted image. 9. A presentation method in a presentation system in which an image is displayed on a screen and a pointing device indicates the light spot on the image, the image being generated on the screen, and the generated image. Image is displayed on the screen, the image projection surface of the screen is imaged, the light spot designated by the pointing device is extracted from the imaged image, and the shape of the extracted light spot is determined. Recognizing, detecting the position indicated by the pointing device, generating an image in which a symbol indicating indicating the position is combined at the detected position, and generating an image in which the symbol is combined, A presentation method characterized by displaying on a screen. 10. A first image information, which is information about an image to be displayed on a screen, and a second image information, which is information about an image in which the image is irradiated with a light spot, are received, and An image recognition method comprising extracting a light spot, recognizing a shape of the extracted light spot, and detecting a position indicated by the light spot and a direction of the shape of the light spot.