JP4725383B2 - Pointing device, external information processing device, pointing position specifying device, and pointing position specifying method - Google Patents

Description

  The present invention relates to a pointing device suitable for use in an image projection system, an external information processing device, an indicated position specifying device including them, and an indicated position specifying method.

  Conventionally, a projection system in which a personal computer or an external video device is connected to a projector and various display images are enlarged and displayed on an arbitrary projection surface is used in a presentation or the like.

  When a presenter or the like indicates an arbitrary position in a projection image during projection of a display image using such a projection system, a laser pointer is generally used. In addition, if the personal computer has a pointer display function for superimposing a pointer on the display image or a similar function is provided on the projector side, the personal computer or projector side key operation, and further the personal computer By operating a pointing device such as a connected mouse or a remote control unit attached to the projector, the superimposed pointer is moved so that an arbitrary position in the projected image can be indicated.

  By the way, in a projection system, for example, when an additional function such as enlargement of an arbitrary part of a display image instructed by a presenter or the like, emphasis or enlargement of a character of the instructed part, or display of a line drawing is provided, It is indispensable to acquire information on the indicated position.

In that case, as described above, in the case of a system in which an arbitrary position can be indicated by the pointer, the display position (superimposition position) of the pointer at an arbitrary time point can be acquired as it is as the indicated position. However, when a laser pointer is used, the indicated position cannot be acquired immediately. As a technique for solving this, for example, in Patent Document 1 below, a projected image is captured, the image data of the projected image area in the captured image is compared with the original display image data, and the difference image data of both images is obtained. There are described a method for specifying an instruction position by obtaining an instruction image, and a method for specifying a point with high luminance in a captured image as an instruction position.
JP 2002-229735 A

  However, when an arbitrary position in the projection image is indicated using the laser pointer in the projection system, the indication position can be specified using the above-described technique, but the projection image is captured. Therefore, it is necessary to incorporate a camera device for the projector or personal computer, or to connect it externally. Therefore, in the former, the cost of a projector or the like is high, and in the latter, preparation work before projection becomes complicated, such as preparing a camera device separately from the laser pointer and securing the installation location. In addition, the projected image may be a still image, but there is a problem that it is difficult to specify the designated position when the projected image is a moving image.

  On the other hand, the pointer display function as described above is provided in the projector or personal computer, and the above problem does not occur when an arbitrary position in the projection image is indicated by using the function, but in this case, the pointer is moved. Since the operation is a key operation, there is a problem that it is difficult to say that the workability is always good compared to the case of using a laser pointer.

  The present invention has been made in view of such a conventional problem, and is used for indicating an arbitrary position of a projection image and is physically and economically used for a projection system and the like while ensuring the workability of a user. A pointing device, an external information processing device, a pointing position specifying device, a pointing position specifying method, and a program used for realizing the pointing position that can specify the pointing position in the projection image without overloading The purpose is to do.

To solve the above problems, when the invention of claim 1, a pointing device for designating an arbitrary position on the display screen, an imaging means for photographing the display screen, shot taken by the imaging means an acquisition unit configured to acquire coordinate positions of a plurality of stationary feature point indicating a display screen area in the image, the coordinate position of a predetermined reference point in images picked by the imaging means, the plurality of stationary feature point Based on the coordinate position, when the display screen area in the photographed image is transformed into a normal form, conversion means for converting to the corresponding coordinate position of the display screen, and the coordinate position of the reference point after conversion by the conversion means and transmitting means for transmitting to the external information processing apparatus as an instruction position on the screen, and the converting means of the plurality of stationary feature points obtained by the obtaining means When the coordinate conversion of the reference point is performed based on the target position and any of the coordinate positions of the plurality of stationary feature points cannot be acquired, the same as the previous time based on the coordinate position acquired by the acquisition unit According to the conversion condition, the coordinate conversion of the reference point is performed.

In such a configuration, it is possible to specify the indicated position in the display screen without using an external information processing device or the like and supply it to the external information processing device. An arbitrary position on the display screen can be indicated simply by changing. In such a configuration, even if any of a plurality of stationary feature points necessary for specifying the designated position in the display screen is not within the shooting angle of view, the designated position is determined according to the same conversion condition as the previous time. By specifying, the indicated position in the display screen can be specified.

  In the invention of claim 2, the indicated position transmitted to the external information processing apparatus by the transmitting means is a display position of a predetermined designated position mark to be displayed on the display screen.

  In such a configuration, the user of the pointing device can confirm the position of the display screen instructed by the user on the display screen.

  In the invention of claim 3, the reference point is the center of the captured image.

  In such a configuration, the conversion of the coordinate position of the reference point can be easily performed as compared with the case where it is other than the center of the captured image.

In the invention of claim 4 , the stationary feature points are the four corners of a frame existing on the outer periphery of the display image displayed on the display screen.

In the invention of claim 5 , the stationary feature point exists outside the display screen area, and the conversion unit determines a coordinate position of the feature point acquired by the acquisition unit as a predetermined value. Correction is performed using a correction value set in advance by a setting operation, and coordinate conversion of the reference point is performed based on the corrected coordinate position.

Even in such a configuration, even if any of a plurality of stationary feature points necessary for specifying the indicated position in the display screen is not within the shooting angle of view, the indicated position is specified according to the same conversion conditions as the previous time. By doing so, it is possible to specify the designated position in the display screen.

Further, a pointing device for In the invention of claim 6 for instructing an arbitrary position on the display screen, an imaging means for photographing the display screen, the display screen in images picked up by the imaging means An acquisition unit that acquires the coordinate positions of a plurality of stationary feature points indicating a region, and a coordinate position of a predetermined reference point in a captured image captured by the imaging unit is based on the coordinate positions of the plurality of stationary feature points. Conversion means for converting the display screen area in the captured image into a corresponding coordinate position of the display screen when transformed into a normal form, and the coordinate position of the reference point after conversion by the conversion means is indicated on the display screen As a transmission means for transmitting to an external information processing apparatus and a known specific pattern indicating a coordinate reference of the display screen as being displayed on the display screen. Recognizing means for recognizing the specific pattern, and if any of the coordinate positions of a plurality of stationary feature points required for the coordinate conversion of the reference point by the converting means cannot be acquired by the acquiring means, The transmission means transmits a request signal for requesting the display of the specific pattern on the display screen to the external information processing apparatus, and the conversion means is based on the coordinate reference indicated by the specific pattern recognized by the recognition means. The coordinate conversion of the reference point was performed .

In such a configuration, even if any of a plurality of stationary feature points necessary for specifying the indicated position in the display screen is not within the shooting angle of view, by displaying a specific pattern on the display screen, The designated position in the display screen can be specified.
In such a configuration, the display of the specific pattern on the display screen can be minimized.

In the invention of claim 7 , the specific pattern is an image displayed on the display screen at regular time intervals.

In the invention of claim 8 , the feature point of the display image in the display screen area,
A positional relationship acquisition unit that acquires positional relationship information indicating a positional relationship with the feature points of the specific pattern in the display screen area; and a storage unit that stores the positional relationship information acquired by the positional relationship acquisition unit, The conversion means stores the characteristic points of the display image in the display screen area in the storage means during the coordinate conversion of the reference point based on the captured image in a state where the specific pattern is not displayed on the display screen. The coordinate conversion of the reference point is performed based on the coordinate reference indicated by the specific pattern including the feature point in the positional relationship indicated by the positional relationship information.

  In such a configuration, by displaying a specific pattern on the display screen at regular time intervals, any one of a plurality of stationary feature points required for specifying the indicated position in the display screen within the shooting angle of view. Even if is not within the specified range, the indicated position in the display screen can be specified. In addition, since the designated position can be specified even when the display interval of the auxiliary image is long, the display interval of the specific pattern can be increased.

In the invention of claim 9, further comprising: a specifying unit that specifies a vertical direction of a display screen in the photographed image; and a character recognition unit that recognizes a character existing in the photographed image, The specifying means specifies the vertical direction of the display screen in the photographed image based on the rotation state of the character recognized by the character recognition means in the photographed image, and the conversion means is a predetermined reference in the photographed image. The coordinate position of the point corresponds to when the display screen area in the photographed image is transformed into a normal form, and is converted to the coordinate position of the display screen according to the direction specified by the specifying means.

In such a configuration, it is possible to accurately specify the indicated position in the display screen even when the vertical direction of the captured image does not match that of the display screen. In such a configuration, even if the vertical direction of the captured image does not match that of the display screen, if a recognizable character is displayed on the display screen, the indicated position in the display screen is changed. It can be accurately identified.

The invention according to claim 10 further comprises averaging means for averaging the coordinate positions of the latest multiple times coordinate-converted by the converting means, and the transmitting means is averaged by the averaging means. The coordinate position thus set is transmitted to the external information processing apparatus as the indicated position on the display screen.

  In such a configuration, it is possible to eliminate a change in position due to the swing of the apparatus main body when specifying the designated position.

Further, in the invention of claim 11 , when the distance between the current coordinate position and the previous coordinate position transformed by the conversion means 11 is greater than a predetermined threshold value, the average A means for prohibiting the averaging of the most recent coordinate positions by the conversion means, and the transmission means uses the current coordinate position converted by the conversion means as the designated position on the display screen as an external information process. It was supposed to be sent to the device.

  In such a configuration, when the designated position is specified, a change in position due to the swinging of the apparatus main body is eliminated, and at the same time, it is possible to immediately cope with the intentional movement of the designated position.

According to a twelfth aspect of the present invention, there is provided operation means for instructing a predetermined operation to the external information processing apparatus, and the transmission means sends a predetermined command signal in response to an operation of the operation means. The information is transmitted to the information processing apparatus, and the conversion unit transmits the coordinate position converted before the time when the operation unit is operated to the external information processing apparatus as an instruction position corresponding to the command signal. It was supposed to be.

  In such a configuration, it is possible to eliminate a change in position caused by hand shake of the apparatus main body accompanying a user's operation when specifying the designated position.

According to the invention of claim 13 , there is provided an instruction position specifying method for indicating an arbitrary position of a display screen instructed by a user, and includes an imaging process for imaging the display screen and the imaging process. An acquisition step of acquiring coordinate positions of a plurality of stationary feature points indicating a display screen area in the photographed captured image, and a coordinate position of a predetermined reference point in the photographed captured image is represented by the coordinates of the plurality of stationary feature points. A conversion step of converting the display screen area in the captured image into a normal form based on the position, and a transmission step of transmitting the converted coordinate position to the external information processing apparatus; And the converting step performs coordinate conversion of the reference point based on the coordinate positions of the plurality of stationary feature points acquired by the acquiring step, and the plurality of stationary feature points. If any of the positions can not be acquired, and the method further comprises the step of performing coordinate transformation of the reference point according to the same conversion conditions as the previous based on the coordinate position can be acquired by the acquiring step.

  According to such a method, it is possible to specify the indicated position in the display screen only by the pointing device used by the user to indicate an arbitrary position on the display screen without using an external information processing apparatus or the like. When using the pointing device, it is possible to indicate an arbitrary position on the display screen simply by changing the orientation of the device main body in the same manner as a laser pointer or the like.

In the invention of claim 14 , the photographing means is included in a computer which is used for indicating an arbitrary position on the display screen and has a photographing means for photographing the display screen and a transmission means. A photographing process for photographing the display screen, an acquisition process for obtaining coordinate positions of a plurality of stationary feature points indicating a display screen area in a photographed image photographed by the photographing process, and a photographing photographed by the photographing means Based on the coordinate positions of the plurality of stationary feature points, the coordinate position of the predetermined reference point in the image is converted to the corresponding coordinate position of the display screen when the display screen area in the captured image is transformed into a normal form. Conversion processing and transmission processing for causing the transmission means to transmit the converted coordinate position to the external information processing apparatus as the designated position on the display screen. The conversion processing further performs coordinate conversion of the reference point based on the coordinate positions of the plurality of stationary feature points acquired by the acquisition step, and any one of the coordinate positions of the plurality of stationary feature points is When it was not able to acquire, it was set as the program which performs the coordinate conversion of the said reference point according to the same conversion conditions as last time based on the coordinate position acquired by the said acquisition process.

The invention according to claim 15 is an external information processing apparatus in which the pointing position designated by the pointing device is transmitted from the pointing device used for pointing to an arbitrary position on the display screen, In response to reception of the request signal by the receiving means, the receiving means for receiving a request signal for requesting display on the display screen of the specific pattern indicating the coordinate reference of the display screen transmitted from the pointing device, Display control means for performing an operation for displaying the specific pattern on the display screen is provided.

  In such a configuration, it is possible to ensure that the pointing device indicates the indicated position in the display screen only by displaying the specific pattern on the display screen in response to a request from the pointing device having a function of specifying and transmitting the indicated position on the display screen. The specified position can be acquired while specifying.

Further, in the invention of claim 16 , a pointing device used for indicating an arbitrary position on the display screen, and external information for receiving the indicated position on the display screen instructed by the user from the pointing device A request signal for requesting display on the display screen of a specific pattern indicating a coordinate reference of the display screen transmitted from the pointing device. Receiving means, and display control means for performing an operation for displaying the specific pattern on the display screen in response to reception of the request signal by the receiving means, and the pointing device includes the display screen Imaging means for photographing the image, transmission means for transmitting the request signal, and a photographed image photographed by the photographing means, the display An acquisition unit that acquires coordinate positions of a plurality of stationary feature points indicating a surface area, and coordinates of a predetermined reference point in a captured image captured by the imaging unit based on the plurality of coordinate positions acquired by the acquisition unit A conversion means for converting the position into a coordinate position of the display screen corresponding to the display screen area in the captured image transformed into a normal form, and a known specific pattern indicating the coordinate reference of the display screen is displayed on the display screen Recognizing means for recognizing the specific pattern from the captured image in a state of being made, and the transmitting means is one of the coordinate positions of a plurality of stationary feature points required for coordinate conversion of the reference point by the converting means. Is not acquired by the acquisition means, a request signal for requesting display of the specific pattern on the display screen is transmitted to the external information processing apparatus, The conversion unit recognizes the coordinate position of the plurality of stationary feature points required for the coordinate conversion of the reference point by the recognition unit and recognizes the request signal by the transmission unit when the acquisition unit cannot acquire any of the coordinate positions. The reference point coordinate conversion is performed based on the coordinate reference indicated by the specific pattern displayed in response to the transmission.

  In such a configuration, it is possible to specify the indicated position in the display screen only by the pointing device used by the user to indicate an arbitrary position on the display screen without using an external information processing device or the like. When the apparatus is used, it is possible to indicate an arbitrary position on the display screen simply by changing the orientation of the apparatus main body in the same manner as a laser pointer or the like. In addition, even if any of a plurality of stationary feature points required for specifying the indicated position in the display screen is not within the shooting angle of view, a specific pattern is displayed on the display screen to display the specific pattern on the display screen. The designated position can be specified.

  As described above, according to the pointing device of the present invention, it is possible to specify the indicated position in the display screen without using an external information processing device or the like and supply it to the external information processing device. In the same manner as described above, an arbitrary position on the display screen can be designated simply by changing the orientation of the apparatus main body. Therefore, it is possible to specify the designated position in the projection image without imposing an excessive physical and economical burden on the projection system or the like while ensuring the workability of the user.

  Further, according to the pointing position specifying method of the present invention, the pointing position in the display screen is specified only by the pointing device used by the user for pointing to an arbitrary position on the display screen without using an external information processing apparatus or the like. In addition, when using the pointing device, it is possible to indicate an arbitrary position on the display screen simply by changing the orientation of the device main body in the same manner as a laser pointer or the like. Therefore, it is possible to specify the designated position in the projection image without imposing an excessive physical and economical burden on the projection system or the like while ensuring the workability of the user.

  According to the external information processing apparatus of the present invention, the position of the display screen instructed to the user of the pointing device having the function of identifying and transmitting the designated position of the display screen is displayed on the display screen. Can be confirmed. Therefore, it is possible to improve the usability of a specific pointing device including the pointing device of the present invention.

  According to the pointing position specifying device of the present invention, the pointing position in the display screen is specified only by the pointing device used by the user for pointing to an arbitrary position on the display screen without using an external information processing device or the like. In addition, when using the pointing device, it is possible to indicate an arbitrary position on the display screen simply by changing the orientation of the device main body in the same manner as a laser pointer or the like.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
(Embodiment 1)
FIG. 1 is a configuration diagram of a projection system including a pointing device 1 and a personal computer (hereinafter referred to as a PC) 2 constituting a pointing position specifying device according to the present invention common to each embodiment. In this projection system, the projector 3 projects a display image based on the image data output from the PC 2 onto the screen S so as to be enlarged.

  The pointing device 1 is used to indicate a display image projected on the screen S, that is, an arbitrary position of the projection image. The pointing device 1 detects an indication position indicated by the user and uses the detected position of the display image. The following configuration for sending to the PC 2 as a coordinate position (coordinate value) in the coordinate space is provided.

  That is, the pointing device 1 has a function of photographing the screen S. The photographing lens 11 and an image sensor 12 such as a CCD which is photographing means for digitizing and capturing an optical image formed on the photographing lens 11. And an image processing unit 13, a calculation unit 14, a communication unit 15, a memory 16, and an operation unit 17 that perform various image processing on the captured image data (captured image data).

  The calculation unit 14 is configured by a microcomputer (not shown) and a microcomputer including a ROM that stores programs for causing the CPU to execute the various calculations. In the present embodiment, the calculation unit 14 operates in accordance with the program stored in the ROM, thereby performing various calculations based on the image data after the image processing by the image processing unit 13, and the conversion means of the present invention. Further, by functioning as an acquisition unit, the coordinate position is calculated by an operation described later during use.

  The memory 16 is a working memory for the calculation unit 14 and stores various data as necessary. The communication unit 15 is a transmission unit according to the present invention. For example, the infrared communication method with a relatively narrow communication range, a Bluetooth (registered trademark) method, a wireless communication method such as a wireless LAN, a USB (Universal Serial Bus), or the like. The coordinate position calculated by the calculation unit 14 is transmitted to the PC 2 by an arbitrary data transmission method such as a wired communication method.

  The operation unit 17 includes, for example, one or a plurality of operation buttons that are operated at the start of use of the pointing device 1 or during use, and a circuit that sends the operation content to the calculation unit 14.

  On the other hand, the PC 2 is a notebook computer, for example, and is used as the external information processing apparatus of the present invention to constitute the pointing position specifying apparatus of the present invention together with the pointing device 1 as described above.

  As shown in the drawing, the PC 2 has a general configuration including a control unit 21 including a CPU and its peripheral circuits, a RAM 22, a storage device 23, an operation unit 24, a video adapter 25, a VRAM 26, a display device 27, a communication unit 28, and the like. is there.

  The video adapter 25 generates a display video signal (RGB signal) and outputs it to a display device 27 including an LCD and its driving circuit. The VRAM 26 stores the display image data generated by the video adapter 25 as needed. To do. The video adapter 25 is connected to the projector 3 via an RGB cable or the like (not shown), and outputs display image data to the projector 3 via the RGB cable in accordance with an instruction from the control unit 21.

  The communication unit 28 is a receiving unit of the present invention including a communication interface circuit based on a data transmission method to which the communication unit 15 of the pointing device 1 corresponds. The communication unit 28 receives the coordinate position sent from the pointing device 1 and sends it to the control unit 21. send.

  The operation unit 24 is a plurality of keys provided on the PC 2 main body, a mouse connected to the main body, the RAM 22 is a general-purpose work memory used for the control unit 21, and the storage device 23 has a storage capacity. It is a relatively large hard disk.

  The storage device 23 stores various application programs. In particular, in the present embodiment, the communication program for enabling the communication unit 28 to receive the coordinate position from the pointing device 1 and the control unit 21 function as the position mark display control means of the present invention, and will be described later. Stores a program for performing

  Next, an operation according to the present invention of the pointing device 1 and the PC 2 when the user uses the pointing device 1 in a state where an arbitrary display image is projected on the screen S by the projection system will be described. The pointing device 1 is used with the photographing lens 11 side facing the screen S. Here, for example, the following operation is performed only when the user presses a predetermined operation button or while the predetermined operation button is pressed.

  That is, FIG. 2 is a flowchart showing the operation of the pointing device 1 in the use state. The pointing device 1 captures the screen S at a fixed frame period (for example, 30 fps) when the operation starts, and FIG. The captured image 100 as shown is acquired (step SA1). Next, the projected image area 101 which is a part of the captured image 100 is recognized, and the coordinate positions of the stationary feature points indicating the four corners A, B, C, D, that is, the projected image area 101 are acquired (step SA2). . The projected image area 101 is recognized by, for example, performing straight line recognition on the captured image 100, detecting a plurality of rectangular areas partitioned along the recognized plurality of straight lines, and the rectangular area having the largest area among them. This is performed by an arbitrary method such as acquiring a contour line.

  Here, when coordinate positions can be acquired for all of the four corners A, B, C, and D (YES in step SA3), conversion parameters for transforming (correcting) the projection image region 101 into a normal form, that is, conversion matrix A coefficient is obtained and stored in the memory 16 (step SA4). The conversion parameter obtained here is a parameter used for arbitrary coordinate conversion for performing distortion correction known as, for example, oblique conversion, and is a coefficient of a conversion matrix. In addition, when performing multiple types of coordinate transformation, the coefficient used for each coordinate transformation is calculated | required.

  Then, the coordinate position P of the display image 200 corresponding to the center O (reference point of the present invention) of the captured image 100 is calculated using the conversion parameter (step SA5). Specifically, in order to reduce the burden of calculation processing, only the coordinate position of the center O of the captured image 100 is subjected to coordinate conversion, and the coordinate position P when the converted coordinate position is mapped to the coordinate space of the display image 200 is converted. Ask for. FIG. 4B shows an example in which the pixel size of the display image 200 is 1024 × 768 (XGA) and the coordinate position P corresponding to the center O of the captured image 100 in the coordinate space is (742, 379). It is. Thereafter, the calculated coordinate position is transmitted to the PC 2 as an instruction position by the user (step SA6).

  Further, while the above processing is repeated, for example, the orientation (instructed position) of the photographic lens 11 changes, and the entire projection image does not fit within the angle of view of the photographic image 100. In step SA2, the four corners A, B, C , D cannot be acquired (NO in step SA3), the conversion parameter used in the previous coordinate conversion is read from the memory 16 (step SA7). In step SA5 described above, coordinate conversion using the read conversion parameter is performed, that is, the coordinate position P of the display image 200 corresponding to the center O of the captured image 100 is calculated according to the same conversion conditions as the previous time, It transmits to PC2 as an instruction | indication position by a user (step SA6).

  On the other hand, FIG. 3 is a flowchart showing the operation of the PC 2 during use of the pointing device 1. When the PC 2 receives the coordinate position (instructed position) described above from the pointing device 1 (YES in step SB1), the PC 2 combines a pointer having a predetermined shape with the received coordinate position P separately from the display image being output to the projector 3. A new display image is generated (step SB2), and the display image being output is switched to a new projection image (step SB3). Thereafter, the same processing is repeated. As a result, a pointer is displayed on the projected image on the screen S at a position designated by the user, and the pointer moves according to the orientation of the pointing device 1.

  As described above, in the present embodiment, the user can instruct an arbitrary position of the projection image only by changing the direction of the pointing device 1, that is, by the same operation as in the case of using the laser pointer. In addition, since the PC 2 displays a pointer at the position designated by the user in the projection image, the user can confirm the position in the projection image designated by the user on the projection image. The pointing device 1 can ensure the same feeling of use as the laser pointer.

  In addition, since the pointing position of the projection image (the coordinate position of the display image 200) can be specified (detected) only by the pointing device 1 and supplied to the PC 2, a conventional laser pointer is used for the PC 2 and the projector 3. It is not necessary to provide hardware or software for performing the specified position specifying operation based on the comparison between the projected image and the captured image, which is necessary in some cases. Further, even when the projected image changes frequently, the designated position can be specified, and it is possible to cope with the case where the projected image is a moving image.

  Therefore, it is possible to specify the designated position in the projection image without imposing an excessive burden physically and economically on the projection system while ensuring the workability of the user.

  Further, in the present embodiment, even when any of the four corners A, B, C, and D of the projection image (projection image region 101) is not within the photographing field angle of the pointing device 1, the same as the previous time. By specifying the designated position according to the conversion condition, the designated position in the projection image can be specified. Therefore, for example, even when the display image (projected image) 200 is switched to an image with a low background color luminance and a difficult contour from the captured image 100 during use of the pointing device 1, it is possible to cope with it. .

  In the present embodiment, the case where the center O of the captured image 100 is used as a reference point and the coordinate position P of the display image 200 corresponding to the center O is used as the designated position has been described. Other coordinate positions can be used as reference points. However, the calculation required for the coordinate conversion described above becomes easier when the center O of the captured image 100 is used as the reference point.

  In addition, the pointing device 1 acquires the coordinate positions of the four corners A, B, C, and D of the projection image region 101 from the captured image 100, and based on the acquired coordinate positions of the center O of the captured image 100, the coordinate position of the display image 200 is obtained. It is assumed that P is converted into the indicated position, but if the coordinate conversion indicates the projected image area 101 regardless of the change in the content of the display image 200, the four corners A, B, C, You may make it carry out based on the coordinate position of other stationary feature points other than D.

  For example, when the brightness of the background color of the display image 200 is low, it is difficult to recognize the outline of the projection image itself from the captured image 100. Therefore, it is possible to cause the PC 2 to perform a process of adding a frame to the outer peripheral edge of the display image 200 in advance, and to cause the pointing device 1 to recognize the frame and to perform the above-described coordinate conversion based on the coordinate positions of the four corners. Good.

  Further, the stationary feature point does not have to directly indicate the projected image area 101. For example, when a whiteboard is used as the screen S, the outer frame of the whiteboard, not the projected image area 101, is taken from the captured image 100. 4 portions, that is, portions that are always present outside the projection image region 101 and indirectly indicate the projection image region 101. In that case, even if a black background presentation material (display image) is projected onto the whiteboard, the designated position in the projection image can be specified.

  However, in that case, correction values for correcting the coordinate positions of the four corners of the outer frame of the whiteboard to the coordinate positions of the four corners A, B, C, and D of the projected image area 101 are previously determined by a predetermined calibration operation. In the coordinate conversion described above, the coordinate positions of the four corners of the outer frame of the whiteboard need to be corrected once with the correction values, and coordinate conversion based on the corrected coordinate positions must be performed. As the calibration operation, for example, the pointing device 1 transmits a signal instructing the PC 2 to display an operation for displaying predetermined marks at the four corners of the projection image in response to a predetermined button operation. While the predetermined marks are displayed at the four corners, the offset amounts of the four corners A, B, C, and D of the projected image area 101 with respect to the four corners of the outer frame of the whiteboard are acquired from the captured image captured in that state, and the correction values are obtained. It can be set as

(Embodiment 2)
Next, a second embodiment of the present invention will be described. In the present embodiment, in the same configuration as that shown in FIG. 1, the calculation unit 14 of the pointing device 1 functions as a recognition unit of the present invention, and the control unit 21 of the PC 2 has a display control unit of the present invention. , The pointing device 1 and the PC 2 operate as follows. Here, for convenience, the operation on the PC 2 side will be described first.

  That is, FIG. 5 is a flowchart showing the operation of the PC 2 in the present embodiment. Also in the present embodiment, PC 2 receives the coordinate position (instructed position) described above from pointing device 1 (YES in step SB101), in the same way as in the first embodiment, separately from the projection image being output to projector 3. Then, a new projection image is generated by synthesizing a pointer having a predetermined shape at the received coordinate position P (step SB102), and the projection image being output is switched to a new image by synthesizing the pointer (step SB103), and then step SB101. Return to.

  On the other hand, when a predetermined request signal different from the coordinate position from the pointing device 1 is received (NO in step SB101, YES in step SB104), a specific pattern of a certain time interval (for example, 1/5 second interval) is set. After the display is set (step SB105), a new display image is generated by synthesizing a specific pattern separately from the display image currently being output to the projector 3 (regardless of the presence or absence of the pointer) (step SB106). Thereafter, the process proceeds to step SB103, the display image being output is switched to a new image obtained by synthesizing the specific pattern, and the process returns to step SB101.

  The specific pattern to be synthesized in step SB106 is a pattern prepared in advance to indicate the coordinate reference in the captured image acquired by the pointing device 1, and in this embodiment, as shown in FIG. It is a grid pattern which shows orthogonal coordinates.

  Further, the PC 2 does not display the specific pattern at that time (step SB107) while neither the coordinate position nor the request signal is transmitted from the pointing device 1 (NO in both step SB101 and step SB104). NO) and returns to step SB101 as it is. This operation is the same when the display setting of the specific pattern in step SB105 is not performed. On the other hand, if the time is the display timing of the specific pattern (YES in step SB107), a new display image synthesized with the specific pattern is generated (step SB106), and is being output at that time The display image is switched to a new image (step SB103), and thereafter the above-described operation is repeated.

  On the other hand, FIG. 6 is a flowchart showing the operation of the pointing device 1. Also in the present embodiment, the pointing device 1 captures the screen S at a fixed frame period, acquires the captured image (step SA101), and then transmits a request signal for requesting the display of the specific pattern described above to the PC 2. If not (NO in step SA102), a projected image region that is a part of the acquired captured image is recognized, and the coordinate positions of the four corners are acquired (step SA103).

  If the captured image 100 as shown in FIG. 4 has been acquired in step SA101 and all the coordinate positions of the four corners A, B, C, and D of the projected image area 101 have been acquired (YES in step SA104). As in the first embodiment, conversion parameters for deforming (correcting) the projection image area 101 into a normal form are obtained (step SA105). Then, using the conversion parameter, the coordinate position P of the display image 200 corresponding to the center O of the captured image 100 is calculated (step SA106), and then the calculated coordinate position is transmitted to the PC 2 as an instruction position by the user (step SA106). Step SA107). Thus, a pointer is displayed on the projected image on the screen S at a position designated by the user.

  On the other hand, when the determination result in step SA104 is NO and any of the coordinate positions of the four corners of the projection image area cannot be acquired in step SA102, that is, in step SA101, for example, as shown in FIG. When the photographed image 100 is obtained such that only the upper left corner A of the image (display image) 200 is captured, a request signal for requesting the display of the specific pattern (square pattern) is transmitted to the PC 2 ( Step SA108), the process returns to step SA101. Thereby, on the screen S, the display image 200 in a state where the specific pattern is synthesized as shown in FIG. 8A is displayed at regular time intervals separately from the display image 200 as shown in FIG. The Rukoto.

  Once the request signal is transmitted (YES in step SA102), the specific pattern described above is recognized from the captured image 100 acquired in step SA101 (step SA109). When the specific pattern can be recognized (YES in step SA110), the positional relationship data indicating the positional relationship between the original projection image without the specific pattern being synthesized and the specific pattern is acquired from the captured image 100, Store in the memory 16 (step SA111). The positional relationship data indicates the positional relationship between the feature points of the original projection image (display image) in which the specific pattern is not synthesized and the feature points of the specific pattern. For example, the original display image 200 is shown in FIG. In the case of a document image as shown in (a), the coordinate position of one of the four corners of each of the rectangular areas 200a and 200b showing the character layout shown in FIG. In the case where the original projected image is a natural image or the like, the coordinate position of the part having a characteristic brightness, color, etc. and the specific pattern It is data indicating a positional relationship with the coordinate position of one or a plurality of feature points.

  Subsequently, based on the coordinate reference indicated by the specific pattern, a conversion parameter for transforming the projection image region 102 into a normal form is obtained (step SA112), and the display image corresponding to the center O of the captured image 100 using the conversion parameter. The coordinate position P of 200 is calculated (step SA106). FIG. 8B shows an example in which the pixel size of the display image 200 is 1024 × 768 (XGA) and the coordinate position P corresponding to the center O of the captured image 100 in the coordinate space is (355, 478). It is. Thereafter, the calculated coordinate position is transmitted to the PC 2 as an instruction position by the user (step SA107). Thus, a pointer is displayed on the projected image on the screen S at a position designated by the user.

On the other hand, when the determination result in step SA110 is NO and the specific pattern cannot be recognized from the captured image 100, that is, when the display image 200 in which the specific pattern is combined is not displayed at the capturing timing in step SA101. The presence / absence of storage of the positional relationship data described above is confirmed. If it is not stored (NO in step SA113), the process directly returns to step SA101.
If positional relationship data has already been stored (YES in step SA113), a non-rectangular projection image area corresponding to a part of the projection image is recognized, and the character layout and one or more features described above are recognized. While confirming the position of the point, the specific pattern is restored in the projection image area based on the position relation data and the above-described positional relationship data (step SA114). Then, after obtaining a conversion parameter for transforming the projected image region 102 into a normal form based on the coordinate reference indicated by the restored specific pattern (step SA112), the conversion parameter is used to obtain the center O of the captured image 100. The coordinate position P of the corresponding display image 200 is calculated and transmitted to the PC 2 as an instruction position by the user (steps SA106 and SA107). Thus, a pointer is displayed on the projected image on the screen S at a position designated by the user.

  As described above, in the present embodiment, when the entire projected image is within the shooting angle of view of the pointing device 1, the projected image area 101 is extracted from the captured image 100 as in the first embodiment. It is possible to recognize and specify the designated position designated by the user based on the positional relationship between the coordinate positions of the four corners A, B, C, and D and the center O of the captured image. Furthermore, when the entire projection image does not fit within the shooting angle of view of the pointing device 1 from the beginning of use, the specific pattern is synthesized and projected on the display image 200, and the specific pattern is recognized. As a result, it is possible to specify the position indicated by the user even in such a case. Therefore, even when the user is relatively close to the screen S or when the screen S and the projected image are extremely large, it is possible to cope with it.

  In addition, since the display process of the specific pattern in the PC 2 is performed only when there is a request from the pointing device 1, the display of the specific pattern can be stopped to the minimum necessary.

  Further, positional relationship data indicating the positional relationship between the original projected image and the specific pattern is stored, and during the period when the specific pattern is not displayed, the specific pattern is restored based on the feature points of the projected image. The position pointed by the user was specified. Therefore, the display interval of the specific pattern can be lengthened.

  In the present embodiment, the description has been given of the case where the PC 2 continuously displays the specific pattern at a certain time interval when the pointing device 1 requests the specific pattern to be displayed. It may be performed only for a predetermined period (for example, several minutes). Further, although the center O of the photographed image 100 is used as a reference point, a coordinate position other than the center O can be used as a reference point as in the first embodiment.

(Embodiment 3)
Next, a third embodiment of the present invention will be described. As shown in FIG. 10, the present embodiment relates to a pointing device 1 having a configuration in which a gravity detector 18 is added to the configuration described in FIG. The gravity detection unit 18 is a part that detects the direction of gravity and sends the detection result to the calculation unit 14, and includes a gravity sensor, an amplification circuit for the detection signal, a signal processing circuit after amplification, and the like. In the present embodiment, the gravitational direction sent from the gravity detection unit 18 to the calculation unit 14 is the optical axis of the photographing lens 11 based on the state where the upper and lower sides of the photographed image taken by the image sensor 12 are horizontal. It is information which shows the rotation angle of rotation.

  And in the pointing device 1 of this Embodiment, the said calculating means functions as a specific means of this invention at the time of the use, It operate | moves as follows.

  That is, as shown in the flowchart of FIG. 11, also in the present embodiment, the pointing device 1 captures the screen S at a fixed frame period, acquires a captured image (step SA201), and detects the direction of gravity at that time. (Step SA202). Next, after specifying the projection image area from the acquired captured image (step SA203), the top and bottom direction is corrected by rotating the projection image area according to the gravitational direction at the time of shooting detected in step SA201 (step SA203). SA204).

  FIG. 12A shows a photographed image 100 acquired in step SA201, and FIG. 12B shows an example of a state in which the projected image area 101 is rotated.

  Then, the coordinate positions of the four corners A, B, C, and D of the projected image area 101 after correction are acquired (step SA205), and thereafter, all the four corners A, B, C, and D are obtained as in the first embodiment. When the coordinate position can be obtained (YES in step SA206), a conversion parameter for deforming (correcting) the projection image area 101 into a normal form is obtained and stored in the memory 16 (step SA207). Using the conversion parameter, the coordinate position P of the display image 200 corresponding to the center O of the captured image 100 is calculated (step SA208). Thereafter, the calculated coordinate position is transmitted to the PC 2 as an instruction position by the user (step SA209).

  If the coordinate position of any of the four corners A, B, C, and D cannot be acquired in step SA205 while the above processing is repeated (NO in step SA206), the conversion parameters used in the previous coordinate conversion are stored in the memory. 16 (step SA210), coordinate conversion using the read conversion parameters is performed (step SA208), the coordinate position P of the display image 200 corresponding to the center O of the captured image 100 is calculated, and this is indicated by the user The position is transmitted to the PC 2 (step SA209).

  As described above, in the present embodiment, before the coordinate O of the center O of the captured image 100 is converted to the coordinate position P in the coordinate space of the display image (projected image) corresponding to the center O, the projected image area 101 is previously set in the vertical direction. Is corrected by the gravitational direction at the time of shooting, so that even when the apparatus body is tilted in the rotation direction around the optical axis of the shooting lens 11 at the time of shooting, the indicated position in the projection image is accurately specified. be able to.

  Therefore, it is convenient for the user to use the pointing device 1 without being aware of the vertical direction of the main body of the device. In other words, since the pointing device generally needs to point to a specific part, it is often in the shape of an elongated shape. In consideration of ease of holding, it is generally necessary to make it as thin as possible. It becomes a shape. For this reason, if it is necessary to be aware of the top-and-bottom direction at the time of use, it is very difficult to use. In particular, when the user explains in a half-body posture, it is expected that the pointing device 1 is frequently used in a state of being tilted by about 90 degrees. It can be accurately identified.

  Here, in the present embodiment, the gravity detecting unit 18 is provided in the pointing device 1 and the vertical direction (directly inclined) of the captured image is specified based on the direction of gravity at the time of shooting. The vertical direction of the captured image may be specified without providing the gravity detection unit 18. Hereinafter, the modification is demonstrated.

  First, as a first modification, a sign such as a specific mark or pattern is added in advance to a predetermined position of the projection image, and the pointing device 1 recognizes the sign from the photographed image, depending on the display position. This specifies the vertical direction of the captured image. In that case, the pointing device 1 is caused to recognize the sign and determine the display position in, for example, step SA202 during the operation shown in FIG. 11, and in step SA204, the projection image area is rotated according to the display position. What should I do? FIG. 13 is a diagram illustrating an example of a captured image 100 (a) and a projected image region 101 (b) after rotation when the sign is a line 300 along the lower side of the projected image. It should be noted that the projection position of the sign is fixed or may be variable. When the sign is fixed, the position needs to be stored in the pointing device 1. It is necessary to send.

  In addition, as a second modification, a specific mark or a mark such as a pattern having a predetermined shape (such as an arrow) indicating a vertical direction is added to a part of the projection image in advance, and the pointing device 1 Further, the sign is recognized from the photographed image, and the vertical direction of the photographed image is specified by the direction of the sign. In that case, for example, the recognition of the sign and the determination of the direction of the sign may be performed in step SA202 described above, and the rotation processing of the projection image area corresponding to the direction of the sign may be performed in step SA204.

  In any of the above-described modifications, the sign may be projected regularly or intermittently. However, when the sign is projected intermittently, the sign is projected and not projected. It is possible to clearly and very easily determine the sign as a difference only by comparing the captured images. Further, a logo mark of a company or the like may be used as the specific mark used as the sign.

  As a third modification, when there is a recognizable character in the photographed image 100, it is recognized and the vertical direction of the photographed image is specified by the rotation state of the recognized character. In that case, in the case of complex characters such as kanji, not only does the recognition process take time, but also the probability of recognition errors increases, so it is preferable to limit the recognition targets to, for example, alphanumeric characters and hiragana. The pointing device 1 stores a reference character image having a different rotation state, for example, every 90 °, for each character to be recognized. For example, in step SA202 described above, a character string portion is extracted from the captured image. Character recognition based on criteria with different rotation states in units corresponding to one character, and if the recognition accuracy is low, the same character recognition is performed for the next part. When the highly accurate recognition result is obtained, the process for specifying the vertical direction of the captured image is performed, and the projection image area is rotated according to the rotation state in step SA204. That's fine.

  FIG. 14 is a diagram illustrating an example of a captured image 100 (a) and a rotated projected image area 101 (b) when the recognition target is hiragana and the character “a” can be recognized.

  In the above description, the projection image region 101 is first corrected (rotated) in accordance with the vertical direction of the captured image 100 and then the coordinate conversion described above is performed. However, the present invention is not limited to this. After the coordinate conversion based on the four corners A, B, C, and D of the projection image area 101 is performed first, the coordinate value obtained by the coordinate conversion is determined according to the vertical direction of the captured image 100. Correction may be performed by rotating the coordinate space, and the corrected coordinate value may be used as the designated position.

(Embodiment 4)
Next, a fourth embodiment of the present invention will be described. In the present embodiment, in the configuration shown in FIG. 1, the operation unit 17 is given a predetermined process such as enlargement of an arbitrary portion instructed on a projection image, emphasis or enlargement of instructed characters, or display of a line drawing. Operation buttons for command transmission for causing the PC 2 to perform processing are provided, and the memory 16 during use is in the first to fourth directions for storing the data (coordinate values) of the indicated position detected a plurality of times. This is related to a pointing device in which Buf1, Buf2, Buf3, and Last shown in FIG.

  In the pointing device according to the present embodiment, the operation unit 17 functions as the operation unit of the present invention, and the arithmetic unit 14 functions as the averaging unit and the prohibition unit of the present invention. To do.

  That is, as shown in the flowchart of FIG. 16, the pointing device of the present embodiment performs initialization to erase data in each buffer area Buf1, Buf2, Buf3, Last at the beginning of the operation (step SA301). The screen S is photographed at a fixed frame period to obtain a photographed image (step SA302).

  Next, it is determined whether or not the operation button is pressed and data (coordinate values) is stored in the fourth buffer area (Last) at that time (step SA303). Here, since the fourth buffer area (Last) is empty at the beginning of the operation, that is, at the time of the first shooting, the determination result is NO. Therefore, the pointing coordinate calculation for detecting the user's designated position is performed as it is. The calculation result is detected as the current sample data R0 (step SA305). Note that the pointing coordinate calculation is performed, for example, by the method described above in the first embodiment.

  Next, it is determined whether or not the difference (coordinate distance) between the calculation result and the coordinate value stored in the fourth buffer area (Last) exceeds a threshold value (Thresh) (step SA306). At the beginning of the operation, since the fourth buffer area (Last) is empty and the determination result is NO, the data in the second and third buffer areas (Buf2, Buf3) are immediately transferred to the first and While moving to the second buffer area (Buf1, Buf2), the coordinate value R0 calculated previously is stored in the third buffer area (Buf3) (step SA309). That is, the data of the third buffer area (Buf3) is set as a newly detected coordinate value, the data of the second buffer area (Buf2) is set as the previous coordinate value, and the data of the first buffer area (Buf1) is used in advance. The coordinate value of.

  Furthermore, since the first and second buffer areas (Buf1, Buf2) are empty at the beginning of the operation (both steps SA310 and SA310 are YES), the third buffer area is added to the fourth buffer area (Last). The data of (Buf3) is migrated (step SA312). Thereafter, the data of the fourth buffer area (Last), that is, the newly detected coordinate value R0 is transmitted as it is to the PC 2 as the current coordinate position indicating the position indicated by the user (step SA315), and then the process returns to step SA302. Repeat the process described above.

  In the processing associated with the second shooting, since the previous coordinate value is stored in the second buffer area (Buf2) (YES in step SA310 and NO in step SA311), the newly detected coordinate value is used. 3 times, weighting the previous coordinate value twice, weighting and averaging the coordinate values of both, and storing the coordinate value in the fourth buffer area (Last) (step SA314), The coordinate value is transmitted to the PC 2 as the current coordinate position indicating the position indicated by the user (step SA315).

  In the processing associated with the third and subsequent shootings, since the previous coordinate value is stored in the first buffer area (Buf1) (NO in step SA310), the newly detected coordinate value is tripled. In addition, the previous coordinate value is doubled and a weight is added to multiply the previous coordinate value by 1, and then the last three coordinate values are weighted and averaged, and the coordinate value is stored in the fourth buffer area ( Last) is stored (step SA313), and the coordinate value is transmitted to the PC 2 as the current coordinate position indicating the position indicated by the user (step SA315).

  As a result, when the user designates a specific part of the projection image with the pointing device, it is possible to suppress a change in the designated position when the device body is swinging, and to reduce the wobbling of the pointer on the projection image. Can do.

  On the other hand, while the above processing is repeated, the difference between the new coordinate value (R0) detected at any shooting timing and the previous coordinate value stored in the fourth buffer area (Last) is the threshold value ( If it exceeds (Thresh) (YES in step SA306), the stored data in the first and second buffer areas (Buf1, Buf2), that is, the previous and previous coordinate values are erased, and the third and fourth buffers A new coordinate value is stored in the area (Buf3, Last) (step SA307). Thereafter, the coordinate value is transmitted to the PC 2 as the current coordinate position indicating the position designated by the user (step SA308). That is, in the case where a coordinate position that is more than a certain distance from the previous designated position is detected, the above-described weighted average process in step SA313 or step SA314 is cancelled.

  As a result, when the designated position is moved intentionally by the user, it is possible to move the pointer in response immediately.

  When the operation button is pressed at any shooting timing and the coordinate value is stored in the fourth buffer area (Last) at that time (YES in step SA303), the operation button described above is used. Along with the input information, that is, the command signal for causing the PC 2 to perform a predetermined process, the coordinate value is transmitted to the PC 2 as the current coordinate position indicating the position indicated by the user (step SA304). That is, when the operation button is pressed, the designated position detected at the photographing timing immediately before the operation button is transmitted as the designated position corresponding to the command signal.

  As a result, when the user operates the operation button, it is possible to prevent the instruction position from being changed due to a hand movement accompanying the operation. It is possible to cause the PC 2 to perform predetermined processing such as enlargement, emphasis and enlargement of the designated character, or display of line drawings.

  In the present embodiment, the coordinate value data used in the above-described weighted average process (step SA313) for suppressing the change of the indicated position when the apparatus main body is swinging is the data for the latest three times. However, data for other times may be used.

  In addition, the threshold value (Thresh) used in the determination in step SA306 described above is a fixed value. However, the present invention is not limited to this. The threshold value may be dynamically changed in accordance with the change of. For example, the tendency of the movement amount within a certain period until the previous time may be confirmed, and the threshold value may be increased as the movement amount is increased, and the threshold value may be decreased as the movement amount is decreased.

  When the operation button is pressed, the instruction position detected at the photographing timing immediately before the operation is transmitted as the instruction position corresponding to the instruction signal. . That is, it is generally known that it takes several tens of msec from when the operation button is pressed down until it is detected. Therefore, the designated position detected a certain time before the operation detection time of the operation button (for example, before 50 msec) may be transmitted as the designated position corresponding to the command signal.

  Further, as described above, a process of transmitting an instruction position detected in the past as an instruction position corresponding to the instruction signal, and a weighted average process for suppressing a change in the instruction position when the apparatus main body is swinging, The process of canceling it as necessary is not necessarily performed in a lump, and each process may be performed independently. Further, as a matter of course, these processes may be performed during the operations described in the first to third embodiments.

  On the other hand, in the first to fourth embodiments described above, the pointer is displayed at the designated position of the projection image specified by the pointing device 1, but it is displayed only on the display device 27 of the PC 2, not the projection image. Or may be displayed on both sides.

  Further, only the case where the external information processing apparatus according to the present invention is the PC 2 has been described. However, the projector 3 has received the pointing position described above from the pointing apparatus 1, the request signal described in the second embodiment, If the configuration is such that the command signal described in the embodiment can be received, the PC 2 side processing described above using the projector 3 as the external information processing apparatus of the present invention, that is, the pointer display processing and the display processing of a specific pattern such as a grid pattern Etc. may be performed. Furthermore, it may be realized in a projection system that does not include the PC 2.

  In addition, the pointing device of the present invention is not limited to the projected image that is enlarged and displayed on the screen by the projector 2, and may be any device that indicates an arbitrary position of the display image. For example, it can be used to indicate an arbitrary position on a display screen of a plasma display (PDP) or a liquid crystal display (LCD). Furthermore, in a personal computer, for example, it can be used as a device for moving a mouse cursor, or can be used as a device for inputting absolute position information of an arbitrary position in a display screen of a monitor to the personal computer. Can do.

It is a block diagram of the projection system common to each embodiment of this invention. It is a flowchart which shows operation | movement of the pointing device in 1st Embodiment. It is a flowchart which shows operation | movement of PC in the same embodiment. It is explanatory drawing which shows the content of the coordinate transformation process by a pointing device. It is a flowchart which shows operation | movement of PC in 2nd Embodiment. It is a flowchart which shows operation | movement of the pointing device in the embodiment. It is a figure which shows the example of the picked-up image in the state where a projection image does not fit. It is explanatory drawing which shows the content of the coordinate transformation process based on the grid pattern by a pointing device. It is explanatory drawing which shows the feature point in case a display image is a document image. It is a block diagram of the pointing device which shows 3rd Embodiment. It is a flowchart which shows operation | movement of the pointing device in the embodiment. It is the figure which showed an example of the picked-up image and the projection image area | region after rotation. It is a figure corresponding to FIG. 12 which shows the 1st modification in the embodiment. It is a figure corresponding to FIG. 12 which shows the 3rd modification in the embodiment. It is a conceptual diagram which shows the buffer area ensured in the memory of a pointing device in 4th Embodiment. It is a flowchart which shows operation | movement of the pointing device in the embodiment.

Explanation of symbols

1 Pointing device 2 PC
DESCRIPTION OF SYMBOLS 3 Projector 11 Shooting lens 12 Image sensor 13 Image processing part 14 Calculation part 15 Communication part 16 Memory 17 Operation part 21 Control part 22 RAM
23 Storage Device 24 Operation Unit 25 Video Adapter 26 VRAM
27 Display Device 28 Communication Unit 100 Captured Image 101 Projected Image Area 102 Projected Image Area 200 Projected Image 200 Displayed Image O Center P Coordinate Position S Screen

Claims (16)

A pointing device for indicating an arbitrary position on a display screen,
Photographing means for photographing the display screen;
Obtaining means for obtaining coordinate positions of a plurality of stationary feature points indicating a display screen area in a photographed image photographed by the photographing means;
The coordinate position of the predetermined reference point in the photographed image photographed by this photographing means corresponds to when the display screen area in the photographed image is transformed into a normal form based on the coordinate positions of the plurality of stationary feature points. A conversion means for converting the coordinate position of the display screen;
Transmitting means for transmitting the coordinate position of the reference point converted by the conversion means to the external information processing apparatus as the indicated position on the display screen ,
The converting unit performs coordinate conversion of the reference point based on the coordinate positions of the plurality of stationary feature points acquired by the acquiring unit, and cannot acquire any of the coordinate positions of the plurality of stationary feature points. In this case, the pointing device performs coordinate conversion of the reference point according to the same conversion condition as the previous time based on the coordinate position acquired by the acquisition means .   2. The pointing device according to claim 1, wherein the indication position transmitted to the external information processing apparatus by the transmission means is a display position of a predetermined indication position mark to be displayed on the display screen.   The pointing device according to claim 1, wherein the reference point is a center of the captured image.   The pointing device according to claim 1, wherein the stationary feature points are four corners of a frame existing on an outer peripheral edge of a display image displayed on the display screen.   The stationary feature point exists outside the display screen area,
  The converting unit corrects the coordinate position of the feature point acquired by the acquiring unit using a correction value set in advance by a predetermined setting operation, and based on the corrected coordinate position, Perform coordinate transformation
  The pointing device according to claim 1.   A pointing device for indicating an arbitrary position on a display screen,
  Photographing means for photographing the display screen;
  Obtaining means for obtaining coordinate positions of a plurality of stationary feature points indicating a display screen area in a photographed image photographed by the photographing means;
  The coordinate position of the predetermined reference point in the photographed image photographed by this photographing means corresponds to when the display screen area in the photographed image is transformed into a normal form based on the coordinate positions of the plurality of stationary feature points. A conversion means for converting the coordinate position of the display screen;
  Transmitting means for transmitting the coordinate position of the reference point converted by the converting means to the external information processing apparatus as the indicated position on the display screen;
  Recognizing means for recognizing the specific pattern from the captured image in a state in which a known specific pattern indicating the coordinate reference of the display screen is displayed on the display screen,
  When any of the coordinate positions of the plurality of stationary feature points required for the coordinate conversion of the reference point by the conversion unit cannot be acquired by the acquisition unit, the transmission unit displays the specific pattern on the display screen. A request signal for requesting to the external information processing device,
  The pointing device according to claim 1, wherein the conversion means performs coordinate conversion of the reference point based on a coordinate reference indicated by the specific pattern recognized by the recognition means.   The pointing device according to claim 6, wherein the specific pattern is an image displayed on the display screen at regular time intervals.   Positional relationship acquisition means for acquiring positional relationship information indicating a positional relationship between a feature point of a display image in the display screen region and a feature point of the specific pattern in the display screen region;
  Storage means for storing the positional relation information acquired by the positional relation acquisition means;
  With
  The conversion means stores the characteristic points of the display image in the display screen area in the storage means during the coordinate conversion of the reference point based on the captured image in a state where the specific pattern is not displayed on the display screen. The coordinate conversion of the reference point is performed based on the coordinate reference indicated by the specific pattern including the feature point in the positional relationship indicated by the positional relationship information.
  The pointing device according to claim 7.   A specifying means for specifying the vertical direction of the display screen in the captured image;
  Character recognition means for recognizing characters present in the captured image;
  Further comprising
  The specifying means specifies the vertical direction of the display screen in the photographed image based on the rotation state in the photographed image of the character recognized by the character recognition means,
  The converting means converts the coordinate position of a predetermined reference point in the captured image into the captured image.
9. The display screen area corresponding to when the display screen area is transformed into a normal form, and converted into a coordinate position of the display screen according to the orientation specified by the specifying means. Pointing device.   Further comprising averaging means for averaging the coordinate positions of the latest multiple times coordinate-converted by the converting means;
  The transmission means transmits the coordinate position averaged by the averaging means to the external information processing apparatus as an indicated position on the display screen.
  The pointing device according to any one of claims 1 to 9, wherein   When the distance between the current coordinate position coordinate-transformed by the conversion means and the previous coordinate position exceeds a predetermined threshold, the averaging of the latest multiple coordinate positions by the averaging means is performed. With prohibition means to prohibit,
  The transmission means transmits the current coordinate position transformed by the conversion means to the external information processing apparatus as the designated position on the display screen.
  The pointing device according to claim 10.   An operation means for instructing the external information processing apparatus to perform a predetermined operation;
  The transmission unit transmits a predetermined command signal to the external information processing apparatus in response to the operation of the operation unit, and the coordinates converted by the conversion unit before the time when the operation unit was operated. A position is transmitted to the external information processing apparatus as an instruction position corresponding to the command signal.
  12. The pointing device according to claim 1, wherein   An indicated position specifying method for indicating an arbitrary position of a display screen instructed by a user,
  A photographing process for photographing the display screen;
  An acquisition step of acquiring coordinate positions of a plurality of stationary feature points indicating a display screen area in the captured image captured by the imaging step;
  Based on the coordinate positions of the predetermined reference points in the photographed photographed image, the corresponding coordinate positions of the display screen when the display screen area in the photographed image is transformed into a normal form. A conversion process for converting to
  A transmitting step of transmitting the converted coordinate position to the external information processing apparatus;
  Including
  The converting step performs coordinate conversion of the reference point based on the coordinate positions of a plurality of stationary feature points acquired by the acquiring step, and any of the coordinate positions of the plurality of stationary feature points cannot be acquired. In this case, the pointing position specifying method further includes a step of performing coordinate conversion of the reference point according to the same conversion condition as the previous time based on the coordinate position acquired by the acquisition step.   A computer having a pointing device that is used for indicating an arbitrary position on the display screen and includes a photographing unit and a transmission unit for photographing the display screen,
  Photographing processing for causing the photographing means to photograph the display screen;
  An acquisition process for acquiring coordinate positions of a plurality of stationary feature points indicating a display screen area in the captured image captured by the imaging process;
  The coordinate position of a predetermined reference point in the photographed image photographed by the photographing means corresponds to when the display screen area in the photographed image is transformed into a normal form based on the coordinate positions of the plurality of stationary feature points. A conversion process to convert the coordinate position on the display screen;
  A transmission process for causing the transmission means to transmit the converted coordinate position to the external information processing apparatus as the designated position on the display screen;
  And execute
  The conversion processing further performs coordinate conversion of the reference point based on the coordinate positions of the plurality of stationary feature points acquired by the acquisition step, and can acquire any of the coordinate positions of the plurality of stationary feature points. If not, a program that executes coordinate conversion of the reference point according to the same conversion conditions as the previous time based on the coordinate position acquired by the acquisition process.   An external information processing apparatus to which an instruction position instructed by the pointing device is transmitted from a pointing device used for instructing an arbitrary position on the display screen,
  Receiving means for receiving a request signal transmitted from the pointing device and requesting display on the display screen of a specific pattern indicating a coordinate reference of the display screen;
  Display control means for performing an operation for displaying the specific pattern on the display screen in response to reception of the request signal by the receiving means;
  An external information processing apparatus comprising:   A pointing position specifying device comprising: a pointing device used for pointing to an arbitrary position on a display screen; and an external information processing device that receives the pointing position on the display screen instructed by a user from the pointing device. ,
  The external information processing apparatus
  Receiving means for receiving a request signal transmitted from the pointing device and requesting display on the display screen of a specific pattern indicating a coordinate reference of the display screen;
  Display control means for performing an operation for displaying the specific pattern on the display screen in response to reception of the request signal by the receiving means;
  With
  The pointing device includes:
  Photographing means for photographing the display screen;
  Transmitting means for transmitting the request signal;
  Obtaining means for obtaining coordinate positions of a plurality of stationary feature points indicating the display screen area from a photographed image photographed by the photographing means;
  Based on the plurality of coordinate positions acquired by the acquisition means, the coordinate position of a predetermined reference point in the captured image captured by the imaging means corresponds to when the display screen area in the captured image is transformed into a normal form. A conversion means for converting the coordinate position of the display screen;
  Recognizing means for recognizing the specific pattern from the captured image in a state where a known specific pattern indicating the coordinate reference of the display screen is displayed on the display screen;
  With
  The transmitting unit displays the specific pattern on the display screen when any of the coordinate positions of the plurality of stationary feature points required for the coordinate conversion of the reference point by the converting unit cannot be acquired by the acquiring unit. A request signal for requesting to the external information processing device,
  The conversion means is recognized by the recognition means and requested by the transmission means when any of the coordinate positions of a plurality of stationary feature points required for coordinate conversion of the reference point cannot be obtained by the acquisition means. The coordinate conversion of the reference point is performed based on the coordinate reference indicated by the specific pattern displayed in response to the transmission of
  A pointing position specifying device characterized by that.

Images