JP5573908B2 - Pointing device, information transmission method thereof, and information transmission program - Google Patents

Description

The present invention relates to a pointing device suitable for use in an image projection system, an information transmission method thereof, and an information transmission program.

  Conventionally, for presentations such as introduction of business and product outlines, presentation of research contents, explanation of lecture contents, etc., an external information device such as a personal computer (hereinafter referred to as a PC) is connected to a projector and consists of characters and images. A projection system that projects various explanatory materials on an arbitrary projection surface (hereinafter referred to as a screen) and displays the enlarged image is used. In presentations, a laser pointer is often used for the purpose of a presenter (explainer or the like) pointing to a noticeable part of an image (projected image) projected on a screen.

  Here, when the laser pointer is used, if the locus of the light spot (indicated position) of the laser beam can be displayed on the projected image, underline and handwritten characters can be added to the projected image even during the presentation. It is convenient to write. In order to make it possible, for example, a camera is provided at the tip of the apparatus main body corresponding to the laser pointer instead of the laser light emitting unit, and the screen is sequentially photographed by such a camera, and the projected image range and the photographed image in the photographed image are displayed. Based on the positional relationship with the center, the position of the projected image is specified by the user, the coordinate data is output to an external information device such as a PC, and when a predetermined operation button is pressed In response to this, a camera-type pointing device having a configuration for outputting a predetermined command signal to a PC or the like has already been proposed by the present applicant (see Patent Document 1 below).

  Since the pointing position in the projection image by the user can be output to a PC or the like by using the pointing device, for example, a pointer (mark) corresponding to a light spot when a laser pointer is used is projected on the PC or the like. In addition, the pointer trajectory can be displayed as necessary, and underline and handwritten characters can be written on the projected image as necessary. Further, by sending a command signal together with the indicated position (coordinate data) in response to the operation of the operation button, for example, an arbitrary portion designated by the user on the projection image is enlarged, or the designated character is emphasized or Specific processing such as enlargement can be performed by a PC or the like.

JP 2007-257438 A

  By the way, if the pointing device is used as described above, it is possible to directly display the locus of the pointer and write to the projection image during the presentation, but the operation of the PC or the like that can be operated by the pointing device is as follows. As described above, this is merely a limitation related to the designated position in the projection image. Therefore, it is desirable that the pointing device can be used as a versatile input device such as a mouse for a PC, that is, an input device that can instruct a PC or the like an operation not related to the designated position in the projection image. In order to realize this, there were the following problems.

  In other words, in order to secure the function as a laser pointer and the function as an input device in the pointing device, and to use each function, the pointing device indicates the notable part of the projected image in the same manner as the laser pointer. It is necessary to inform the PC or the like of the occasional use purpose, such as whether it is used for the purpose of (designated position display) or as an input device such as a PC. Therefore, for example, if it is performed based on the operation of the PC or the like by the user or the operation of the operation button or the like of the pointing device, it is complicated for the user to change the purpose of use, that is, to switch the function to be used. The burden on the user is large. Further, when switching with the operation buttons, for example, in a configuration in which a plurality of operation buttons are provided, there is a problem that an erroneous operation by a user who is not accustomed to use is concerned.

The present invention has been made in view of such a conventional problem, and in a configuration having a function as a laser pointer and a function as a versatile input device, the user has a function according to the purpose of use. pointing device immediately becomes possible to use a desired function without switching operation, and the information transmission method, and to provide an information transmission program used for their implementation.

In order to solve the above-mentioned problem, in the pointing device according to the first aspect of the present invention, the position on the display screen is indicated, and the indicated position information indicating the indicated position on the display screen is transmitted to the external information processing apparatus. In the pointing device having a function to perform the above operation, a photographing unit used for photographing the display screen, a transmission unit that transmits operation information indicating an operation content of the pointing device to the external information processing device, and the designated position information are acquired. includes the indicated position obtaining means and the movement information acquisition section for acquiring movement information indicating a movement amount of the pointing device main body, and determining means for determining the operational status of the pointing device main body by the user, the of the determination means If the pointing device determines that the pointing device is not held toward the display screen, the transmitting means transmits the operation information. Information containing the moving information transmitted to the external information processing apparatus as the determination means, the image taken by the imaging means, whether the display screen area corresponding to the display screen is present By determining the operating state of the pointing device main body, and when the determining unit determines that the display screen area does not exist in the captured image captured by the capturing unit, the transmitting unit includes the operation information as the operation information. When the information including the movement information is transmitted to the external information processing apparatus, and the determination unit determines that the display screen area exists in the captured image captured by the imaging unit, the transmission unit Information including the indicated position information as information is transmitted to the external information processing apparatus .

  According to the pointing device and the information transmission method of the present invention, when the pointing device has a function as a laser pointer and a function as a versatile input device, the user can select a function according to the purpose of use. It is possible to immediately use a desired function without performing the switching operation. Also, the information transmission program of the present invention can be used to implement the pointing device and the information transmission method. In the external information processing apparatus of the present invention, the user of the pointing device can be used to confirm the position of the display screen instructed on the display screen by using the external information processing apparatus.

It is a block diagram of the projection system which shows embodiment of this invention. It is a flowchart which shows operation | movement of the pointing device in 1st Embodiment. It is explanatory drawing which shows the coordinate position in the coordinate space of a display image corresponding to the center of a picked-up image and a picked-up image. It is the figure which illustrated the change of the position in the image of the subject part in the subject image in the last and this photographed image, and the feature point corresponding to each other in both subject parts. It is a flowchart which shows operation | movement of PC in the same embodiment. It is a block diagram which shows the structure of the pointing device in 1st Embodiment.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
(Embodiment 1)
FIG. 1 is a configuration diagram of a projection system including a pointing device 1 of the present invention and a PC (personal computer) 2. In this projection system, the projector 3 projects a display image based on the image data output from the PC 2 onto the screen S so as to be enlarged. The PC 2 is a notebook computer, for example.

  The pointing device 1 has a function (first function) as a laser pointer and a function (second function) as an input device of the PC 2, and a display image projected on the screen S. That is, the following configuration is provided to enable an instruction of a notable part of the projection image (display of a designated position) and operation of the PC 2.

  That is, the pointing device 1 is a camera type having a function of photographing the screen S, and includes a photographing lens 11 and a CCD that is a photographing means for digitizing and capturing an optical image formed on the photographing lens 11. The image sensor 12 includes an image processing unit 13 that performs various image processing on captured image data (captured image data), an arithmetic control unit 14, a communication unit 15, a memory 16, and an operation unit 17.

  The calculation control unit 14 is configured by a microcomputer (not shown) and a microcomputer including a ROM that stores programs for causing the CPU to execute the various calculations. The arithmetic control unit 14 operates according to the program stored in the ROM, thereby functioning as an indicated position acquisition unit, a movement information acquisition unit (including a feature point extraction unit), a determination unit, and an information control unit of the present invention. While controlling each part of the pointing device 1, various calculations based on the image data after the image processing by the image processing unit 13 are performed.

  The memory 16 is a working memory for the arithmetic control unit 14, and stores various data as necessary. The communication unit 15 is a transmission unit according to the present invention. For example, the infrared communication method with a relatively narrow communication range, a Bluetooth (registered trademark) method, a wireless communication method such as a wireless LAN, or a USB (Universal Serial Bus). Coordinate position data (indicated position information) indicating a coordinate position, which will be described later, calculated by the calculation control unit 14 by an arbitrary data transmission method such as a wired communication method such as a wire or a relative movement from the previous position in the apparatus main body. Operation information indicating the operation content of the pointing device 1 such as movement amount data (relative movement information) indicating the relative movement amount in the X and Y directions indicating the amount and operation button identification data is transmitted (output) to the PC 2.

  The operation unit 17 includes a power switch, one or a plurality of operation buttons that are operated as needed during use, and a circuit that sends operation information to the arithmetic control unit 14.

  On the other hand, the PC 2 has a general configuration including a control unit 21 including a CPU and its peripheral circuits, a RAM 22, a storage device 23, an operation unit 24, a video adapter 25, a VRAM 26, a display device 27, a communication unit 28, and the like.

  The video adapter 25 generates a display video signal (RGB signal) and outputs it to a display device 27 including an LCD and its driving circuit. The VRAM 26 stores the display image data generated by the video adapter 25 as needed. To do. The video adapter 25 is connected to the projector 3 via an RGB cable or the like (not shown), and outputs display image data to the projector 3 via the RGB cable in accordance with an instruction from the control unit 21.

  The communication unit 28 is a receiving unit according to the present invention, includes a communication interface circuit based on a data transmission method to which the communication unit 15 of the pointing device 1 corresponds, and receives the above-described various operation information sent from the pointing device 1. The data is sent to the control unit 21.

  The operation unit 24 is a plurality of keys provided on the PC 2 main body, a mouse connected to the main body, the RAM 22 is a general-purpose work memory used for the control unit 21, and the storage device 23 has a storage capacity. It is a relatively large hard disk.

  The storage device 23 stores various application programs. In particular, in the present embodiment, the communication program for enabling the communication unit 28 to receive the coordinate position from the pointing device 1 and the control unit 21 include the confirmation unit, the operation mode setting unit, and the position mark display control of the present invention. A program for causing the control unit 21 to perform processing described later based on the above-described operation information sent from the pointing device 1 is stored.

  Hereinafter, operations of the pointing device 1 and the PC 2 when the pointing device 1 is used in a state where an arbitrary display image is projected on the screen S by the projection system will be described.

  First, assuming a general situation in which the explainer using the pointing device 1 is explaining at a position offset to the left or right side of the screen S and the PC 2 is placed in front of the explainer, The operation of the pointing device 1 will be described. FIG. 2 is a flowchart showing this.

  The pointing device 1 starts to operate when the power is turned on, and sequentially checks whether any operation button is pressed. If any operation button is pressed (YES in step SA1), the pointing device 1 is pressed. Operation button identification data for identifying the operation button is transmitted to the PC 2 (step SA2).

  During the operation, the operation button operation confirmation is performed and shooting is performed at a predetermined frame period (for example, 30 fps), and a captured image is acquired at regular intervals (step SA3). After each shooting, a projected image area corresponding to the projected image on the screen S is recognized from the acquired shot image (step SA4). The recognition of the projected image area is performed by, for example, classifying captured images by labeling, or extracting a linear component to determine the largest square, and determining the square area as the projected image area.

  When the projected image area can be recognized (YES in step SA5), it is determined that it is used as a laser pointer, that is, the function of the laser pointer is intended by the instructor. Process. Here, the presenter points the pointing device 1 (on the photographing lens 11 side) toward the screen S, the captured image 100 shown in FIG. 3A is acquired in step SA3, and the four corners A, are acquired in step SA4. In the following description, it is assumed that an obliquely deformed rectangular area having B, C, and D is recognized as the projected image area 101.

  When the projected image area 101 as described above can be recognized, the pointing device 1 first calculates a degree of distortion indicating how much the shape of the projected image area 101 is distorted compared to a rectangle (step SA6). The degree of distortion calculated here is the ratio of the length of the short side (line segment AB) to the long side (line segment DC) of the opposite left and right sides (line segment AB and line segment DC) of the projection image region 101. Yes, when the presenter (pointing device 1) is located in front of the screen S and the projected image area 101 is a regular square, the degree of distortion is minimized. The value calculated as the degree of distortion is not particularly limited.

  Then, it is determined whether or not the calculated degree of distortion is equal to or higher than a predetermined reference, and the degree of distortion is equal to or higher than the reference, and the probability that the image in the projected image area 101 is a projected image on the screen S is high. (Step SA7 YES), a conversion parameter for transforming (correcting) the projection image area 101 into a normal form, that is, a coefficient of the conversion matrix is acquired (step SA8). The conversion parameter acquired here is a parameter used for arbitrary coordinate conversion for performing distortion correction known as, for example, oblique conversion, and is a coefficient of a conversion matrix. In addition, when performing multiple types of coordinate transformation, the coefficient used for each coordinate transformation is calculated | required.

  Subsequently, the coordinate position (absolute position information) P of the display image 200 corresponding to the center O (reference point) of the captured image 100 is calculated using the conversion parameter (step SA9). Specifically, the coordinate position when the coordinate position of the center O of the captured image 100 is converted into the coordinate position of the projection image area 101 after normalization and the converted coordinate position is mapped to the coordinate space of the display image 200. The position P is obtained. FIG. 3B shows an example in which the pixel size of the display image 200 is 1024 × 768 (XGA) and the coordinate position P corresponding to the center O of the captured image 100 in the coordinate space is (742, 379). It is. Thereafter, the coordinate position data is transmitted to the PC 2 as an instruction position by the presenter (step SA10).

  Thereafter, the process returns to step SA1 and the above-described processing is repeated, so that while the projected image on the screen S can be photographed, new instruction positions (coordinate position data) by the presenter are successively given to the PC2 at regular intervals. Whenever any one of the operation buttons is pressed, the operation button identification data is transmitted to the transmission.

  On the other hand, for example, when the pointing device 1 (on the photographing lens 11 side) is directed to a part other than the screen S by the presenter during that time, the pointing device 1 is not recognized or immediately after the operation is started. If the projected image area is not recognized in the first captured image that is directed to a part other than the screen S (NO in step SA5), it is used as an input device of the PC 2, that is, by the explainer. It is determined that the use of the function as the input device of the PC 2 is intended, and the following processing is performed.

  Here, it is assumed that the pointing device 1 is pointed toward the presenter's feet (down) and the captured image 300 shown in FIG. 4A has been acquired in step SA3 described above. When the projected image area cannot be recognized, the pointing device 1 first determines whether or not the immediately preceding image is stored in the memory 16 at that time, and if not stored (NO in step SA11), The captured image is stored in the memory 16 as the previous image (step SA15), and the process returns to step SA1.

  On the other hand, when the immediately preceding image is stored in the memory 16 (YES in step SA11), a feature point is extracted from the previous captured image, and a corresponding feature point is extracted from the current captured image using a method such as pattern matching. It detects using (step SA12). FIG. 4B shows a subject portion in the previous and current captured images (in this example, the shoes of the explainer) when the pointing device 1 moves to the left between the previous imaging timing and the current imaging timing. Change of the position in the image (the left side is the previous position, the right side is the current position), the feature points E and F extracted from the previous captured image, and the corresponding feature of the current captured image It is the figure which illustrated points E 'and F'.

  Thereafter, the pointing device 1 moves relative to each other in the X and Y directions in the device body based on the difference in the positions of the corresponding feature points E and E ′ and the change in the position of the feature points F and F ′. After calculating the amount, that is, the relative movement amount from the immediately preceding position in the apparatus main body in accordance with the operation of the presenter (step SA13), and transmitting the movement amount data to the PC 2 (step SA14), The image is stored in the memory 16 (step SA15). That is, the previous image in the memory 16 is updated.

  After that, by returning to step SA1 and repeating the above processing, while the pointing device 1 is directed to a portion other than the screen S, the relative movement amount (from the immediately preceding position in the device body) at a certain time ( (Movement amount data) is sequentially transmitted to the PC 2 and operation button identification data is transmitted to the transmission every time one of the operation buttons is pressed.

  Furthermore, even if the determination result in step SA7 described above is NO and the projection image area can be recognized from each captured image acquired at regular intervals, the degree of distortion is not equal to or higher than the reference, and the recognized projection image area When it can be determined that there is a high probability that the image is not a projected image on the screen S, for example, even when the pointing device 1 is directed to the monitor screen of the PC 2 placed in front of the explainer, the explainer It is determined that the use of the function as an input device of the PC 2 is intended, and the movement amount data is sequentially transmitted to the PC 2 by repeating the above-described steps SA11 to SA15.

  Next, the operation of the PC 2 corresponding to the operation of the pointing device 1 described above will be described. FIG. 5 is a flowchart showing the operation content of the PC 2 when the image data of the display image is being output to the projector 3 in accordance with the presentation application program corresponding to the pointing device 1.

  While outputting image data to the projector 3, the PC 2 sequentially confirms whether or not data has been received from the pointing device 1. If any data is received (YES in step SB 1), first, the received data The type is confirmed (step SB2).

  If the received data is the coordinate position data described above (YES in step S3), and if the current processing mode is not a pointing mode prepared in advance (NO in step SB4), the processing mode is changed at that time. After changing to the pointing mode (step SB5), a new display image is generated by combining a pointer of a predetermined shape with the coordinate position indicated by the received coordinate position data, separately from the display image being output to the projector 3. It is output to the projector 3 (step SB6). If the pointing mode is selected (YES in step SB4), a new display image is generated immediately by synthesizing a pointer of a predetermined shape at the coordinate position indicated by the received coordinate position data, and is output to the projector 3. (Step SB6).

  Thereafter, while receiving the coordinate position data, the same processing as described above is repeated, so that a pointer is displayed at the position indicated by the presenter on the projected image on the screen S, which is the direction of the pointing device 1 Is moved in accordance with the change of the angle of view (movement of the angle of view). Thereby, the presenter can use the pointing device 1 as a laser pointer.

  Further, when the received data is not the coordinate position data described above but movement amount data (NO in step SB3, YES in step SB7), it is confirmed whether or not the current processing mode is a mouse mode prepared in advance. (Step SB8). If the mouse mode is not set (NO in step SB8), the processing mode is temporarily changed to the mouse mode at that time (step SB9), and the mouse pointer displayed on the monitor screen is indicated by the received movement amount data. Is moved by the amount of movement in the X and Y directions (step SB10).

  Thereafter, while the movement amount data is being received, the same processing as described above is repeated, so that the mouse pointer displayed on the monitor screen changes according to the change in the direction of the pointing device 1 (movement of the angle of view). Moved. Thus, the presenter can use the pointing device 1 as an input device similar to a general mouse.

  Further, when the received data is not the coordinate position data or the movement amount data described above but the operation button identification data (NO in both steps SB3 and SB7), it depends on the processing mode at that time, and the operated button operation is different. Correspondingly, a predetermined process is executed (step SB11).

  That is, when the processing mode is the pointing mode, for example, processing for changing the color of the pointer combined with the display image (displayed in the projection image) or processing for changing the size of the pointer is performed. When the processing mode is the mouse mode, for example, processing corresponding to the position of the mouse pointer or display image change (page feed) processing is performed.

  As described above, the pointing device 1 according to the present embodiment is operation information indicating the operation content of the device to be transmitted to the PC 2 as described above, and changes according to a change in the direction of the pointing device 1 (movement of the angle of view). By switching the information to be coordinate position data or movement amount data, it can be used not only as a laser pointer but also as an input device of the PC 2. Moreover, in use, when the presenter intends to use the function as a laser pointer, it is directed to the screen S, and when intended to use the function as an input device of the PC 2, As a result, the pointing device 1 automatically determines the function that the presenter intends to use as a result, and the operation information transmitted to the PC 2 according to the determination result is coordinate position data or movement amount data. Switched to.

  Therefore, in the presenter, the function as the laser pointer (first function) and the function as the input device of the PC 2 are performed without instructing the change of the purpose of use, that is, the operation for switching the function to be used. (Second function) can be used as needed and seamlessly. That is, a desired function can be used immediately without performing a function switching operation according to the purpose of use.

  Further, in the pointing device 1 of the present embodiment, even if the projected image area can be recognized from the captured image, the degree of distortion is not higher than the reference, and the image of the recognized projected image area is not a projected image on the screen S. If it can be determined that the function is high, it is determined that the function that the instructor intends to use is a function as an input device of the PC 2, and the movement amount data is transmitted to the PC 2. Therefore, as described above, even when the pointing device 1 is directed to the monitor screen of the PC 2 placed in front of the presenter's eyes, the function that the presenter intends to use (the function being used) ) Is not erroneously determined to be a function as a laser pointer. At the same time, the pointing device 1 can thereby be used as a mouse.

  When the projected image area can be recognized from the captured image, it is unconditionally determined that the function intended for use by the presenter is a function as a laser pointer, and the coordinate position data is transmitted to the PC 2. Even in this case, it is possible to immediately use a desired function without performing a function switching operation according to the purpose of use.

  In the present embodiment, a case has been described in which the PC 2 simply displays a pointer on the projection image and performs a process of moving the pointer in accordance with the movement operation of the angle of view in the pointing device 1 in the pointing mode. As described above, a process of drawing a locus line representing the movement locus of the pointer on the projection image may be performed. In this case, when the data received in the pointing mode is the operation button identification data In response to the reception of the data, processing such as drawing start or drawing of the locus line, erasing the locus line, or the like may be performed.

(Embodiment 2)
Next, a second embodiment of the present invention will be described. As shown in FIG. 6, the present embodiment relates to a pointing device 51 having a configuration in which an angle detection unit 18 is added to the configuration described in FIG. The angle detection unit 18 detects a vertical tilt angle of the optical axis (axis line along the shooting direction) of the photographing lens 11 in the pointing device 51 and sends the detection result to the arithmetic control unit 14. It comprises a sensor, an amplification circuit for the detection signal, a signal processing circuit after amplification, and the like. In the present embodiment, the inclination angle sent from the angle detection unit 18 to the calculation control unit 14 is information indicating an inclination angle in a predetermined direction with reference to a state where the optical axis of the photographing lens 11 is horizontal.

  In the pointing device 51 according to the present embodiment, information that changes according to a change in the orientation of the pointing device 51 (movement of the angle of view) is generated by using an operation different from that of the first embodiment, although not shown. , Switch to coordinate position data or movement amount data.

  Specifically, although not shown, after the photographed image is acquired in step SA4 during the operation described in the flowchart of FIG. 2, the optical axis of the photographing lens 11 at that time is replaced with the processing in steps SA5 to SA7 described above. Is detected by the angle detector 18 and a process is performed to confirm whether or not the detected inclination angle is within a predetermined angle range. Here, the predetermined angle range is expected when the explainer points to an arbitrary position of the projected image under a general situation where the explainer is explaining at a position offset to the left or right side of the screen S. Angle range.

  If the detected tilt angle is within a predetermined angle range, it is determined that the laser pointer is being used at that time (the function of the laser pointer is intended by the explainer), By the processes in steps SA8 to SA10 described above, the coordinate position of the designated position is calculated based on the captured image, and the coordinate position data is transmitted to the PC2. If the detected tilt angle is not within the predetermined angle range, it is determined that the PC 2 is used as an input device at that time (the function of the PC 2 as an input device is intended by the presenter). Then, the movement amount in the X and Y directions of the apparatus main body is calculated based on the previous and current photographed images by the processing in steps SA11 to SA15 described above, and the movement amount data is transmitted to the PC2.

  Also in the second embodiment described above, as in the first embodiment, the instructor instructs the change of the purpose of use, that is, without performing an operation for switching the function to be used. And the function (second function) as the input device of the PC 2 can be used seamlessly as necessary. In addition, as compared with the first embodiment, it is possible to quickly determine the function intended to be used by the presenter from time to time.

  In the present embodiment, the predetermined angle range that is a criterion for determining the function that the explainer intends to use is the angle range that is expected when the explainer points to an arbitrary position of the projection image. However, the predetermined angle range may be, for example, an angle range in a case where the presenter points the pointing device 51 downward. In this case, contrary to the present embodiment, if the detected tilt angle is within a predetermined angle range, it is determined that the PC 2 is used as an input device, and if not within the predetermined angle range, What is necessary is just to judge that it is used as a pointer.

(Other embodiments)
Here, in the first and second embodiments described above, when the presenter determines that the function intended to be used is a function as an input device of the PC 2, the previous shooting was performed in sequence. The main body operation information of the present invention indicating the movement amount of the apparatus main body in the X and Y directions, that is, the operation state of the apparatus main body, is acquired based on the change in the corresponding feature point position between the image and the current photographed image. However, the movement amount of the apparatus main body in the X and Y directions may be obtained by other methods.

  For example, instead of the angle detection unit 18 of the pointing device 51 shown in FIG. 6, an acceleration sensor or a gyro sensor provided in the apparatus main body, or both of the acceleration sensor and the gyro sensor, an amplification circuit for the detection signal, and after amplification When a function that the instructor intends to use is a function as an input device of the PC 2 is provided. The movement amount in the X and Y directions of the apparatus main body may be acquired based on the detection result by the motion detection unit. According to such a configuration, it is possible to quickly acquire the amount of movement of the apparatus main body in the X and Y directions and transmit it to the PC 2 as compared to the first and second embodiments.

  Furthermore, if the configuration includes both the angle detection unit 18 and the motion detection unit described above, the function intended to be used by the presenter can be quickly determined, and the function is a function as an input device of the PC 2. For a while, the movement amount of the apparatus main body in the X and Y directions can be quickly acquired and transmitted to the PC 2. In that case, unlike the first and second embodiments, the function that is intended to be used by the presenter at that time is determined by determining whether or not the operation button is operated (step SA1 in FIG. 2). Similarly to the above, if the cycle is shorter than the captured image acquisition interval (imaging interval), the pointing device can have the same responsiveness as an input device such as a mouse.

  Here, in the above description, only the case where the external information processing apparatus of the present invention is the PC 2 has been described, but the coordinate position data, movement amount data, operation button identification data transmitted from the pointing device 1 by the projector 3 is described. As long as the operation information can be received, the projector 3 may be configured to perform the above-described processing on the PC 2 side, that is, the pointer display processing, as the external information processing apparatus of the present invention. Furthermore, it may be realized in a projection system that does not include the PC 2.

  Furthermore, the pointing device of the present invention is not limited to the projected image displayed on the screen by the projector 2 and may be any device that indicates an arbitrary position of the display image. For example, it can be used to indicate an arbitrary position on a display screen of a plasma display (PDP) or a liquid crystal display (LCD).

1,51 Pointing device 2 PC
DESCRIPTION OF SYMBOLS 3 Projector 11 Shooting lens 12 Image sensor 13 Image processing part 14 Computation control part 15 Communication part 16 Memory 17 Operation part 18 Angle detection part 21 Control part 23 Memory | storage device 27 Display apparatus 28 Communication part 100 Captured image 101 Projection image area 200 Display image 300 Photographed image O Center P Coordinate position S Screen

Claims (15)

In a pointing device having a function of instructing a position in a display screen and transmitting instruction position information indicating the instructed position in the display screen to an external information processing device,
Photographing means used for photographing the display screen;
Transmitting means for transmitting operation information indicating operation contents of the pointing device to the external information processing device;
An instruction position acquisition means for acquiring the instruction position information; a movement information acquisition means for acquiring movement information indicating a movement amount of the pointing device body;
Determining means for determining an operation status of the pointing device main body by a user ;
With
When the determination unit determines that the pointing device is not held toward the display screen, the transmission unit transmits information including the movement information as the operation information to the external information processing device ,
The determining means determines whether or not an operation state of the pointing device main body is determined by determining whether or not a display screen area corresponding to the display screen exists in the captured image captured by the capturing means;
When the determination unit determines that the display screen area does not exist in the captured image captured by the imaging unit, the transmission unit transmits information including the movement information as the operation information to the external information processing apparatus. Send
When the determination unit determines that the display screen area exists in the captured image captured by the imaging unit, the transmission unit transmits information including the designated position information as the operation information to the external information processing apparatus. A pointing device characterized by transmitting . The determination unit determines whether or not the captured image captured by the imaging unit includes a display screen region corresponding to the display screen and having a distortion degree equal to or greater than a reference, thereby determining an operation state of the pointing device main body. And when the determination unit determines that there is no display screen area having the distortion degree equal to or greater than a reference, the transmission unit sets the information including the movement information as the operation information. The pointing device according to claim 1, wherein the pointing device transmits to a processing device. If the determination means determines that there is a display screen area whose degree of distortion is greater than or equal to a reference, the transmission means transmits information including the indicated position information as the operation information to the external information processing apparatus. The pointing device according to claim 2, wherein: An angle detecting means for detecting an inclination angle in a predetermined direction of an axis along the photographing direction in the photographing means;
The pointing device according to claim 1, wherein the determination unit determines an operation state of the pointing device main body by a user based on the tilt angle detected by the angle detection unit. In a pointing device having a function of instructing a position in a display screen and transmitting instruction position information indicating the instructed position in the display screen to an external information processing device,
Photographing means used for photographing the display screen;
Transmitting means for transmitting operation information indicating operation contents of the pointing device to the external information processing device;
Indicated position acquisition means for acquiring the indicated position information;
Movement information acquisition means for acquiring movement information indicating the movement amount of the pointing device body;
Determining means for determining an operation status of the pointing device main body by a user;
With
When the determination unit determines that the pointing device is not held toward the display screen, the transmission unit transmits information including the movement information as the operation information to the external information processing device,
The determination unit determines whether or not the captured image captured by the imaging unit includes a display screen region corresponding to the display screen and having a distortion degree equal to or greater than a reference, thereby determining an operation state of the pointing device main body. And when the determination unit determines that there is no display screen area having the distortion degree equal to or greater than a reference, the transmission unit sets the information including the movement information as the operation information. Sent to the processing unit,
If the determination means determines that there is a display screen area whose degree of distortion is greater than or equal to a reference, the transmission means transmits information including the indicated position information as the operation information to the external information processing apparatus. A pointing device characterized by that. The determining means determines whether or not an operation state of the pointing device main body is determined by determining whether or not a display screen area corresponding to the display screen exists in the captured image captured by the capturing means;
When the determination unit determines that the display screen area does not exist in the captured image captured by the imaging unit, the transmission unit transmits information including the movement information as the operation information to the external information processing apparatus. 6. The pointing device according to claim 5, wherein the pointing device transmits. When the determination unit determines that the display screen area exists in the captured image captured by the imaging unit, the transmission unit transmits information including the designated position information as the operation information to the external information processing apparatus. The pointing device according to claim 6, wherein the pointing device transmits. An angle detecting means for detecting an inclination angle in a predetermined direction of an axis along the photographing direction in the photographing means;
The pointing device according to claim 5, wherein the determining unit determines an operating state of the pointing device main body by a user based on the tilt angle detected by the angle detecting unit. The movement information acquisition unit includes a feature point extraction unit that extracts a feature point from a photographed image that has been captured by the photographing unit, and one of the feature points extracted by the feature point extraction unit. captured image and the other pointing device according to any one of claims 1 to 8, characterized in that acquires the movement information on the basis of the change in position between the captured images. A movement detecting means for directly detecting the movement of the pointing device main body;
The movement information acquisition means, a pointing device according to any one of claims 1 to 8, characterized in that acquires the movement information on the basis of the motion of the pointing device body that is detected by the motion detecting means. The indicated position acquisition means is a display screen corresponding to a coordinate position of a predetermined reference point in a photographed image photographed by the photographing means when a display screen area corresponding to the display screen in the photographed image is transformed into a normal form. pointing device of converting the coordinate position, wherein the coordinate position after the conversion in any one of claims 1 to 10, characterized in that to obtain, as the indicated position information. In a pointing device having a function of instructing a position in a display screen and transmitting instruction position information indicating the instructed position in the display screen to an external information processing device,
A shooting process for shooting the display screen;
A movement information acquisition step of acquiring movement information indicating a movement amount of the pointing device body;
A determination step of determining an operation status of the pointing device main body by a user;
If it is determined in the determining step that the pointing device is not held toward the display screen, information including the movement information is transmitted to the external information processing device as operation information indicating the operation content of the pointing device. and a transmission step of,
Only including,
The determining step determines an operation state of the pointing device main body by determining whether or not a display screen area corresponding to the display screen exists in the captured image captured by the capturing step.
When the determining step determines that the display screen area does not exist in the captured image captured by the capturing step, the transmitting step transmits information including the movement information as the operation information to the external information processing apparatus. Send
When the determination step determines that the display screen area exists in the captured image captured by the imaging step, the transmission step transmits information including the designated position information as the operation information to the external information processing apparatus. information transmission method and transmitting. In a pointing device having a function of instructing a position in a display screen and transmitting instruction position information indicating the instructed position in the display screen to an external information processing device,
A shooting process for shooting the display screen;
A movement information acquisition step of acquiring movement information indicating a movement amount of the pointing device body;
A determination step of determining an operation status of the pointing device main body by a user;
If it is determined in the determining step that the pointing device is not held toward the display screen, information including the movement information is transmitted to the external information processing device as operation information indicating the operation content of the pointing device. A transmission process to
Including
The determination step determines whether or not the captured image captured by the imaging step includes a display screen region corresponding to the display screen and having a distortion degree equal to or higher than a reference, thereby operating the pointing device main body. And when the determining step determines that there is no display screen area having the distortion degree equal to or higher than a reference, the transmitting step includes information including the movement information as the operation information. Sent to the processing unit,
When the determination step determines that there is a display screen area having the distortion degree equal to or higher than a reference, the transmission step transmits information including the indicated position information as the operation information to the external information processing apparatus. An information transmission method characterized by: A computer having a pointing device having a function of instructing a position in a display screen and transmitting designated position information indicating the instructed position in the display screen to an external information processing device,
Shooting procedure to shoot the display screen,
A movement information acquisition procedure for acquiring movement information indicating the movement amount of the pointing device body;
A determination procedure for determining an operation status of the pointing device main body by a user;
When it is determined in the determination procedure that the pointing device is not held toward the display screen, information including the movement information is transmitted to the external information processing device as operation information indicating an operation content of the pointing device. and the transmission procedure to,
Was executed,
The determination procedure determines the operation state of the pointing device main body by determining whether or not the captured image captured by the imaging procedure includes a display screen area corresponding to the display screen,
When the determination procedure determines that the display screen area does not exist in the photographed image photographed by the photographing procedure, the transmission procedure sends information including the movement information as the operation information to the external information processing apparatus. Send
When the determination procedure determines that the display screen area is present in the captured image captured by the imaging procedure, the transmission procedure transmits information including the indicated position information as the operation information to the external information processing apparatus. Submit to information transmission program characterized Rukoto. A computer having a pointing device having a function of instructing a position in a display screen and transmitting designated position information indicating the instructed position in the display screen to an external information processing device,
Shooting procedure to shoot the display screen,
A movement information acquisition procedure for acquiring movement information indicating the movement amount of the pointing device body;
A determination procedure for determining an operation status of the pointing device main body by a user;
When it is determined in the determination procedure that the pointing device is not held toward the display screen, information including the movement information is transmitted to the external information processing device as operation information indicating an operation content of the pointing device. Sending procedure to
And execute
In the determination procedure, the operation state of the pointing device main body is determined by determining whether or not the captured image captured by the imaging procedure includes a display screen area corresponding to the display screen and having a distortion degree equal to or greater than a reference. Judging
When the determination procedure determines that there is no display screen area having the distortion degree equal to or higher than the reference, the transmission procedure transmits information including the movement information as the operation information to the external information processing apparatus. And
When the determination procedure determines that there is a display screen area having the distortion degree equal to or greater than a reference, the transmission procedure transmits information including the indicated position information as the operation information to the external information processing apparatus. An information transmission program characterized by:

Images