JP2011113216A - Pointing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a pointing system capable of solving display delay and the precision and operability of pointing. <P>SOLUTION: A pointing system 1 is provided with: a display part 230 for displaying the photographic video of an object space on a screen; an instruction part slave unit 250 for accepting the instruction of target coordinates on the screen on which the photographic video of the object space is displayed and an infrared LED 240; a space pointing part 140 for generating visual change at a corresponding position in the object space with respect to the target coordinates; and control parts 110 and 210 for displaying a video obtained by suppressing the visual change generated in the object space as the result of the instruction on the screen. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a pointing system, and more particularly, to a pointing system that indicates a target coordinate on a screen that displays a captured image of a target space and points to a corresponding position in the target space in response to the instruction.

  Video conferencing systems that connect multiple remote locations over a network with bidirectional video images and audio and support communication between physically distant members are becoming increasingly popular with the development of digital compression and display technologies. The quality and screen size are increasing.

  In particular, in a video conference system called a telepresence system, a sense of reality is produced by, for example, displaying remote conference participants in a life-size manner, and the person can feel as if they are in the same space.

  Many video conference systems are provided with a function for sharing a screen on which presentation materials are displayed and a function for displaying an independent mouse pointer for each site by distinguishing them with colors on the shared screen. Accordingly, the participants can efficiently discuss while pointing to and pointing to the data on the shared screen (pointing).

  However, only electronic data such as presentation materials can be referred to by the above function, and it is not possible to refer to a real object in a remote place. Therefore, it is not possible to prepare materials in advance such as brainstorming, and in a meeting where information is written on a whiteboard or sticky note on the spot, or in a meeting that surrounds a physical model such as a mockup, In order to refer to an object, there is a problem that a complicated procedure such as explaining to a person at a remote place with words or the like and pointing to a substitute is usually required.

  A technique for solving such problems is described in Patent Document 1. In the video conference apparatus of Patent Document 1, a video of a remote place is displayed on a screen, and a user can point a part of an image using a touch panel or the like. Then, the beam irradiation direction of the light beam device installed in a remote location (for example, integrated with a camera for taking images) is controlled using a motor or the like based on the coordinates pointed to by the touch panel. The As a result, the light beam is irradiated to the place designated by the touch panel, and the person at the remote place can recognize the pointed place.

  In Patent Document 2, although a similar technique is disclosed, a method of projecting a pointer using a video projector instead of irradiating a light beam is also disclosed.

  However, in the above technique, a delay from when the user points to a part of the image using a touch panel or the like until the information is transmitted to a remote place and a pointing device such as a light beam device points to a corresponding place, There is no consideration about the delay until the image irradiated with a pointer such as a light beam is transmitted in the opposite direction from the remote location and fed back to the user on the screen display.

  In the case of designating a single point at a remote place, no major problem occurs even if the above-described conventional technique is used. However, as many people know from the use of indicators such as a stick, people use the indicator to surround a region (draw a circle), in addition to specifying a single point, Often, more complicated gestures, such as drawing directions or shapes or letters, are performed.

  When performing a pointing operation that draws a trajectory over a certain amount of time in this way, if the visual feedback on the user's operation is delayed, the total time until the intended operation is completed increases, and further, the indicated position and shape It is known that the accuracy of the is reduced.

  For example, as shown in Non-Patent Document 1, in the operation of pointing the target area using the mouse and the cursor, when the user moves the mouse and there is a delay in the change of the cursor display corresponding thereto, the target area Experimental results have been disclosed in which the time to move the cursor in and the accuracy increases with delay.

  For example, as shown in the table of FIG. 1 of Non-Patent Document 1, when the delay is 225 milliseconds (ms), the cursor is placed in the target area compared to the case where the delay is negligibly small (8.5 ms). It has been reported that the average time to move is increased by 60% and the error rate of operation is increased by 214% on average.

  Incidentally, a delay of about 225 ms is not uncommon in a remote video conference system. In general, delay associated with information transmission to a remote location is roughly divided into input sampling delay, propagation delay, transmission delay, output sampling delay, and software overhead.

  The input sampling delay is a delay from when the user touches the touch panel until the system reads the coordinates. For example, when sampling is performed at 100 Hz, a delay of 10 ms at maximum and 5 ms on average occurs.

  The propagation delay is a delay required for the coordinate information to propagate through the transmission medium, and is a value obtained by dividing the distance by the propagation speed. Transmission delay is the time from the arrival of the beginning of a network packet including coordinate information to the completion of reception of the entire packet. The general store-and-forward process is performed after the entire packet is read into memory. This is a delay that occurs in the type of processing device. This is a value obtained by dividing the packet length by the transmission bit rate.

  The output sampling delay is a delay until the coordinate information stored in the output buffer is read by the output device (for example, the light beam device). For example, when sampling is performed at 100 Hz, a delay of 10 ms at maximum and an average of 5 ms occurs. The software overhead is various processing delays that occur when coordinate information is coded and decoded in a network packet.

  In the case where the image of the light beam is taken and visual feedback is performed, the image taken at a remote place is displayed again through the network, so that the above delay occurs twice.

JP-A-5-204534 JP 2003-209832 A Japanese Patent No. 3422383

By I. Scott MacKenzie, Colin Wear, "Lag as a Determinant of Human Performance in Human Systems" Interactive Systems), Conference on Human Factors in Computing Systems, 1993, p. 488-493

  For example, even if only the propagation delay is taken into account, an optical signal (speed = 3 × 10 8 sq.m) for transmitting information from the back side of the earth (distance = approximately 20,000 kilometers) is about 66 ms one way. Since a delay (2E7 ÷ 3E8 seconds) and a delay of about 130 ms occur due to folding, a total delay of several hundred ms often occurs when other delays are included. When using the technique described in the above-mentioned document in a situation involving such a delay, there is a problem that the time required for pointing and the pointing accuracy deteriorate, and communication between remote locations is hindered. .

  The reason why such a problem occurs is that, in the technique described in the above document, a pointer image photographed at a remote place is used as a visual feedback to the user for a pointing operation. This situation is schematically shown in FIG. As shown in FIG. 5A, in the technique described in the above document, an image 23 such as a laser pointer is displayed at a position pointed to in the past on the trajectory instead of the position 21 actually pointed on the screen 20 by the user. As part of the screen 20. For this reason, with the increase in the delay, the difference between the actually indicated position 21 and the visual feedback is remarkably increased, which takes time.

  In some systems, there is a system in which a local pointer such as the mouse pointer 31 is displayed on the screen 20 separately from the image of the pointer photographed at a remote place and provided to the operator. As shown in FIG. 6B, the visual feedback is doubled with the image 23 such as the laser pointer in addition to the mouse pointer 31, and as a result of the user's confusion, the time required for pointing and the deterioration of the pointing accuracy are also deteriorated. Resulting in.

  An object of the present invention is to provide a pointing system that solves the above-described problems of display delay and pointing accuracy and operability.

The pointing system of the present invention includes display means for displaying a captured image of a target space on a screen,
Instruction accepting means for accepting an instruction of target coordinates on the screen displaying the captured video of the target space;
Spatial pointing means for generating a visual change at a corresponding position in the target space with respect to the target coordinates;
Control means for displaying an image on which the visual change occurring in the target space as a result of the instruction is suppressed is displayed on the screen;
Is provided.

The first processing apparatus of the present invention includes a spatial pointing means for generating a visual change at a corresponding position in the target space with respect to target coordinates indicated on a screen displaying a captured image of the target space.
Control means for displaying on the screen an image in which the visual change occurring in the target space as a result of the instruction is suppressed.

The second processing apparatus of the present invention comprises a display means for displaying a captured video of the target space on a screen,
Instruction accepting means for accepting an instruction of target coordinates on the screen displaying the captured video of the target space;
As a result of the instruction, there is provided control means for displaying on the screen an image in which a visual change generated at a corresponding position in the target space is suppressed with respect to the target coordinates.

The data processing method of the processing device of the present invention displays a captured video of the target space on the screen,
Receiving an instruction of target coordinates on the screen displaying the captured video of the target space;
Generating a visual change at a corresponding position in the target space with respect to the target coordinates;
An image in which the visual change that occurs in the target space as a result of the instruction is suppressed is displayed on the screen.

The computer program of the present invention is a computer program for realizing a processing device,
A display procedure for displaying the captured image of the target space on the screen;
An instruction receiving procedure for receiving an instruction of a target coordinate on the screen displaying the captured video of the target space;
A generation procedure for generating a visual change at a corresponding position in the target space with respect to the target coordinates;
A control procedure for causing the computer to execute a control procedure for displaying on the screen an image in which the visual change occurring in the target space as a result of the instruction is suppressed.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.

  The various components of the present invention do not necessarily have to be independent of each other. A plurality of components are formed as a single member, and a single component is formed of a plurality of members. It may be that a certain component is a part of another component, a part of a certain component overlaps with a part of another component, or the like.

  In addition, although a plurality of procedures are described in order in the data processing method and the computer program of the present invention, the described order does not limit the order in which the plurality of procedures are executed. For this reason, when implementing the data processing method and computer program of this invention, the order of the several procedure can be changed in the range which does not interfere in content.

  Furthermore, the plurality of procedures of the data processing method and the computer program of the present invention are not limited to being executed at different timings. For this reason, another procedure may occur during the execution of a certain procedure, or some or all of the execution timing of a certain procedure and the execution timing of another procedure may overlap.

  According to the present invention, a pointing system that solves display delay and pointing accuracy and operability is provided.

It is a functional block diagram which shows the structure of the pointing system which concerns on embodiment of this invention. It is a block diagram which shows the detailed structure of the space pointing part of the pointing system which concerns on embodiment of this invention. FIG. 3 is a block diagram illustrating a detailed configuration of a galvano scanner of the space pointing unit in FIG. 2. It is a block diagram which shows the detailed structure of the instruction | indication part subunit | mobile_unit of the pointing system which concerns on embodiment of this invention. It is a flowchart which shows an example of operation | movement of the imaging | photography part of this embodiment. It is a figure which shows the control signal which the control part of the 1st processing apparatus of this embodiment transmits. It is a flowchart which shows an example of operation | movement of the pointing system of this embodiment. It is a flowchart which shows an example of operation | movement of the pointing system of this embodiment. It is a state transition diagram showing operation | movement of the control circuit of the space pointing part of this embodiment. It is a timing chart which shows the timing of operation of each component of the 1st processing unit of this embodiment. It is a figure which shows an example of the image | video image | photographed by the visual change and imaging | photography part which the space pointing part of this embodiment produces. It is a figure for demonstrating an example of operation | movement of each unit of the pointing system of this Embodiment. It is a figure which shows an example of the image | video image | photographed with the imaging | photography part of this embodiment, and the screen displayed on a display part. It is a figure which shows an example of the screen displayed on a display part, when a circular locus | trajectory is drawn with the instruction | indication part of this embodiment. It is a figure which shows an example of the screen displayed on a display part when the instruction | indication part of this embodiment is moved at a constant speed horizontally. It is a state transition diagram showing operation | movement of the control circuit of the space pointing part of this embodiment. It is a timing chart which shows the timing of operation of each component of the 1st processing unit of this embodiment. It is a figure which shows an example of the 3rd image | video obtained by combining two types of image | videos image | photographed with the imaging | photography part of this Embodiment, and them. It is a figure which shows an example of the screen displayed on a display part, when a circular locus | trajectory is drawn with the instruction | indication part of this Embodiment. It is a figure showing the area | region where the color information of a pixel is replaced by the control part of this Embodiment. It is a figure for demonstrating the method of calculating automatically the area | region where the color information of a pixel is replaced by the control part of this Embodiment. It is a figure which shows the example of the screen display in the remote pointing system of the technique of literature description.

(First embodiment)
Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same reference numerals are given to the same components, and the description will be omitted as appropriate.

FIG. 1 is a functional block diagram showing a configuration of a pointing system 1 according to an embodiment of the present invention.
As shown in FIG. 1, the pointing system 1 includes a first processing device 100 and a second processing device 200. The first processing device 100 and the second processing device 200 include a LAN (Local Area Network), a WAN ( Wide Area Network) and the network 3 such as the Internet. The connection between each device and the network 3 may be wireless or wired.

  In the pointing system 1 of this embodiment, the 1st processing apparatus 100 and the 2nd processing apparatus 200 can each be installed in the place distant, for example. Although not particularly limited, the installation location can be a remote location in the same room, a different room, a different floor, a different building, a different area, or the like.

  The first processing apparatus 100 includes a control unit 110, a space pointing unit 140, and a photographing unit 142. Further, the second processing device 200 includes a control unit 210, a display unit 230, an instruction unit slave device 250, and an instruction unit parent device 280. The control unit 210 of the second processing apparatus 200 and the control unit 110 of the first processing apparatus 100 are connected via the network 3. The connection between the control unit 210 and the control unit 110 and the network 3 may be either wired or wireless.

  The second processing device 200 displays the video of the remote target space captured by the first processing device 100 on the screen and presents it to the user B, and also displays the video of the target space by the user B using the instruction unit slave unit 250. Accept instructions on the screen. The first processing device 100 captures an image of the target space and transmits it to the second processing device 200, and causes a visual change in the position indicated by the user B with respect to the target space viewed by the user A.

That is, the pointing system 1 of the present embodiment, for example, in an electronic conference or the like, in a situation where the user A is in front of the whiteboard 132 or the like, an image of the whiteboard 132 or the like in the target space is captured via the network 3. The video is displayed on the screen of the display unit 230 viewed by the user B.
Then, when the user B points to a desired position on the screen of the captured video displayed on the display unit 230, the information is transmitted to the user A side, and the space pointing unit 140 is visually displayed at the corresponding position in the target space. The whiteboard 132 is irradiated with a pointer 172 by the light beam 170, for example. Thereby, the position instructed by the user B can be transmitted to the user A.

  Each component of the pointing system 1 is centered on an arbitrary computer CPU, memory, a program for realizing the components shown in the figure loaded in the memory, a storage unit such as a hard disk for storing the program, and a network connection interface. Realized by any combination of hardware and software. It will be understood by those skilled in the art that there are various modifications to the implementation method and apparatus. Each figure described below shows functional unit blocks, not hardware unit configurations.

  The first processing device 100 and the second processing device 200 include, for example, a CPU (Central Processing Unit) (not shown), a memory, a hard disk, and a communication network interface, and input devices such as a keyboard and a mouse, and outputs such as a display and a printer. It can be realized by a server computer or a personal computer (PC) connected to the device, or a device corresponding to them. The CPU can realize each function of each unit by reading a program stored in the hard disk into the memory and executing it with a general-purpose operating system. In the following drawings, the configuration of parts not related to the essence of the present invention is omitted and is not shown.

  The pointing system 1 according to the present embodiment includes a display unit 230 that displays a captured image of the target space on the screen, and an instruction reception unit (instruction unit that receives an instruction of target coordinates on the screen that displays the captured image of the target space. Machine 250, pointing unit master 280, infrared LED (Light Emitting Diode) 240 a to d), space pointing unit 140 that generates a visual change at a corresponding position in the target space with respect to the target coordinates, And a control unit (110, 210) that displays on the screen of the display unit 230 an image that suppresses a visual change that occurs as a result in the target space.

  In addition, the pointing system of the present embodiment further includes an imaging unit 142 that images the target space, and the control units (110, 210) repeat a time interval in which a visual change occurs and a time interval in which a visual change does not occur. In addition, by causing the space pointing unit 140 to generate a visual change at a corresponding position in the target space, and causing the shooting unit 142 to repeatedly capture the target space at a time interval at which the spatial pointing unit 140 does not generate a visual change. A video that suppresses a visual change composed of a series of obtained images is generated, and the generated video that suppresses the visual change is displayed on the screen of the display unit 230.

Hereinafter, each component of the pointing system 1 of this embodiment is demonstrated in detail.
First, the 1st processing apparatus 100 which generates a visual change in the position instruct | indicated remotely from object space is demonstrated.
In the first processing apparatus 100, the imaging unit 142 captures an image of a space in which an object to be pointed is placed (here, an example of a whiteboard 132 on which characters are drawn as a pointing target is illustrated). . The time interval at which shooting is performed by the shooting unit 142 is controlled by a control signal input from the control unit 110. For the photographing unit 142, for example, a generally available IEEE 1394 camera can be used.

  The control unit 110 is connected to the imaging unit 142 and the space pointing unit 140, and reads out the captured video from the imaging unit 142 and issues a pointing instruction to the space pointing unit 140. In addition, the control unit 210 and the control unit 110 exchange video information, in-video coordinate information, and the like via the network 3.

  The space pointing unit 140 is a device that is usually installed integrally with the photographing unit 142, and irradiates the target space with the light beam 170, thereby causing a visual change at a specific position in the space. In FIG. 1, a pointer 172 is shown as a visual change area. By this visual change, a pointing instruction can be presented to the user A.

  A time interval at which the light beam 170 is irradiated by the space pointing unit 140 is controlled by the control unit 110. As the spatial pointing unit 140, a red laser pointer capable of controlling the on / off of the light beam 170 and the irradiation direction can be used.

  FIG. 2 is a block diagram illustrating a configuration example of the space pointing unit 140. Referring to the figure, the spatial pointing unit 140 includes a laser 148, two sets of galvano scanners 152a, b (hereinafter referred to as "galvano scanner 152" if no particular distinction is required), and mirrors 150a, b ( Hereinafter, when there is no need for distinction, it is referred to as “mirror 150”) and a control circuit 146. The laser 148 is used to generate a light beam 170 having directivity, and a commercially available class 2 laser pointer can be used.

  The galvano scanner 152 is a device that controls the direction of the mounted mirror 150 at high speed in accordance with the amount of current to flow, and has a structure as shown in FIG. When a current is passed through the coil 164, a magnetic field is generated via the magnetic body 166 and a magnetic force is generated with respect to the magnet 162. The magnet 162 rotates until this magnetic force balances with the stress of the spring 160. As the magnet 162 rotates, the angle of the mirror 150 attached thereto changes.

  Such a galvano scanner 152 is generally used in an apparatus that generates an image by deflecting the light beam 170 such as a laser projector, and an off-the-shelf product can be obtained from a specialized manufacturer. Returning to FIG. 2, the control circuit 146 controls the direction of the light beam 170 by adjusting the current flowing through the galvano scanner 152 based on the deflection angle information received from the control unit 110 of FIG. 1 via the connection cable. As the control circuit 146, a general microcomputer and a driver circuit can be used in combination.

Next, returning to FIG. 1, a description will be given of the second processing device 200 that displays an image of a remote target space on the screen and receives an instruction from the user B and transmits it to the target space.
In the second processing device 200, the control unit 210 is connected to the display unit 230 and the instruction unit master 280, and performs display control of the image on the display unit 230, reading of the coordinates in the image from the indicator, and the like.

  The display unit 230 is, for example, a liquid crystal display, an organic EL (ElectroLuminescence) display, a plasma display, or the like. On the screen of the display unit 230, a target space imaged by the imaging unit 142, here, a video 232 of a whiteboard is displayed. Four infrared LEDs 240 are arranged at the four corners of the screen of the display unit 230. In FIG. 1, the four infrared LEDs 240 are indicated by reference numerals 240a, 240b, 240c, and 240d, respectively. Hereinafter, particularly when it is not necessary to distinguish the four infrared LEDs 240, they are referred to as “infrared LEDs 240”.

  In addition, the second processing device 200 includes an instruction unit slave unit 250 and an instruction unit master unit as indicators used by the user B to point to a specific position (coordinates in the image) of the image displayed on the display unit 230. 280. The instruction unit slave unit 250 gives an instruction to the image of the target space displayed on the display unit 230 by the user B. In the second processing device 200, the instruction unit slave unit 250 instructs the image on the whiteboard image 232 of the display unit 230. A cursor 242 is displayed at a position designated by the slave unit 250. The instruction unit parent device 280 transmits the position indicated by the instruction unit child device 250 to the first processing apparatus 100.

  FIG. 4 is a block diagram illustrating a configuration example of the instruction unit slave unit 250 that is a part of the instruction unit. Referring to the drawing, the instruction unit slave unit 250 includes a camera module 252, a switch 254, an arithmetic circuit 256, and a wireless module 258.

  The camera module 252 is a device having sensitivity in the infrared region, and captures infrared LEDs (infrared LEDs 240a to 240d) arranged at the four corners of the display unit 230 in FIG. 1 and specifies their positions. As a camera module, a visible light cut filter is attached to a camera equipped with a semiconductor image sensor such as a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. Can be used.

  The switch 254 is pressed when the user B wants to perform pointing, and is not particularly limited. For example, a general push switch can be used. The arithmetic circuit 256 is based on the positions of the infrared LEDs 240a to 240d photographed by the camera module 252 and the intersection of the direction indicated by the pointing unit slave unit 250 and the screen of the display unit 230, that is, the coordinates (X, Y) is calculated. Further, the arithmetic circuit 256 also detects a change in the pressed state of the switch 254 (button event) and passes it to the instruction unit master 280. As the arithmetic circuit 256, a general microcomputer can be used.

  The wireless module 258 transmits the in-video coordinates and the button event calculated by the arithmetic circuit 256 to the instruction unit master 280 via a wireless link. As the wireless module 258, for example, a wireless transceiver conforming to a wireless standard such as Bluetooth can be used. The connection between the instruction unit master 280 and the control unit 210 is not particularly limited, and a general-purpose interface can be used. For example, USB (Universal Serial Bus), IEEE 1394, or RS232C may be used.

  Returning to FIG. 1, the control unit 210 updates the display of the cursor 242 on the screen of the display unit 230 based on the in-video coordinates (X, Y) and the button event received from the instruction unit main unit 280, and the same information. Is included in a network packet and transmitted to the control unit 110 via the network 3. The button event transmitted to the control unit 110 is an instruction to control the visual change generated at the indicated position in the target space, that is, the on / off control of the light beam 170. The button event includes, for example, a button down event indicating that the switch 254 has transitioned to a pressed state, and a button up event indicating that the switch 254 has transitioned from a pressed state to a released state. Can do.

  In the first processing apparatus 100 and the second processing apparatus 200 of the present embodiment configured as described above, various processing operations corresponding to the computer program are executed by the CPU incorporated in each control circuit or the like. Thus, the various units and circuits as described above are realized as various functions.

  The computer program according to the present embodiment causes a computer for realizing the first processing device 100 and the second processing device 200 to display a captured image of the target space on the screen of the display unit 230 and to capture the target space. An instruction receiving procedure for receiving an instruction of a target coordinate from the instruction unit slave unit 250 on a screen on which an image is displayed, and a generation that causes the space pointing unit 140 to generate a visual change at a corresponding position in the target space with respect to the target coordinate It is described that a procedure and a control procedure for displaying on the screen of the display unit 230 an image in which a visual change that occurs in the target space as a result of the instruction is suppressed are described.

  The computer program of this embodiment may be recorded on a computer-readable recording medium. The recording medium is not particularly limited, and various forms can be considered. The program may be loaded from a recording medium into a computer memory, or downloaded to a computer through a network and loaded into the memory.

The operation of the pointing system 1 of the present embodiment configured as described above will be described below. Before describing the overall operation of the pointing system 1 including the first processing device 100 and the second processing device 200 according to the present embodiment, first, the imaging operation of the target space by the imaging unit 142 in the first processing device 100 and the space The generation operation of the visual change to the target space by the pointing unit 140 will be described in detail.
As shown below, the operations of the photographing unit 142 and the space pointing unit 140 are controlled by the control signal of the control unit 110 and synchronized with each other, thereby displaying the pointer displayed in the target space of the first processing device 100. The visual change in the image obtained by photographing 172, that is, the image of the pointer 172 by the light beam 170 can be suppressed and transmitted to the second processing device 200.

First, the operation of the imaging unit 142 in the first processing apparatus 100 of the present embodiment will be described.
FIG. 5 is a flowchart illustrating an example of the operation of the imaging unit 142 of the present embodiment.

  The imaging unit 142 captures a moving image in a space where an object to be pointed (in this example, the whiteboard 132 (FIG. 1)) is installed. For shooting a moving image, a series of operations of opening the shutter of the camera (step S11), exposing for a fixed time (exposure time) (step S13), and closing the shutter (step S15) is repeated at a high speed (for example, 30 Hz). (NO in step S17, step S19). In the system shown in the present embodiment, the opening timing and closing timing of the shutter are controlled by the control unit 110.

  For example, the control unit 110 transmits a rectangular wave control signal as shown in FIG. 6 to the imaging unit 142 at a repetition interval of about 33 milliseconds (ms), and the imaging unit 142 has a control signal of High. It is assumed that the shutter is opened in the state and is closed in the low state. Then, the photographing unit 142 opens the shutter at the rising timing of the control signal, performs exposure for 15 ms, and closes the shutter at the falling timing of the control signal. As a result, from the photographing unit 142, a video (still image sequence) of 30 frames per second is generated at the timing controlled by the control unit 110. The captured video is sent from the imaging unit 142 to the control unit 110 as needed.

  Here, as an example of the method in which the control unit 110 controls the photographing unit 142, the shutter is opened when the control signal is High, and the shutter is closed when the control signal is Low. A control method is also conceivable in which, after the shutter opens at the rising timing, the shutter is closed after a predetermined count of a clock built in the photographing unit 142 has elapsed. Since such control is generally known as a shutter control method for an IEEE 1394 camera or the like, detailed description thereof is omitted.

Next, the operation of the space pointing unit 140 of the first processing apparatus 100 of this embodiment will be described.
Hereinafter, an example of a specific laser output control method for the laser control process in the spatial pointing unit 140 of the first processing apparatus 100 of the present embodiment will be described with reference to FIGS. 9 and 10. FIG. 9 is a state transition diagram of the control circuit 146 of the space pointing unit 140 in accordance with an instruction from the instruction unit slave unit 250. FIG. 10 is a timing chart showing a relationship between a control signal transmitted from the control unit 110 to the imaging unit 142 and the space pointing unit 140, an open / close state of the shutter of the imaging unit 142, and an on / off state of the laser of the space pointing unit 140.

  When the control circuit 146 (FIG. 2) of the space pointing unit 140 in the initial state S21 receives a button down event from the control unit 110 (S31), the control state shifts to a standby state S23 in which the laser output is off. When the falling edge of the control signal received from the control unit 110 is detected in the standby state S23 (S33), the control circuit 146 transitions to the laser on state S25 and starts laser output (refer to the button down and after in FIG. 10). . At the same time, the duration of the laser-on state S25 is also measured. When the laser-on state S25 reaches a predetermined time (15 ms in this example) (S35), the laser output is stopped and the state transits to the standby state S23 again. This is repeated at the repetition frequency (about 30 Hz) of the control signal.

  In the standby state S23 and the laser-on state S25 from the reception of the button-down event, the control circuit 146 (FIG. 2) of the space pointing unit 140 responds to the deflection angle received from the control unit 110 according to the coil 164 of the galvano scanner 152 (FIG. 2). Current is passed through FIG. 3) to adjust the angle of the mirror 150 (FIG. 2). Here, the galvano scanner 152a is controlled independently in the horizontal deflection, and the galvano scanner 152b is controlled independently in the vertical deflection. Further, this control is usually performed within a much shorter time than the interval (in this example, 10 Hz, that is, 100 ms) at which the in-video coordinates (Xi, Yi) are sampled. Note that i (where i = 1, 2, 3,...) Indicates a value for each control cycle of the control unit 110.

  When a button-up event is received from the control unit 110 in the laser-on state S25 or the standby state S23 (S39 or S37) (when the laser output is on, the output is stopped), the state transitions to the initial state S21. In the initial state S21, the laser output remains off even if the falling edge of the control signal is detected (see after button up in FIG. 10).

  In this way, while the transition between the standby state S23 and the laser-on state S25 is repeated according to the control signal, the laser output is not constantly turned on, but control can be performed to generate pulsed light that repeatedly turns on and off. . This on / off control is controlled in synchronization with the photographing unit 142 in terms of time.

  That is, it is assumed that the photographing unit 142 operates as described above to perform exposure for 15 ms after the shutter is opened at the rising edge of the control signal. As a result of such control, as shown in FIG. 10, the time interval at which the shutter of the photographing unit 142 is opened and the time interval at which the laser output is on are alternately and complementarily generated.

  Thus, the frame of the video imaged by the imaging unit 142 is, for example, as shown in FIG. That is, the visual change (in this example, the light beam image 182) generated by the space pointing unit 140 (FIG. 1) is removed from the actual space state as shown in FIG. ) Is taken by the photographing unit 142 (FIG. 1). On the other hand, when the repetition frequency of the control signal is sufficiently high, such as 30 Hz in this example, the image 182 of the laser light beam is almost the same as seen from the person (for example, user A) in the space to be imaged due to the afterimage phenomenon. Looks like it's constantly lit.

  In this way, in the first processing apparatus 100, by controlling the operations of the light beam 170 of the imaging unit 142 and the space pointing unit 140 as described above, an image in which a visual change occurring in the target space is suppressed can be obtained. It can be transmitted to the second processing device 200. That is, in the pointing system 1 of the present embodiment, the control unit 110 and the control unit 210 have the space pointing unit 140 in the target space so that a time interval in which a visual change occurs and a time interval in which a visual change does not occur are repeated. A visual change composed of a series of images obtained by causing the photographing unit 142 to repeatedly photograph the target space at a time interval in which a visual change is generated at a corresponding position of the image and the space pointing unit 140 does not cause a visual change. An image in which the change is suppressed can be generated, and the generated image in which the visual change is suppressed can be displayed on the screen of the display unit 230.

  Next, a data processing method by the first processing apparatus 100 and the second processing apparatus 200 in the pointing system 1 of the present embodiment will be described below. 7 and 8 are flowcharts showing an example of the operation of the entire pointing system 1 of the present embodiment. In addition, about 2nd processing apparatus 200, a process is divided into the control part 210 and the instruction | indication part subunit | mobile_unit 250, and is demonstrated. Hereinafter, description will be made with reference to FIGS. 1, 4, 7, and 8.

  In the data processing method of the present embodiment, a captured image of the target space is displayed on the screen of the display unit 230 (step S205 in FIG. 7), and an instruction for target coordinates is received on the screen displaying the captured image of the target space. (YES in step S301 in FIG. 7), a visual change is generated at a corresponding position in the target space with respect to the target coordinates (step S115 in FIG. 8), and the visual change that occurs in the target space as a result of the instruction is generated. The suppressed video is displayed on the screen of the display unit 230 (step S205 in FIG. 7).

  In the flowcharts of FIGS. 7 and 8, after a series of processing ends, the process returns to the first step and the same processing is repeated. It is assumed that the process starts or ends when a start instruction or an end instruction (such as pressing a switch) is received from the user.

  As shown in FIG. 7, in the first processing apparatus 100, the video imaged by the imaging unit 142 (FIG. 1) (step S <b> 101 in FIG. 7) and sent to the control unit 110 (FIG. 1) (Not shown) (for example, H.264 standard) (step S103 in FIG. 7) and transmitted to the control unit 210 (FIG. 1) via the network 3 (FIG. 1) (step S105 in FIG. 7). ).

  In the second processing device 200, the control unit 210 (FIG. 1) receives the compressed video (YES in step S201 of FIG. 7), and decodes the received compressed video by a corresponding decoder (not shown) (step of FIG. 7). (S203), displayed on the display unit 230 (FIG. 1) (step S205 in FIG. 7), and proceeds to step S207. If no video is received in step S201 (NO in step S201 in FIG. 7), the process proceeds to step S207, bypassing steps S203 and S205.

  It is assumed that the user B (FIG. 1) looks at the whiteboard video 232 (FIG. 1) displayed on the display unit 230 and intends to point to a specific place on the whiteboard. In this case, the user B presses the switch 254 (FIG. 4) of the instruction unit slave unit 250 (FIG. 1) and performs a gesture such that the instruction unit slave unit 250 points to the screen. Then, the arithmetic circuit 256 of the instruction unit slave unit 250 receives the ON of the switch 254 (YES in step S301 in FIG. 7), and the camera module 252 (FIG. 4) built in the instruction unit slave unit 250 has infrared LEDs 240a to 240C. An image including d (FIG. 1) is taken (step S303 in FIG. 7). On the other hand, when pressing of the switch 254 is not detected in step S301 (NO in step S301 in FIG. 7), the process proceeds to step S309.

  Based on the information obtained in step S303, the arithmetic circuit 256 (FIG. 4) of the instruction unit slave unit 250 calculates the coordinates (X, Y) on the video screen indicated by the instruction unit slave unit 250 ( Step S305 in FIG. When the coordinates on the images of the infrared LEDs 240a to 240d photographed by the camera module 252 are (Xa, Ya), (Xb, Yb), (Xc, Yc), (Xd, Yd), respectively, the instruction unit slave unit The coordinates (X, Y) on the video screen indicated by 250 are given by the following equations (1) and (2).

X = K1 [{Ya-Xa (Ya-Yd) / (Xa-Xd) -Yb + Xb (Yb-Yc) / (Xb-Xc)} / {(Yb-Yc) / (Xb-Xc)-(Ya- Yd) / (Xa−Xd)}] (1)
Y = K2 {X (Ya−Yd) / (Xa−Yd) + Ya−Xa (Ya−Yd) / (Xa−Xd)} (2)

  Here, K1 and K2 are constants determined by the angle of view and the number of pixels of the camera module 252 and the arrangement interval of the infrared LEDs 240a to 240d, and are obtained by performing calibration. Note that the method for obtaining coordinates on the video screen described here is described in Patent Document 3 (Japanese Patent No. 3422383) and the like, and thus the details are omitted.

  The coordinates (X, Y) on the video screen calculated as a result of the above processing are sent to the control unit 210 via the wireless module 258 (FIG. 4) and the instruction unit master 280 (FIG. 1) (FIG. 7). Step S307). At this time, a button down event is also transmitted.

  When the control unit 210 receives the button down event and the coordinates (YES in step S207 of FIG. 7), the control unit 210 moves the cursor 242 (FIG. 1) to the position of the coordinates (X, Y) of the video obtained from the imaging unit 142. The image is displayed superimposed on the screen of the display unit 230 (step S209 in FIG. 7). The cursor 242 to be displayed may be an abstract symbol such as an arrow, but it is desirable that the cursor 242 be close to the visual change generated by the spatial pointing unit 140. For example, in this example, a cursor 242 reminiscent of irradiation with a red laser pointer is desirable.

  Here, it is assumed that the user B performs a gesture of drawing a locus such as a circle with the instruction unit slave unit 250 while pressing the switch 254 of the instruction unit slave unit 250 (NO in step S321 in FIG. 8). While the switch 254 is being pressed (NO in step S321 in FIG. 8), the process returns to step S309 in FIG. 7, and a plurality of in-video coordinates (X, Y) is sent at intervals (for example, 10 Hz) at which the camera module 252 captures the infrared LED 240 (step S309, step S311, and step S312 in FIG. 7). These coordinates are called (Xi, Yi) (where i = 1, 2, 3,...). Upon receiving the in-video coordinates (Xi, Yi) (YES in step S213 in FIG. 8), the control unit 210 sequentially updates the display position of the cursor 242 on the screen of the display unit 230 (step S215 in FIG. 8). Therefore, on the display unit 230 screen, the cursor moves along the locus drawn by the gesture of the user B.

When the switch 254 is released from the depressed state (YES in step S321 in FIG. 8), a button up event indicating that the switch 254 has transitioned from the depressed state to the released state is displayed in the control unit 210. Notification is made (step S323 in FIG. 8).
As described above, in step S307 in FIG. 7, the instruction unit slave unit 250 transits to the control unit 210 via the instruction unit master unit 280 in addition to the in-video coordinates (Xi, Yi) and the switch 254 is depressed. It is assumed that a button-down event indicating that the switch has been performed is notified to the control unit 210, and a button-up event indicating that the switch 254 has been changed from the pressed state to the released state is notified to the control unit 210. Using these button events (YES in step S207 in FIG. 7, YES in step S221 in FIG. 8), the control unit 210 displays the cursor 242 on the display unit 230 screen (step S209 in FIG. 7) or deletes ( Step S223) of FIG. 8 is controlled.

  As described above, the control unit 210 reflects the received in-video coordinates (Xi, Yi) and the button event on the display of the cursor 242 on the screen of the display unit 230, and includes the same information in the network packet so that the network 3 (Step S211 in FIG. 7, step S217 in FIG. 8, step S225). For example, here, assuming that an average delay of about T milliseconds (ms) occurs in the network 3, the control unit 110 causes the user B to press the switch 254 of the instruction unit slave unit 250 to display the screen on the display unit 230. The button event and the in-video coordinates (Xi, Yi) are received about T [ms] after pointing (YES in step S111 in FIG. 8).

  In the first processing apparatus 100, the control unit 110 converts the in-video coordinates (Xi, Yi) received from the control unit 210 into the laser deflection angle of the spatial pointing unit 140 (step S113 in FIG. 8). Assuming that the spatial pointing unit 140 is close to the photographing unit 142 and the optical axis of the camera and the laser can be approximated, the horizontal deflection angle αih and the vertical deflection angle αiv of the laser are as follows: It is calculated by the formula. Here, i indicates a value for each control cycle of the control unit 110.

tanαih = (2Xi / Xnum−1) · tanθh / 2 Formula (3)
tanαiv = (2Yi / Ynum−1) · tanθv / 2 Formula (4)

  Here, Xnum and Ynum are the horizontal and vertical resolutions of the video imaged by the imaging unit 142, respectively, and θh and θv are the horizontal and vertical field angles of the imaging unit 142, respectively.

  The control unit 110 transmits the calculated deflection angle to the control circuit 146 of the spatial pointing unit 140, controls the deflection angle of the light beam 170 by the spatial pointing unit 140, and responds to the button event received from the second processing device 200. Then, on / off of the light beam 170 is controlled (step S115 in FIG. 8).

  In this manner, in the pointing system 1 of the present embodiment, the control unit 110 and the control unit 210 allow the spatial pointing unit 140 to repeat a time interval in which a visual change occurs and a time interval in which a visual change does not occur. It is composed of a series of images obtained by causing the photographing unit 142 to repeatedly photograph the target space at a time interval that causes a visual change at a corresponding position in the target space and the spatial pointing unit 140 does not cause a visual change. Thus, an image with suppressed visual change can be generated, and the generated image with suppressed visual change can be displayed on the screen of the display unit 230.

  A specific example when each unit operates in accordance with the operation of the pointing system 1 described above will be described below with reference to FIGS. First, the operation timing of each unit of the entire pointing system 1 will be described, and then the image displayed on the screen of the display unit 230 of the second processing device 200 and the position designated by the user on the second processing device 200 side. The visual change generated at the position in the target space on the first processing apparatus 100 side corresponding to the above will be described with a specific example.

  FIG. 12 is a timing chart showing an example of the operation of each unit of the pointing system 1. As shown in the figure, the video 192 photographed by the photographing unit 142 is compressed and encoded by an appropriate encoder by the control unit 110 and transmitted to the control unit 210 via the network 3 as described above. After being decoded by the control unit 210, it is displayed on the display unit 230. At this time, the network 3 receives a delay of about T [ms]. In addition, the transfer of the in-video coordinates (Xi, Yi) and the button event 194 is similarly delayed by about T [ms] in the network 3. That is, there is a delay of about 2T [ms] from when the user B gives an instruction using the instruction unit slave unit 250 until the video is reflected on the screen of the display unit 230.

  As described above, in the display unit 230, while the switch 254 of the instruction unit slave unit 250 is being pressed down, the cursor 242 reminiscent of the irradiation of the red laser pointer with the coordinates (Xi, Yi) in the captured image. Is superimposed and displayed. Accordingly, as shown in FIG. 13B, the video displayed on the display unit 230 is a screen 302 as compared to the photographed video 192 shown in FIG.

  Here, as illustrated in FIG. 14, it is assumed that the user B performs a gesture of drawing a locus such as a circle with the instruction unit slave unit 250 while pressing the switch 254 of the instruction unit slave unit 250. Then, while the switch 254 is pressed, a plurality of in-video coordinates (Xi, Yi) representing the locus 304 is sent from the instruction unit slave unit 250 to the control unit 210 at regular intervals. When the control unit 210 receives the in-video coordinates (Xi, Yi), the display position of the cursor 242 is immediately updated.

  Therefore, on the display unit 230 screen, the cursor 242 is displayed at the position A in FIG. On the other hand, the visual change generated by the space pointing unit 140 controlled by the in-video coordinates (Xi, Yi) sent from the instruction unit slave unit 250 occurs at the position B in the actual space. Due to the control mechanism of the photographing unit 142 and the space pointing unit 140, the image does not appear on the screen of the display unit 230.

  As described above, according to the pointing system 1 of the present invention, the spatial pointing unit 140 and the control unit 110 that controls the imaging unit 142 are delayed so as to cause the imaging unit 142 to capture images with suppressed visual changes. By providing the display unit 230 that displays an image in which visual feedback is suppressed, problems of display delay and pointing accuracy and operability can be solved.

  According to the present invention, for example, when the remote pointing system 1 that points to and refers to a real object in a remote place reflected in a video image is delayed, the user B is provided with a pointing feeling equivalent to a situation where there is no delay. The effect of shortening the time required for pointing and improving the pointing accuracy can be achieved.

  The reason why such an effect occurs is that the control units 110 and 210 of the present invention control the spatial pointing unit 140 and the photographing unit 142 that cause visual changes in a remote space such as a laser pointer, and the degree of visual change. This is to generate a video with less and suppress delayed visual feedback. As a result, the user B has an illusion that the mark such as the cursor displayed on the display unit 230 is an actual image of the laser pointer. Since the cursor responds to the user's pointing operation with almost no delay, even when there is a delay in the network 3 with the remote place, the user B can be given a feeling of pointing equivalent to the situation without a delay. In this way, confusion due to delayed visual feedback, such as the image of a laser pointer in the video, at user B can be avoided.

  According to the pointing system of the present invention, the usability of the remote pointing system for pointing and referring to the real object in the remote location shown in the video image is improved in the situation where the transmission of information between the remote locations is delayed as described above. can do. More specifically, in a situation where there is a delay, the user B can provide a feeling of pointing equivalent to that in a situation where there is no delay, and the time required for pointing can be reduced and the pointing accuracy can be improved.

  In the above embodiment, as a result of not displaying an image such as an actual laser pointer on the display unit 230, the difference between the position pointed to by the user B and the position pointed to in the actual space is described in the above-described literature. Although it seems that it increases compared with the technology of no, etc., it is not actually such a thing. For example, as shown in FIG. 15, it is assumed that the user B points on the screen at a constant speed in the horizontal direction. It is assumed that the position A is indicated at the time t, and the positions B and C are indicated at the times t−Δt and t−2Δt, respectively. If the one-way delay through the network is Δt, the coordinates acquired at time t−Δt, that is, the coordinates of B are reflected in the space pointing unit 140 at time t. The position pointed to in the actual space is B.

  On the other hand, since what is displayed on the screen is an image taken at time t−2Δt, if the image of the pointer is taken, it is displayed at the position C. Therefore, in the feedback method using the pointer image, a deviation corresponding to Δt occurs from the position indicated in the actual space. On the other hand, in the present invention, the cursor is visually fed back to the position A indicated by the user B and displayed. Similar to the feedback method using the image of the pointer, this position also has a difference corresponding to Δt from the position indicated in the actual space. Therefore, also in the method of the present technology, the difference between the visual feedback and the position indicated in the actual space is the same, although the direction of the difference is different.

Note that the divergence of properties caused by the present invention is advantageous in a system using voice channels. This is because, as shown in FIG. 15, the user B utters a character at a location corresponding to visual feedback while pointing on the screen of the display unit 230 at a constant speed in the horizontal direction using the pointing unit slave unit 250. Shall.
The uttered voice is collected by a microphone (not shown), compressed and encoded by an appropriate encoder (for example, AAC (Advanced Audio Coding) standard) (not shown), and the control unit 110 via the network 3 through the control unit 110. Can be sent to.

  If the audio data and the in-video coordinate data are subjected to the same delay Δt, both the audio data and the in-video coordinate data arrive almost simultaneously at the remote location on the first processing device 100 side. At t, the space pointing unit 140 points to the position B, and the voice of “I” uttered by the user B is reported to the remote place.

  On the other hand, in the feedback method using the pointer image, for example, the user B utters “A” for the first time when the pointer image is displayed at the position C at time t, so that the voice arrives at a remote place at time t + Δt. When reported, the space pointing unit 140 points to the position A, that is, the letter “u”, and a difference occurs between the position indicated by the voice and the laser pointer. As described above, compared with the feedback method using the pointer image, the present invention exhibits excellent characteristics even in combination with the voice instruction indispensable for the pointing operation.

(Second Embodiment)
The pointing system 1 of the present embodiment is different from the above-described embodiment in that a video (first video) during which the photographing unit 142 causes this visual change and a video during which no visual change occurs. The difference is that the (second video) is photographed separately. Each component of the pointing system 1 of the present embodiment has the same configuration as that of the above-described embodiment, and is generated by the control unit 110 and the control unit (110, 210). Different video generation methods.

  The pointing system of the present embodiment further includes an imaging unit 142 that images the target space, and the control unit (110, 210) repeats a time interval in which a visual change occurs and a time interval in which no visual change occurs. A first change is made up of a series of images obtained by causing the space pointing unit 140 to generate a visual change at a corresponding position in the target space, and the imaging unit 142 to repeatedly capture images at time intervals when the visual change occurs. A second video composed of a video and a series of images obtained by causing the photographing unit 142 to repeatedly shoot at a time interval at which no visual change occurs is acquired, and based on the first video and the second video Then, an image in which the visual change is suppressed is generated, and the generated image in which the visual change is suppressed is displayed on the screen of the display unit 230.

  In the present embodiment, like the above embodiment, the spatial pointing unit 140 has a time interval that causes a visual change and a time interval that does not cause a visual change.

  Hereinafter, an example of a laser output control method of the pointing system 1 of the present embodiment will be described with reference to FIGS. 16 and 17. FIG. 16 is a state transition diagram of the control circuit 146 of the space pointing unit 140. FIG. 17 is a graph showing a relationship between a control signal transmitted from the control unit 110 to the photographing unit 142 and the space pointing unit 140, an open / close state of the shutter of the photographing unit 142, and an on / off state of the laser of the spatial pointing unit 140.

  The state transition diagram of FIG. 16 differs from FIG. 9 of the above embodiment in the operation in the standby state S23. That is, in the above embodiment, when the falling edge of the control signal received from the control unit 110 is detected in the standby state S23 (S33), the control circuit 146 transitions to the laser on state S25 and starts laser output. In the present embodiment, when the rising edge of the control signal received from the control unit 110 is further detected in the standby state S23 (S41), the control circuit 146 transitions to the laser-on state S25 and starts laser output. In either case, the duration of the laser-on state S25 is also measured at the same time. When the laser-on state S25 reaches a predetermined time (in this example, 15 ms) (S35), the laser output is stopped and the standby state is resumed. Transition to S23.

  Then, as shown in FIG. 17, the control unit 110 transmits a control signal to the space pointing unit 140 and the imaging unit 142 at intervals of about 33 ms. The space pointing unit 140 sets the laser output to ON for 15 ms in accordance with the detection of the rising edge of the control signal from the button down event to the button up event. On the other hand, the imaging unit 142 opens the shutter for 15 ms both at the rise and fall of the control signal. As a result, as shown in FIG. 18 (a), an image 311 in which a visual change has occurred in space in an odd frame, and a visual change in a space has occurred in an even frame, as shown in FIG. 18 (b). No video 312 is captured at a frequency of 30 Hz. Here, the former is called a first video 311 and the latter is called a second video 312.

  The control unit 110 adds the first video 311 to the second video 312 with transparency through image processing, and generates a third video 313 as shown in FIG. Here, as a method of adding images with transparency, generally known alpha blending can be used. It is desirable that the degree of transmission can be changed by the user.

  That is, in the pointing system 1 of the present embodiment, the control unit 110 generates the third video 313 in which the visual change is suppressed by adding the first video 311 and the second video 312 with different weights. To do.

  After the third video 313 is generated in this manner, the third video 313 is compressed and encoded by an appropriate encoder by the control unit 110 and the control unit via the network 3 as in the above embodiment. 210, decoded by the control unit 210, and displayed on the screen of the display unit 230.

  Further, in the display unit 230, while the switch of the instruction unit slave unit 250 is pressed, a cursor 242 that reminds the irradiation of the red laser pointer is superimposed on the in-video coordinates (Xi, Yi) in the captured video. Display on the screen. Therefore, when the user B performs a gesture of drawing a locus such as a circle with the instruction unit slave unit 250 while pressing the switch of the instruction unit slave unit 250, the display unit 230 has a cursor 242 and a cursor 242 as shown in FIG. The image 321 such as the laser pointer is displayed with different intensities (the laser pointer image is lighter).

  In the present embodiment, a configuration for adjusting the intensity of the image 321 such as a laser pointer may be provided. For example, the pointing system 1 includes a receiving unit (not shown) that receives an intensity level adjustment instruction for the image 321, an adjusting unit (not shown) that adjusts the intensity of the image 321 according to the received level, and the adjusted video. A confirmation unit (not shown) that presents a preview screen to the user A or B and prompts confirmation may be further provided. The reception unit, the storage unit, and the confirmation unit may be provided in either the control unit 110 or the control unit 210, and either one or both of them may be provided partially or entirely.

  For example, the reception unit prompts the user A or B on the adjustment setting screen to input the percentage or numerical value of the transparency of the first video 311 or operate the slider for level adjustment, and receives the setting to adjust the level. Accept as an instruction. When the level is adjusted on the adjustment setting screen, the confirmation unit displays the adjusted image on the preview screen and presents it to the user A or B for prompting confirmation. The user A or B can adjust the level while referring to the adjustment result.

  User B adjusts the transparency of the first video 311, that is, the intensity of the laser pointer image on the display unit 230 to a level at which he / she does not care about the confusion due to delayed visual feedback in the video. Can be avoided. At the same time, it is possible to recognize the laser pointer image in the actual space by gazing at the image of the laser pointer or adjusting the transparency again, so that the space pointing device is operating correctly. It is possible to realize a pointing operation with high accuracy while checking the operation and knowing the delay state of the network.

  In the above example, the first video 311 is added to the second video 312 with transparency, but the difference between the first video 311 and the second video 312, that is, the laser pointer. It is also conceivable to convert the image-only image into a black and white image and then add it to the second image 312 with transparency again. By doing so, it is possible to further weaken the influence of the image of the laser pointer on the recognition of the user B.

  As described above, according to the pointing system 1 of the present embodiment, the same effects as those of the above-described embodiment can be obtained, and the intensity of the laser pointer image on the display unit 230 can be adjusted to a level that is not noticeable. This avoids confusion due to delayed visual feedback in the video.

(Third embodiment)
The pointing system 1 according to the present embodiment is different from the above embodiment in that the video generation method that suppresses the visual change uses the color information of the pixels in the vicinity of the coordinates in the video designated by the instruction unit slave unit 250 as the surroundings. The difference is that the pixel is replaced with a value close to the color information of the pixel. For other configurations and operations, the pointing system 1 of the present embodiment may be combined with any of the above embodiments.

  The pointing system 1 according to the present embodiment further includes an imaging unit 142 that captures the target space, and the control units (110, 210) of pixels near a position where a visual change occurs with respect to an image captured by the imaging unit 142. By replacing the color information with a value close to the color information of the surrounding pixels, an image in which the visual change is suppressed is generated, and the generated image in which the visual change is suppressed is displayed on the screen of the display unit 230.

  In the present embodiment, control unit 110 does not perform control to repeat the time interval at which space pointing unit 140 causes a visual change and the time interval at which a visual change does not occur. Instead, the control unit 110 suppresses the visual change by replacing the color information of the pixels in the vicinity of the in-video coordinates specified by the instruction unit slave unit 250 with a value close to the color information of the surrounding pixels. Generate video.

  In this embodiment, when the control unit 110 receives the in-video coordinates (Xi, Yi) pointed to by the instruction unit slave unit 250 from the control unit 210, the control unit 110 converts the coordinate into the laser deflection angle of the space pointing unit 140 and the space pointing unit 140. Control. After the control is completed, the control unit 110 performs the following process on the video frame obtained from the photographing unit 142.

  The control unit 110 uses the color information of the pixels included in the circular area A with the diameter a centered on the in-video coordinates (Xi, Yi) of the video frame, and the colors of the pixels included in the annular area with the outer diameter b and the inner diameter a. Replace with the value interpolated by the information (see FIG. 20). In the simplest case, the area A may be uniformly filled with the average value of the color information of the pixels in the area B. Alternatively, for example, the solution of the Laplace equation (∇2F (X, Y) = 0; F (X, Y) is the color intensity) given the pixels in the region B as boundary conditions may be used as the color information of the pixels in the region A. good. Since the numerical solution of the Laplace equation is described in general numerical analysis literature, the details are omitted.

  Note that the values of the radii a and b may be predetermined values, or may be automatically extracted by analyzing pixels around the in-video coordinates (Xi, Yi). This method will be described using an example in which a red laser pointer is used for the space pointing unit 140. In a circular area with a diameter c centered on the in-video coordinates (Xi, Yi) (the value of c is given in advance), only red information is extracted. As a result, it is assumed that a red intensity distribution is obtained as shown by a solid line in FIG. 21 (in the figure, the description is made in one dimension for simplicity, but the actual intensity distribution is two-dimensional).

  When this intensity curve is approximated to a Gaussian curve by the least square method (dotted line in the figure), a variance value σ of the Gaussian curve is obtained. For example, the diameter a is set to 6 times σ. Then, when the intensity distribution can be approximated by a Gaussian curve, about 99.7% of the visual change generated by the spatial pointing unit 140 is included in the region of the diameter a. The diameter b may be set to a + fixed value or a × fixed value.

  In the video obtained in this way, since the degree of visual change is small compared to the actual space, user B avoids confusion due to delayed visual feedback such as the laser pointer image 331 in the video. Can do.

  Thereafter, as in the above-described embodiment, the video is compressed and encoded by an appropriate encoder by the control unit 110, transmitted to the control unit 210 via the network 3, decoded by the control unit 210, and displayed on the display unit 230. Is done. Further, in the display unit 230, while the switch of the instruction unit slave unit 250 is pressed, a cursor reminiscent of the irradiation of the red laser pointer is superimposed on the in-image coordinates (Xi, Yi) in the captured image. indicate.

  As described above, according to the pointing system 1 of the present embodiment, the same effects as those of the above-described embodiment can be achieved.

  As mentioned above, although embodiment of this invention was described with reference to drawings, these are the illustrations of this invention, Various structures other than the above are also employable.

  For example, in the above embodiment, an example of a laser pointer capable of controlling the deflection direction has been described as the space pointing unit 140, but other space pointing means such as a projector may be used.

  In the above-described embodiment, the control method in which the control unit 110 transmits the same control signal to both the photographing unit 142 and the space pointing unit 140 has been described. A method is also conceivable in which one of the sending and receiving signals generates a control signal for the other. Such a difference is well known in the field of synchronous control technology using a clock signal, and therefore description of a configuration with different details is omitted.

  Furthermore, in the above-described embodiment, the display unit 230 and the instruction unit (the instruction unit slave unit 250 and the instruction unit parent unit 280) are installed at one site, and the imaging unit 142 and the space pointing unit 140 are installed at the other site. Although the configuration is shown, a configuration is possible in which these are installed one by one at each base and can be pointed in both directions. In addition, an example of a remote-controlled position specifying device has been shown as the instruction unit. However, for example, a touch panel mounted on the display unit 230 or a controller such as a touch pen, a mouse, or a trackball is used. You can also use it.

  While the present invention has been described with reference to the embodiments and examples, the present invention is not limited to the above embodiments and examples. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Pointing system 3 Network 100 1st processing apparatus 110 Control part 132 White board 140 Spatial pointing part 142 Image | photographing part 146 Control circuit 148 Laser 150 Mirror 152 Galvano scanner 160 Spring 162 Magnet 164 Coil 166 Magnetic body 170 Light beam 172 Pointer 200 Second Processing unit 210 Control unit 230 Display unit 232 Whiteboard image 240 Infrared LED
242 Cursor 250 Pointer unit 252 Camera module 254 Switch 256 Arithmetic circuit 258 Wireless module 280 Pointer unit 311 First image 312 Second image 313 Third image

Claims (17)

Display means for displaying the captured video of the target space on the screen;
Instruction accepting means for accepting an instruction of target coordinates on the screen displaying the captured video of the target space;
Spatial pointing means for generating a visual change at a corresponding position in the target space with respect to the target coordinates;
Control means for displaying an image on which the visual change occurring in the target space as a result of the instruction is suppressed is displayed on the screen;
A pointing system with The pointing system according to claim 1,
It further comprises a photographing means for photographing the target space,
The control means includes
Causing the spatial pointing means to generate the visual change at the corresponding position in the target space so that a time interval in which the visual change occurs and a time interval in which the visual change does not occur are repeated;
Generating a video that suppresses the visual change composed of a series of images obtained by causing the imaging unit to repeatedly capture the target space at a time interval in which the spatial pointing unit does not cause the visual change,
A pointing system that displays the generated video on which the visual change is suppressed on the screen. The pointing system according to claim 1,
It further comprises a photographing means for photographing the target space,
The control means includes
Causing the spatial pointing means to generate the visual change at the corresponding position in the target space so that a time interval in which the visual change occurs and a time interval in which the visual change does not occur are repeated;
A first video composed of a series of images obtained by repeated shooting by the shooting unit at a time interval in which the visual change occurs, and the shooting unit repeatedly shot at a time interval in which the visual change does not occur. To obtain a second video composed of a series of images obtained by
Based on the first video and the second video, generating a video that suppresses the visual change,
A pointing system that displays the generated video on which the visual change is suppressed on the screen. The pointing system according to claim 3,
The control means includes
A pointing system for generating a video in which the visual change is suppressed by adding the first video and the second video with different weights. The pointing system according to claim 1,
It further comprises a photographing means for photographing the target space,
The control means includes
A video in which the visual change is suppressed is generated by replacing the color information of the pixel in the vicinity of the position where the visual change occurs with a value close to the color information of the peripheral pixel with respect to the video shot by the shooting unit. ,
A pointing system that displays the generated video on which the visual change is suppressed on the screen. The pointing system according to any one of claims 2 to 5,
The pointing unit is a pointing system that displays a cursor on the screen in such a manner that a cursor is superimposed on an image in which the visual change is suppressed at an instruction position corresponding to the target coordinate received by the instruction receiving unit. The pointing system according to any one of claims 2 to 6,
A first processing device including the imaging unit and the space pointing unit;
A second processing device including the instruction receiving means and the display means,
The first processing apparatus and the second processing apparatus are provided separately from each other and are connected via a network. The pointing system according to claim 7,
The target coordinates received by the instruction receiving unit of the second processing apparatus are transmitted to the first processing apparatus via the network, and the space pointing unit of the first processing apparatus is configured to transmit the target coordinates to the target coordinates. Causing a visual change in the corresponding position in the object space,
The pointing system in which the video generated by the generation unit of the first processing device is transmitted to the second processing device via the network and displayed on the screen of the display unit of the second processing device. Spatial pointing means for generating a visual change at a corresponding position in the target space with respect to target coordinates indicated on a screen displaying a captured image of the target space;
And a control unit configured to display on the screen an image in which the visual change generated in the target space as a result of the instruction is suppressed. The processing apparatus according to claim 9, wherein
It further comprises a photographing means for photographing the target space,
The control means includes
Causing the spatial pointing means to generate the visual change at the corresponding position in the target space so that a time interval in which the visual change occurs and a time interval in which the visual change does not occur are repeated;
Generating a video that suppresses the visual change composed of a series of images obtained by causing the imaging unit to repeatedly capture the target space at a time interval in which the spatial pointing unit does not cause the visual change,
A processing apparatus for displaying the generated video on which the visual change is suppressed on the screen. The processing apparatus according to claim 9, wherein
It further comprises a photographing means for photographing the target space,
The control means includes
Causing the spatial pointing means to generate the visual change at the corresponding position in the target space so that a time interval in which the visual change occurs and a time interval in which the visual change does not occur are repeated;
A first video composed of a series of images obtained by repeated shooting by the shooting unit at a time interval in which the visual change occurs, and the shooting unit repeatedly shot at a time interval in which the visual change does not occur. To obtain a second video composed of a series of images obtained by
Based on the first video and the second video, generating a video that suppresses the visual change,
A processing apparatus for displaying the generated video on which the visual change is suppressed on the screen. The processing apparatus according to claim 11, wherein
The control means includes
The processing apparatus which produces | generates the image | video which suppressed the said visual change by adding said 1st image | video and said 2nd image | video with different weighting. The processing apparatus according to claim 9, wherein
It further comprises a photographing means for photographing the target space,
The control means includes
A video in which the visual change is suppressed is generated by replacing the color information of the pixel in the vicinity of the position where the visual change occurs with a value close to the color information of the peripheral pixel with respect to the video shot by the shooting unit. ,
A processing apparatus for displaying the generated video on which the visual change is suppressed on the screen. Display means for displaying the captured video of the target space on the screen;
Instruction accepting means for accepting an instruction of target coordinates on the screen displaying the captured video of the target space;
A processing apparatus comprising: control means for displaying, on the screen, an image in which a visual change generated at a corresponding position in the target space is suppressed with respect to the target coordinates as a result of the instruction. The processing apparatus according to claim 14, wherein
The processing device that displays a cursor at an instruction position on the screen corresponding to the target coordinates received by the instruction receiving unit. Display the video of the target space on the screen,
Receiving an instruction of target coordinates on the screen displaying the captured video of the target space;
Generating a visual change at a corresponding position in the target space with respect to the target coordinates;
A data processing method of a processing device for displaying on the screen an image in which the visual change that occurs in the target space as a result of the instruction is suppressed. A computer program for realizing a processing device,
A display procedure for displaying the captured image of the target space on the screen;
An instruction receiving procedure for receiving an instruction of a target coordinate on the screen displaying the captured video of the target space;
A generation procedure for generating a visual change at a corresponding position in the target space with respect to the target coordinates;
The computer program for making a computer perform the control procedure which displays the image | video which suppressed the said visual change which arises in the said object space as a result of the said instruction | indication on the said screen.

Images