JP2018010578A - Image processing apparatus, image processing system and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve a detection accuracy of coordinates.SOLUTION: An image processing apparatus is connected to an imaging apparatus that photographs a plurality of operation devices and to a display device. The image processing apparatus generates an observation point showing a present position of an operation device according to image data inputted from the imaging apparatus, generates a prediction point to be a next position of the present position according to a movement of the operation device, determines states of the observation point and the prediction point, and makes the display device display a line connecting observation points according to the states.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing system, and an image processing method.

  2. Description of the Related Art Conventionally, a method of displaying characters and figures written by a user, symbols, pictures, or a combination thereof (hereinafter simply referred to as “characters”) on a display device using a touch panel, an electronic pen, or the like, as in a so-called whiteboard. It has been known.

In this method, a method of detecting the coordinates of a point where the user performs an operation such as writing is performed.
Also, a method is known in which the state of the point at which the operation is performed is divided into “clear” and “merged”, and different weighting is performed (eg, Patent Document 1).

  However, when a plurality of electronic pens are used, the locus of each electronic pen may be erroneously recognized. Therefore, the problem is that the accuracy of coordinate detection deteriorates due to erroneous recognition.

  One aspect of the present invention aims to improve the accuracy of coordinate detection.

In one aspect, an image processing apparatus that images a plurality of operating devices and an image processing apparatus connected to the display apparatus are
An input unit for inputting image data from the imaging device;
Based on the image data, an observation point generation unit that generates an observation point indicating a current position of the controller,
A prediction point generation unit that generates a prediction point that is the next position of the current position based on the motion of the operation unit;
A determination unit for determining a state of the observation point and the prediction point;
An output unit that displays a line connecting the observation points on the display device based on the state.

  Coordinate detection accuracy can be improved.

1 is a conceptual diagram illustrating an example of the overall configuration of an image processing system according to an embodiment of the present invention. It is a conceptual diagram which shows the example of calculation of the coordinate by the image processing apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows the hardware structural example of the image processing apparatus which concerns on one Embodiment of this invention. It is a schematic diagram which shows the hardware structural example of the operating device which concerns on one Embodiment of this invention. It is a flowchart which shows an example of the whole process by the image processing apparatus which concerns on one Embodiment of this invention. It is a figure which shows the example of a production | generation of the observation point which concerns on one Embodiment of this invention. It is a figure which shows an example of the observation point and prediction point which concern on one Embodiment of this invention. It is a figure showing an example in case there are a plurality of observation points concerning one embodiment of the present invention. It is a flowchart which shows an example of the node-ization process by the image processing apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of the link by the image processing apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of the determination process by the image processing apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of the allocation method of ID by the image processing apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of the link in the "state 5" by the image processing apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of misrecognition. It is a figure which shows an example of the "merged state." It is a figure which shows an example of the effect by the image processing apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of the light-blocking system which concerns on one Embodiment of this invention. 1 is a functional block diagram illustrating an example functional configuration of an image processing system and an image processing apparatus according to an embodiment of the present invention.

  Embodiments of the present invention will be described below.

<First Embodiment>
FIG. 1 is a conceptual diagram showing an example of the overall configuration of an image processing system according to an embodiment of the present invention. As illustrated, the image processing system 1 includes four cameras, which are examples of a plurality of first imaging devices, and an image processing device 10 such as a PC (Personal Computer). Note that there may be three or more first imaging devices.

  As shown in the figure, for example, if the viewing angle of each camera is about 90 °, each camera is installed at four corners or the like so as to easily capture the entire plane of the display DIS as an example of the display device. . Hereinafter, in the figure, among the four cameras, the camera illustrated in the upper left is referred to as the upper left camera CAM11. Similarly, of the four cameras, the camera shown in the upper right is called the upper right camera CAM12. Further, of the four cameras, the camera illustrated in the lower right is referred to as a lower right camera CAM13. Furthermore, of the four cameras, the camera illustrated in the lower left is referred to as a lower left camera CAM14.

  As shown in the figure, the image processing apparatus 10 inputs image data captured by the upper left camera CAM11, upper right camera CAM12, lower right camera CAM13, and lower left camera CAM14, and based on the image data, a subject such as an electronic pen PE Is calculated, ie the position.

  The image processing system 1 and the display DIS are used as follows, for example. For example, the user performs an operation such as writing a character or the like by a touch operation or handwriting, using the electronic pen PE which is an example of an operation device. Specifically, when writing a character or the like at an arbitrary point on the display DIS, the user brings the electronic pen PE into contact with the display DIS at a position where the character or the like is written. Each camera captures the electronic pen PE to generate image data. Based on each image data generated in this manner, the coordinates of the electronic pen PE or a predetermined portion of the electronic pen PE are calculated.

  The coordinates are calculated as follows, for example.

  FIG. 2 is a conceptual diagram illustrating a coordinate calculation example by the image processing apparatus according to the embodiment of the present invention. First, in the following description, an example in which the pen tip of the so-called electronic pen PE is a predetermined location will be described. That is, as illustrated, this example is an example in which coordinates where the pen tip of the electronic pen PE is positioned are output as a calculation result. The predetermined location may be a location other than the pen tip of the electronic pen indicating the position of the electronic pen.

In the following description, a three-dimensional coordinate system indicating a predetermined portion of the electronic pen PE is referred to as a “world coordinate system”. In the illustrated example, the horizontal direction is the X axis in the figure, and the depth direction is the Y axis in the figure. In the drawing, the vertical direction is the Z axis. In the world coordinate system on the display DIS, the value of the Z axis is assumed to be constant, the description of the value of the Z axis is omitted, and the world coordinate (X _world , Y _world ) is indicated. Further, the principal point of the camera, that is, the center position of the image pickup element of the camera is defined as the principal point CAMOR.

  On the other hand, the coordinates in an image captured by each camera are referred to as “camera coordinate system” (also referred to as “image coordinate system” in the following description).

In order to calculate the world coordinates (X _world , Y _world ), as shown in the drawing, first, two pieces of image data (hereinafter referred to as “left image data IMGL” and “right image data” out of image data captured by a plurality of cameras are used. IMGR ") is selected by the image processing apparatus. The camera coordinates in the left image data IMGL are “(x ₁ , y ₁ )”, while the camera coordinates in the right image data IMGR are “(x ₂ , y ₂ )”. As shown in the figure, since each camera captures images with the principal point CAMOR as a base point, as shown in FIG. Imaged.

When (x ₁ , y ₁ ) and (x ₂ , y ₂ ) are obtained, the image processing apparatus can calculate the world coordinates (X _world , Y _world ) by triangulation or the like. Note that triangulation is realized by a known technique or the like.

<Hardware configuration example of image processing apparatus>
FIG. 3 is a block diagram illustrating a hardware configuration example of the image processing apparatus according to the embodiment of the present invention. For example, the image processing apparatus 10 includes a CPU (Central Processing Unit) 10H1, a ROM (Read-Only Memory) 10H2, a RAM (Random Access Memory) 10H3, a keyboard 10H4, and a mouse 10H5. The image processing apparatus 10 further includes a hard disk 10H6, a graphic board 10H7, a network card 10H8, an I / F (interface) 10H9, and a program reading apparatus 10H10.

  The CPU 10H1 is an arithmetic device that performs calculations and data processing for realizing each process. The CPU 10H1 is a control device that controls each hardware. Further, the CPU 10H1 is a device that operates based on a program.

  The ROM 10H2 is a storage device that stores data including programs, parameters, and the like.

  The RAM 10H3 is a main storage device that stores data including programs and parameters. The RAM 10H3 is a device that provides a work area when the CPU 10H1 performs processing.

  The keyboard 10H4 and the mouse 10H5 are examples of input devices. That is, the keyboard 10 </ b> H <b> 4 and the mouse 10 </ b> H <b> 5 serve as interfaces for the user to input operations such as commands and clicks to the image processing apparatus 10.

  The hard disk 10H6 is an example of an auxiliary storage device. As illustrated, the hard disk 10H6 stores data such as an OS (Operating System), device drivers, and various application programs.

  The graphic board 10H7 is an output device that displays images, processing results by the CPU 10H1, and the like. An external device such as a display is connected to the graphic board 10H7, the graphic board 10H7 outputs an image or the like to the external device, and the external device displays the image or the like to the user.

  The network card 10H8 is an interface for transmitting and receiving data to and from an external device via a network by wire or wireless. For example, the network card 10H8 is a connector and a processing IC (Integrated Circuit).

  The I / F (interface) 10H9 is an interface for connecting an external device via a cable or the like. For example, an I / F (interface) 10H9 is a connector and a processing IC.

  The program reading device 10H10 is a device that reads a program stored in the recording medium 11 or the like. For example, when the recording medium 11 is an optical disk, the program reading device 10H10 is an optical disk drive or the like.

  As described above, the image processing apparatus 10 is an information processing apparatus such as a PC, a server, a workstation, or a mobile device that operates based on a program or the like. That is, the image processing apparatus 10 is a computer.

  Note that the image processing apparatus 10 is not limited to the illustrated configuration. For example, the image processing apparatus 10 may further include an auxiliary device that assists the CPU and the like.

<Hardware configuration example of the controller>
FIG. 4 is a schematic diagram illustrating a hardware configuration example of the operation device according to the embodiment of the present invention. For example, as shown in the drawing, the electronic pen PE desirably has a light emitting unit PEH1 and a support unit PEH2. Hereinafter, the configuration shown in the figure will be described as an example.

  The light emitting part PEH1 is preferably a part where ink oozes out in the writing instrument, that is, a position of a pen tip or the like. Further, as shown in the drawing, the light emitting portion PEH1 is desirably a location close to a location that contacts the surface of the display DIS. The light emitting unit PEH1 is realized by, for example, an LED (Light Emitting Diode) or an organic EL (Electro-Luminescence). In addition, the light emitting unit PEH1 emits visible light and infrared light that are sensed by the image pickup device of the camera.

  The support part PEH2 is a part that the user has with his hand when using the electronic pen PE. That is, the user holds the support part PEH2 and performs an operation with the electronic pen PE.

  Moreover, a predetermined location is a position where light emission part PEH1 is installed, for example.

<Example of overall processing>
FIG. 5 is a flowchart showing an example of overall processing by the image processing apparatus according to the embodiment of the present invention.

<Imaging Example> (Step S101)
In step S101, the image processing apparatus causes each camera to capture an image of the operation device. And the camera produces | generates the image data which imaged the operation device, and inputs it into an image processing apparatus.

<Example of Generating Observation Point Based on Captured Image> (Step S102)
In step S102, the image processing apparatus generates an observation point based on the image indicated by the input image data. For example, the observation point is generated as follows.

  FIG. 6 is a diagram illustrating an example of generation of observation points according to an embodiment of the present invention.

  FIG. 9 is a diagram illustrating an example of image data according to an embodiment of the present invention. As shown in the drawing, if an arbitrary point on the surface of the display DIS is indicated by the pen tip of the electronic pen, for example, blobBL indicating the pen tip of the electronic pen is captured in the image data IMG captured by the upper left camera CAM11. The point indicated by the pen tip is an example of the observation point TP. Moreover, as shown in the figure, the observation point TP and blobBL have a projection relationship. Therefore, when the position of blobBL, that is, blobBL is tracked, the image processing apparatus can calculate the observation point TP from the back projection.

  Note that blob (Binary Large Object) is a specific gray scale in an image as described in, for example, “http://www.eizojoho.co.jp/i/i_pdf/0004.pdf”. Refers to a “chunk” of pixels having a value or range of values.

<Prediction point generation example> (step S103)
Returning to FIG. 5, in step S <b> 103, the image processing device generates a current prediction point. The predicted point is generated based on the observation point generated before the observation point indicating the current position. That is, the prediction point is generated by analyzing the movement of the electronic pen. The predicted point is calculated based on, for example, speed or acceleration. Further, the prediction point may be predicted by a Kalman filter or the like.

  FIG. 7 is a diagram illustrating an example of observation points and prediction points according to an embodiment of the present invention. For example, when connecting each point including the previous observation point and the previous prediction point, as shown in the figure, the line connecting the points becomes a writing locus with the electronic pen. In the following description, the current position, that is, the latest time point is indicated as time t. Further, each point generated one time before time t is shown to have been generated at time t-1, and similarly, each point two times before is shown to have been generated at time t-2. Specifically, the observation point generated at time t is the first observation point PO1.

  And the prediction point produced | generated at the time t is 1st prediction point PC1. The first predicted point PC1 is a result of predicting the next position, that is, the position at time t + 1. In addition, the observation point generated immediately before the first observation point PO1, that is, at the time t−1 is PO2. Similarly, the prediction point generated immediately before the first prediction point PC1, that is, at time t−1 is PC2.

<Setting Example of ID (Identification) and Tag Same as Current Observation Point for Current Prediction Point> (Step S104)
Returning to FIG. 5, in step S104, the image processing apparatus sets the same ID and tag as the current observation point as the current prediction point. First, ID which can recognize each electronic pen is set to a plurality of electronic pens.

  The tag is in a state set in advance by a previous determination process or the like. The initial value is, for example, a value indicating “no merge”. The ID is data that can identify each electronic pen. The tag is data indicating the state of the electronic pen determined by the determination process.

<Example of nodeization> (Step S105)
In step S105, the image processing apparatus performs nodalization. First, assume that a plurality of observation points are generated at time t as follows.

  FIG. 8 is a diagram illustrating an example when there are a plurality of observation points according to an embodiment of the present invention. When an operation is performed using a plurality of electronic pens, for example, a third observation point PO3 may be generated at time t separately from the first observation point PO1 as shown in FIG. In such a case, first, the following node processing is performed.

  FIG. 9 is a flowchart showing an example of node processing by the image processing system according to the embodiment of the present invention.

<Judgment example of whether or not there is a current observation point within a predetermined distance> (step S201)
In step S <b> 201, the image processing apparatus determines the currently generated prediction point, that is, the first prediction point. Specifically, in step S201, the image processing apparatus determines whether there is a current observation point, that is, an observation point generated at time t, within a predetermined distance from the first prediction point. For example, as shown in FIG. 8, it is determined whether or not there is a first observation point PO1 within a predetermined distance DST from the first prediction point PC1. The predetermined process DST is a preset distance.

  If it is determined that the first observation point is within the predetermined distance (YES in step S201), the image processing apparatus proceeds to step S202. On the other hand, when it is determined that there is no first observation point within the predetermined distance (NO in step S201), the image processing apparatus proceeds to step S203.

<Example of Linking Observation Point Closest to Prediction Point> (Step S202)
In step S202, the image processing apparatus links the observation point closest to the prediction point. For example, the link is the following process.

  FIG. 10 is a diagram illustrating an example of a link by the image processing apparatus according to the embodiment of the present invention. For example, as shown in the drawing, assuming that the observation point and the prediction point are nodes, when the nodes are linked, a line LK connecting the nodes is generated. In the example shown in the figure, the first observation point PO1 and the first prediction point PC1 are linked in a one-to-one relationship.

<Judgment example of whether or not there is a current observation point with no link> (step S203)
Returning to FIG. 9, in step S203, the image processing apparatus determines whether there is a current observation point for which no link is set. In the subsequent step S204, if there is a predicted point within a predetermined distance from each current observation point, the observation point and the predicted point are linked to generate a line LK connecting the nodes as shown in FIG. In this way, the image processing apparatus determines whether or not it has been determined whether or not all the current observation points can be linked to the prediction points.

  If the image processing apparatus determines that there is a current observation point with no link set (YES in step S203), the image processing apparatus proceeds to step S204. On the other hand, when the image processing apparatus determines that there is a current observation point with no link set (NO in step S203), the image processing apparatus ends the process.

<Example of linking the nearest predicted point within a predetermined distance to the current observation point with no link> (step S204)
In step S204, the image processing apparatus links the latest predicted point that is within a predetermined distance to the current observation point for which no link is set. For example, the image processing apparatus first determines whether or not there is a current predicted point within a predetermined distance from the current observation point. Then, when there is a current prediction point within a predetermined distance, the image processing apparatus links the observation point and the prediction point in the same manner as in FIG.

<Example of determination process> (step S106)
Returning to FIG. 5, in step S106, the image processing apparatus performs determination processing. The determination process determines the following states of each observation point and each prediction point that are nodeized.

  FIG. 11 is a diagram illustrating an example of determination processing by the image processing apparatus according to an embodiment of the present invention. For example, as shown in the figure, each node is determined for each observation point or a unit of a prediction point linked to the observation point (hereinafter referred to as “determination unit”). That is, when a plurality of nodes are linked, the image processing apparatus divides observation points and prediction points so as to be a determination unit. Next, the image processing apparatus determines the state based on the number of nodes as follows.

  “State 1” is a state in which there is only one observation point. Such a state is an observation point newly added from time t. That is, the observation point determined as “state 1” is a point at which writing with the electronic pen is started. Next, when it is determined as “state 1”, after the determination process, the image processing apparatus sets a new ID at the observation point. In addition, the image processing apparatus sets a tag “no merge” at the observation point. Then, for the next time t + 1, the processing shown in FIG. 5 is started.

  “State 2” is a state in which the observation point and the prediction point are linked in a one-to-one relationship. Such a state is a state in which tracking is successful. Next, when it is determined as “state 2”, after the determination process, the image processing apparatus sets the same ID as the prediction point at the observation point. In addition, the image processing apparatus sets a tag “no merge” at the observation point.

  In the state where the tracking is successful as in “State 2”, if the observation points from before to this time are connected, the trace of drawing with the electronic pen is displayed with high accuracy. Therefore, the image processing apparatus displays a line connecting the previous observation points from time t on the display DIS (FIG. 1) or the like. Note that the observation points to be connected are observation points to which the same ID is set. When only one point is tracked, the same processing as “state 2” is performed.

  “State 3” is a state in which a plurality of prediction points are linked to one observation point. For example, as shown in the drawing, two prediction points may be linked to one observation point. Such a state is a “merged state”.

  Thus, the “merged state” is a state in which the positions of the pointers (pens) in the image formed by sensor light reception overlap (a state where separation is difficult). Therefore, the image processing apparatus stores the “merged state” as a tag. In addition, two IDs of “prediction point B1” and “prediction point B2” shown in the figure are set for the observation point. Thus, when two or more IDs are set for the observation point, the image processing apparatus can recognize that it is in the “merged state”. Further, in the “merged state”, since the data related to the observation point at time t−1 is used to separate the “merged state” at the next time t + 1, the image processing apparatus performs processing at the next time t + 1. Therefore, the data related to the observation point at time t−1 is stored.

  Since the state is “no merge” until time t, the tracking is successful until time t as in “state 2”. Therefore, until time t, the image processing apparatus displays a line connecting the previous observation points from time t on the display DIS (FIG. 1) or the like as in “state 2”.

  “State 4” is a state in which one prediction point is linked to a plurality of observation points. Such a state is, for example, a state in which two observation points are linked to one prediction point as shown in the figure. Therefore, such a state is a state in which one prediction point has two IDs. That is, the “state 4” is a “separation state” in which the separation process is performed on the observation point or the prediction point in the “merged state” and the process of changing the “merged state” to the “no merge” state. Therefore, the image processing apparatus distributes the two IDs to the two observation points.

  The method for assigning IDs is related to, for example, the data related to the observation point before the “merged state” stored in advance, that is, the observation point at time t−1, when it is previously determined as “state 3” or the like. This is done using data. Specifically, the processing is as follows.

  FIG. 12 is a diagram showing an example of an ID distribution method by the image processing apparatus according to the embodiment of the present invention. For example, the state shown in FIG. 12A (“no merge” state) changes to the state shown in FIG. 12B (“merged state”), and the ID shown in FIG. An example in which the sorting process is performed will be described. As shown in the figure, in this example, the IDs of the two observation points are “ID = 3” and “ID = 4”. First, when the state shown in FIG. 12B is determined, at time t−1, data related to an observation point “no merge” such as time t-2 is stored.

  Then, at the time t at which the ID is distributed, the distribution is performed using the data related to the observation point at the time t-2 of each observation point. Specifically, as shown in FIG. 12C, the ID closer to the observation point at time t-2 is assigned to each observation point.

  In addition, as a method of assigning IDs, for example, a method described in US Pat. No. 8,869,768 may be used.

  Returning to FIG. 11, "state 5" is a state in which a plurality of prediction points and a plurality of observation points are linked, as shown. In such a state, a prediction point and an observation point are linked as follows, for example.

  FIG. 13 is a diagram illustrating an example of a link in “state 5” by the image processing apparatus according to the embodiment of the present invention. For example, the case of “state 5” as shown in FIG. 13A will be described as an example. Then, the image processing apparatus is linked to a state as shown in FIG.

  For example, the image processing apparatus separates the observation point and the prediction point so that the sum of squares of the distances of the linked prediction point and the observation point is minimum and does not overlap. As shown in FIG. 13B, when separation is performed, each linked prediction point and observation point are in the same state as “state 2”. Therefore, for each linked prediction point and observation point, the image processing apparatus displays a line connecting the previous observation points from time t on the display DIS (FIG. 1), etc., as in the case of “state 2”. To do.

  As described above, when a plurality of electronic pens are used, the state is determined by the determination process, and when it is determined whether the state is the “merge state” or not (the state 3 is determined). Then, the image processing apparatus can separate the observation points with high accuracy even in the “merged state” by distributing IDs or the like. For example, the image processing apparatus can reduce the following erroneous recognition.

  FIG. 14 is a diagram illustrating an example of erroneous recognition. For example, it is assumed that writing is performed using a plurality of electronic pens as shown in FIG. In such a case, as shown in FIG. 14B, when the “merged state” is entered, the locus of one electronic pen and the locus of the other electronic pen are erroneously recognized, and FIG. As shown in the figure, a trajectory different from “actual writing” shown in FIG. 14A may be displayed on the display DIS. This is often caused by the following “merging”.

  FIG. 15 is a diagram illustrating an example of the “merged state”. For example, a case where a plurality of electronic pens are used and two observation points are recognized by the upper left camera CAM11 will be described as an example. First, in the case of “no merging”, that is, in the above “state 2”, as shown in FIG. 15A, two observation points are recognized separately as different IDs. On the other hand, in the “merged state”, as shown in FIG. 15B, even when the observation points are input with different electronic pens, the two observation points are recognized as having the same ID. It is. When such a “merged state” occurs, it may cause a recognition error as shown in FIG. On the other hand, if there are few misrecognitions by a determination process, the image processing apparatus can display as follows.

  FIG. 16 is a diagram illustrating an example of the effect of the image processing apparatus according to the embodiment of the present invention. For example, the trajectory by the observation point of “ID = 3” and the observation point of “ID = 4” can be continuously displayed as shown in the figure. That is, the trajectory can be displayed without mixing the observation point “ID = 3” and the observation point “ID = 4”.

<Modification>
A so-called light blocking method may be performed using an electronic pen. For example, the light blocking method has the following configuration.

  FIG. 17 is a diagram illustrating an example of a light blocking method according to an embodiment of the present invention. As shown in the figure, when the electronic pen PE is used, when a document, a user's hand or clothing touches the display at the time of input, the contact position may be recognized as an instruction position by the electronic pen PE. is there. Therefore, an electronic pen having a light emitting portion as shown in FIG. 4 is used. When an electronic pen having such a light emitting portion is used, the coordinates of the light emitting portion can be detected. However, there are cases where an operation with a finger such as a gesture cannot be detected only by detecting the light emitting portion. Therefore, for example, a method disclosed in JP-A-2006-038902 is used.

  When such a method is used, coordinates can be detected as shown in the figure by combining coordinate detection based on light emission of the indicator and coordinate detection based on light blocking. Such a light blocking method may be used.

<Functional configuration example>
FIG. 18 is a functional block diagram illustrating a functional configuration example of an image processing system and an image processing apparatus according to an embodiment of the present invention. For example, the image processing system 1 includes a plurality of imaging devices CAM, a display device OUT, an observation point generation unit 1F1, a prediction point generation unit 1F2, a determination unit 1F3, and an output unit 1F4. Note that the illustrated example is an example in which there are four imaging devices CAM.

  The observation point generation unit 1F1 generates an observation point indicating the current position of the operation unit IF based on image data input from the imaging device CAM. For example, the observation point generator 1F1 is realized by the CPU 10H1 (FIG. 3) or the like.

  The predicted point generation unit 1F2 generates a predicted point that is the next position of the current position based on the motion of the operation unit IF. For example, the predicted point generation unit 1F2 is realized by the CPU 10H1 (FIG. 3) or the like.

  The determination unit 1F3 determines the state of the observation point and the prediction point. For example, the determination unit 1F3 is realized by the CPU 10H1 (FIG. 3) or the like.

  The output unit 1F4 displays a line connecting the observation points on the display device OUT based on the state determined by the determination unit 1F3. For example, the output unit 1F4 is realized by the CPU 10H1 (FIG. 3), the graphic board 10H7 (FIG. 3), and the like.

  First, the image processing apparatus included in the image processing system 1 inputs image data obtained by imaging the operation unit IF from the imaging apparatus CAM. Next, the image processing apparatus generates an observation point and a prediction point by the observation point generation unit 1F1 and the prediction point generation unit 1F2, respectively.

  Then, the image processing apparatus determines the state of the observation point and the prediction point as shown in “state 1” to “state 5” by the determination unit 1F3 as illustrated in FIG. When such a determination is made, for example, a “merged state” such as “state 3” can be determined.

  The “merged state” is a state in which a plurality of prediction points are linked to the observation point, for example, “state 3”. Further, the “merged state” is a state in which a plurality of observation points are linked to the prediction point, for example, “state 3”. As described above, after the data indicating the “merged state” and the data relating to the previous observation point are stored after the determination of the “merged state”, the image processing apparatus observes the “merged state”. For example, IDs can be assigned to points.

  In this way, erroneous recognition as shown in FIG. 14 is reduced, and the image processing apparatus can improve the coordinate detection accuracy.

<Other embodiments>
The embodiment according to the present invention may be realized by a program for causing a computer described in a low-level language such as an assembler language or C language or a high-level language such as an object-oriented programming language to execute an image processing method. Good. The program can be stored and distributed in a computer-readable recording medium such as an auxiliary storage device, a flash memory, or an optical disc such as a Blu-ray disc. Furthermore, the program can be distributed through a telecommunication line.

  The image processing apparatus may not be a single information processing apparatus. That is, the embodiment according to the present invention may be realized by an image processing system having one or more information processing apparatuses. In other words, each process may be executed redundantly, in a distributed manner or in parallel by a plurality of information processing devices connected via a network or the like.

  Moreover, a part or all of each process may be implement | achieved by electronic circuits, such as a field programmable gate array (FPGA) or ASIC (Application Specific Integrated Circuit). Furthermore, a part or all of each process may be realized by, for example, firmware.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications can be made within the scope of the gist of the present invention described in the claims. Can be changed.

DESCRIPTION OF SYMBOLS 1 Image processing system 10 Image processing apparatus CAM11 Upper left camera CAM12 Upper right camera CAM13 Lower right camera CAM14 Lower left camera DIS Display PE Electronic pen

US Pat. No. 8,692,768

Claims (6)

An image processing device connected to a display device and an imaging device that images a plurality of operation devices,
Based on image data input from the imaging device, an observation point generator that generates an observation point indicating the current position of the controller,
A prediction point generation unit that generates a prediction point that is the next position of the current position based on the motion of the operation unit;
A determination unit for determining a state of the observation point and the prediction point;
An image processing apparatus comprising: an output unit configured to display a line connecting the observation points on the display device based on the state.   The image processing apparatus according to claim 1, wherein the determination unit determines whether at least the observation point and the prediction point are in a merged state.   The image processing apparatus according to claim 2, wherein the determination unit determines that the merged state is established when a plurality of the prediction points are linked to the observation point.   The plurality of prediction points and the plurality of observation points are separated from each other when the determination unit determines that the plurality of prediction points and the plurality of observation points are linked. Image processing apparatus. An image processing system having an imaging device that images a plurality of operating devices, a display device, and the image processing device and an image processing device connected to the display device,
Based on image data input from the imaging device, an observation point generator that generates an observation point indicating the current position of the controller,
A prediction point generation unit that generates a prediction point that is the next position of the current position based on the motion of the operation unit;
A determination unit for determining a state of the observation point and the prediction point;
An image processing system comprising: an output unit configured to display a line connecting the observation points on the display device based on the state. An image processing method performed by an imaging device that images a plurality of operation devices and an image processing device connected to a display device,
Based on image data input from the imaging device, an observation point generation procedure for generating an observation point indicating the current position of the controller,
A prediction point generation procedure in which the image processing device generates a prediction point that is the next position of the current position based on the motion of the operation unit;
A determination procedure in which the image processing apparatus determines the state of the observation point and the prediction point;
An image processing method including: an output procedure for causing the display device to display a line connecting the observation points based on the state.

Images