JP2014110029A - Pointing system, pointing method, display device and electronic apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve user's convenience at the time of pointing.SOLUTION: A pointing system including a display device for displaying video and an electronic apparatus provided with an imaging part for imaging the display device displaying video includes: a motion sensor part for measuring a measurement value that represents a posture of the electronic apparatus; a posture calculation part for calculating a point at which an optical axis of the imaging part crosses a plane including a display area of the display device as a pointing position on the basis of the measurement value measured by the motion sensor part; a determination part for determining whether the pointing position is located within the display area; and a correction part for correcting the pointing position in the case where the determination part determines that the pointing position is located within the display area.

Description

  The present invention relates to a pointing system, a pointing method, a display device, and an electronic device.

  Pointing refers to pointing to a figure or the like. For example, there is a pointing device such as a laser pointer using a laser beam. The technique described in Patent Document 1 includes at least four accelerometer devices provided in an arrangement in which user interface devices are not on the same plane, and each of the at least four accelerometer devices outputs respective accelerometer data. To do. The user interface device further comprises an algorithm for converting the accelerometer data from each of the at least four accelerometer devices into a two-dimensional pointer movement.

Special table 2007-509448

  However, the accelerometer that measures the attitude of the apparatus even using the technique described in Patent Document 1 has a drawback in that errors are accumulated and the pointing position cannot be estimated accurately. Further, in the pointing method by photographing the video displayed on the display device with the camera, the camera is separated from the imaging range of the camera, that is, the display device that displays the video and the camera that captures the video are at a long distance. In addition, there is a drawback that it is impossible to estimate the position where the pointer is pointed. As described above, there are cases where the pointing position cannot be estimated, and there is a problem that the convenience for the user is not sufficient.

  The present invention has been made in view of the above points, and an object thereof is to provide a pointing system, a pointing method, a display device, and an electronic device that can improve user convenience during pointing.

  (1) The present invention has been made to solve the above problems, and one aspect of the present invention includes a display device that displays an image and an imaging unit that captures the display device that displays the image. A pointing system comprising: an electronic device comprising: a motion sensor that measures a measurement value representing an attitude of the electronic device; and an optical axis of the imaging unit based on the measurement value measured by the motion sensor unit Is an attitude calculation unit that calculates a point that intersects a plane including a display area of the display device as a pointing position, a determination unit that determines whether the pointing position is within the display area, and the determination unit that performs the display And a correction unit that corrects the pointing position when it is determined to be within the region.

  (2) One embodiment of the present invention is the pointing system described above, in which the correction unit is configured to compare the pointing position based on a comparison between the image obtained by the imaging unit and the video. The pointing system according to claim 1, wherein the pointing system is corrected.

  (3) One embodiment of the present invention is the pointing system described above, in which the coordinates of the video are determined from the image coordinate system based on the video and the image obtained by the imaging unit. A conversion rule generating unit that generates a conversion rule for a system, and a position of a pointer that causes the display unit to display a point at which the optical axis of the imaging unit intersects a display unit included in the display device with reference to the conversion rule And an estimation unit that estimates the information as information for determining the pointing system.

  (4) One aspect of the present invention is the pointing system described above, in which the conversion rule generation unit detects the corner of the display unit from the image acquired by the imaging unit; An imaging surface corner coordinate calculation unit that estimates coordinates of the corner of the display unit in the image detected by the corner detection unit, and a center of the image based on the coordinates of the corner estimated by the imaging surface corner coordinate calculation unit The pointing system further comprising: a projective conversion unit that performs projective conversion of the coordinates to the center coordinates of the display unit.

  (5) One embodiment of the present invention is the pointing system described above, further including a control unit that controls the display unit to point the pointer, and the motion sensor unit is an acceleration that measures acceleration. A sensor and a geomagnetic sensor that measures the direction of the earth's magnetic field, and the control unit displays the display unit based on the acceleration measured by the acceleration sensor and the direction of the magnetic field measured by the geomagnetic sensor. Outside the area, the movement of the pointer is estimated, and when returning to the display area of the display unit, the estimated movement of the pointer and the pointer estimated from the image captured by the imaging unit The pointing system is characterized in that the pointer is pointed to the display unit based on the position.

  (6) One embodiment of the present invention is the pointing system described above, wherein the motion sensor unit is a gyro sensor that measures acceleration and angular velocity, and the control unit is configured to detect acceleration measured by the gyro sensor. The movement of the pointer is estimated outside the display area of the display unit from the angular velocity, and the estimated movement of the pointer and the imaging unit are imaged when returning to the display area of the display unit. The pointing system is characterized in that the pointer is pointed to the display unit based on the position of the pointer estimated from an image.

  (7) One embodiment of the present invention is the pointing system described above, further including a smoothing unit that smoothes the movement of the pointer, and the control unit smoothes the smoothing unit according to the acceleration. It is a pointing system characterized by controlling whether or not to make it.

  (8) One embodiment of the present invention is a pointing method in a pointing system including a display device that displays an image and an electronic device that includes an imaging unit that images the display device that displays the image. A first step of measuring a measurement value representing the attitude of the electronic device, and a plane in which an optical axis of the imaging unit includes a display region of the display device based on the measurement value measured in the first step A second step of calculating a point that intersects with a pointing position, a third step of determining whether or not the pointing position is within the display region, and being within the display region in the third step And a fourth step of correcting the pointing position when the determination is made.

  (9) One embodiment of the present invention is a display device in a pointing system including a display device that displays an image and an electronic device that includes an imaging unit that images the display device that displays the image. An attitude calculation unit that calculates, as a pointing position, a point where an optical axis of the imaging unit intersects a plane including a display area of the display device based on the measurement value measured by the motion sensor unit of the electronic device; and the pointing A determination unit that determines whether or not a position is within the display area; and a correction unit that corrects the pointing position when the determination unit determines that the position is within the display area. It is a display device.

  (10) One embodiment of the present invention is an electronic device in a pointing system including a display device that displays an image and an electronic device that includes an imaging unit that images the display device that displays the image. A pointing unit that estimates a corner of the display unit included in the display device from an image captured by the imaging unit, calculates a position at which the pointer is pointed to the display unit from the estimated corner, and an attitude of the electronic device Based on the measured value measured by the motion sensor unit that measures the measured value that represents the point, the point where the optical axis of the imaging unit intersects the plane including the display area of the display device is calculated as the pointing position A posture calculation unit that performs determination, a determination unit that determines whether or not the pointing position is within the display area, and the determination unit includes: If it is determined that 示領 a region, which is an electronic device characterized by comprising a correction unit for correcting the pointing position.

  According to the present invention, it is possible to improve user convenience during pointing.

It is the schematic which shows an example of a structure of the pointing system which concerns on the 1st Embodiment of this invention. It is a schematic block diagram which shows an example of a structure of the electronic device which concerns on this embodiment. It is a schematic block diagram which shows an example of a structure of the display apparatus which concerns on this embodiment. It is explanatory drawing which shows an example of the camera image input in the corner detection part of the pointing part of the display apparatus which concerns on this embodiment. It is explanatory drawing which shows an example of the contrast enhancement process in the corner detection part of the pointing part of the display apparatus which concerns on this embodiment. It is explanatory drawing which shows an example of the binarization process in the corner detection part of the pointing part of the display apparatus which concerns on this embodiment. It is explanatory drawing explaining an example of the projection conversion process in the projection conversion part of the pointing part of the display apparatus which concerns on this embodiment concerning this embodiment. It is explanatory drawing explaining an example of the projection conversion process in the projection conversion part of the pointing part of the display apparatus which concerns on this embodiment concerning this embodiment. It is explanatory drawing which shows an example of the attitude | position calculation means of the electronic device in the attitude | position calculation part of the control part of the display apparatus which concerns on this embodiment. It is explanatory drawing explaining the switching process of the pointing coordinate based on the camera image in the switching determination part of the control part of the display apparatus which concerns on this embodiment, and the pointing coordinate based on an estimation part. It is a schematic block diagram which shows an example of a structure of the display apparatus which concerns on the 2nd Embodiment of this invention. It is explanatory drawing explaining an example of operation | movement of the screen division part of the display apparatus which concerns on this embodiment. It is explanatory drawing explaining an example of operation | movement of the screen division part of the display apparatus which concerns on this embodiment. It is a schematic block diagram which shows an example of a structure of the electronic device which concerns on this embodiment. It is explanatory drawing explaining an example of the matching process in the feature point comparison part of the pointing part of the electronic device which concerns on this embodiment. It is explanatory drawing explaining an example of the corner estimation means in the corner estimation part of the pointing part of the electronic device which concerns on this embodiment.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a schematic diagram illustrating an example of a configuration of a pointing system S1 according to the present embodiment.
The pointing system S <b> 1 includes a display device 1 and an electronic device 2. The display device 1 includes a display unit 101 such as a liquid crystal display, an organic EL display, or a plasma display. The electronic device 2 captures a display image displayed on the display unit 101 using a camera (imaging unit) 201. The pointing system S1 estimates a corner (corner) of the display unit in a captured image (camera image) obtained by the camera 201. The pointing system S1 generates a transformation matrix from the estimated corner of the camera image, and estimates a point where the optical axis of the camera 201 in the electronic device 2 intersects the display unit 101 in the display device 1 as a pointing position from the generated transformation matrix. Here, the conversion matrix is a matrix for converting a camera image coordinate system, which will be described later, to a display video coordinate system.

FIG. 2 is a schematic block diagram illustrating an example of the configuration of the electronic device 2 according to the present embodiment.
The electronic device 2 includes a camera 201, a communication unit 203, and a motion sensor unit 211. The electronic device 2 has other generally known electronic device functions, but illustration and description thereof are omitted.
The camera 201 is, for example, an infrared camera (IR camera). The camera 201 captures a display video displayed on the display unit 101 of the display device 1 and transmits a camera image obtained by imaging to the display device 1 via the communication unit 203.

  The motion sensor unit 211 measures a measurement value representing the attitude of the electronic device 2. Here, the motion sensor unit 211 includes an acceleration sensor 2111 and a geomagnetic sensor 2112.

The acceleration sensor 2111 measures the acceleration of the electronic device 2. The acceleration sensor 2111 measures the acceleration based on, for example, a change in the position of a weight (mass, mass) due to acceleration. The acceleration sensor 2111 transmits the measured acceleration value to the display device 1 via the communication unit 203.
The geomagnetic sensor 2112 measures the tilt of the electronic device 2. The geomagnetic sensor 2112 detects the direction of the earth's magnetic field. The geomagnetic sensor 2112 measures the inclination of the electronic device 2 from the detected direction of the magnetic field. The geomagnetic sensor 2112 transmits the measured inclination of the electronic device 2 to the display device 1 via the communication unit 203.

  The communication unit 203 performs various communication processes. The communication unit 203 performs communication processing such as data encoding / decoding, modulation / demodulation, AD (analog to digital) conversion / DA (digital to analog) conversion, frequency conversion, and transmission / reception of data signals by radio waves. The communication unit 203 communicates with the display device 1 using various communication methods such as Bluetooth (registered trademark) and wireless LAN (Local Area Network).

FIG. 3 is a schematic block diagram illustrating an example of the configuration of the display device 1 according to the present embodiment.
The display device 1 includes a display unit 101, a pointing unit 11, and a control unit 12. The pointing unit 11 includes a communication unit 110, a corner detection unit 111, an imaging surface corner coordinate calculation unit 112, and a projection conversion unit 113. The control unit 12 includes an attitude calculation unit 121, an estimation unit 122, a switching determination unit 123, a weighting unit 124, and an output instruction unit 125. Although the display device 1 has other functions of a known display device, illustration and description thereof are omitted.

The display unit 101 is, for example, a liquid crystal display, an organic EL display, a plasma display, or the like. The display unit 101 displays a video (display video).
When a camera image obtained by the camera 201 of the electronic device 2 is input via the communication unit 110, the pointing unit 11 estimates a corner of the display unit 101 in the camera image, and displays the display unit from the estimated corner. The position at which the pointer is displayed on 101 is calculated as the pointing position.

  The communication unit 110 performs various communication processes. The communication unit 110 performs communication processing such as data encoding / decoding, modulation / demodulation, AD (analog to digital) conversion / DA (digital to analog) conversion, frequency conversion, and transmission / reception of data signals by radio waves. The communication unit 110 communicates with the electronic device 2 using various communication methods such as Bluetooth (registered trademark) and wireless LAN (Local Area Network).

When a camera image is input from the camera 201 of the electronic device 2 via the communication unit 110, the corner detection unit 111 detects a corner of the display unit 101 or a corner of the display device in the camera image.
FIG. 4 is an explanatory diagram illustrating an example of the camera image Ga1 input in the corner detection unit 111 of the pointing unit 11 of the display device 1 according to the present embodiment. FIG. 5 is an explanatory diagram illustrating an example of contrast adjustment processing in the corner detection unit 111 of the pointing unit 11 of the display device 1 according to the present embodiment.

  Specifically, the corner detection unit 111 performs at least contrast enhancement processing on a camera image obtained by imaging with the camera 201 as illustrated in FIG. 4 to generate an image Ga2 with enhanced contrast as illustrated in FIG. To do. The contrast enhancement is performed by using an algorithm such as histogram equalization or contrast stretch on the camera image, for example. In the present embodiment, a contrast stretch suitable for binarization described later is used.

The corner detection unit 111 performs binarization processing after the contrast enhancement processing is completed.
FIG. 6 is an explanatory diagram illustrating an example of binarization processing in the corner detection unit 111 of the pointing unit 11 of the display device 1 according to the present embodiment.
The corner detection unit 111 performs binarization processing on the image Ga2 with enhanced contrast shown in FIG. 5 that has been subjected to contrast enhancement processing, thereby obtaining a binarized image Ga3 as shown in FIG. Generate. Here, examples of the algorithm used as the binarization process include adaptive binarization and automatic binarization. The corner detection unit 111 according to the present embodiment uses the automatic binarization that easily separates the background of the display device 1 and the display unit 101 as shown in FIG. ) Is generated.

  Returning to FIG. 3, the corner detection unit 111 performs polygon approximation on the bright area separated from the background in the binarized image Ga3, and estimates a straight line that forms the edge of the bright area. Furthermore, the corner detection unit 111 selects four sides that are most likely to represent the shape of the display unit 101 from the estimated straight line. The corner detection unit 111 detects corners Co1, Co2, Co3, and Co4 based on the four sides selected in the binarized image Ga3. The corner detection unit 111 outputs information on the detected corners Co1, Co2, Co3, and Co4 to the imaging surface corner coordinate calculation unit 112.

  The imaging surface corner coordinate calculation unit 112 calculates the corner coordinates of the display unit 101 in the camera image of the camera 201 indicated by the corner information input from the corner detection unit 111. That is, the imaging surface corner coordinate calculation unit 112 calculates the corner coordinates of the display unit 101 in the camera image based on the coordinates of the pixels of the camera image in which the detected corners Co1, Co2, Co3, and Co4 are present. The imaging surface corner coordinate calculation unit 112 estimates a homography matrix (mat) using the corner coordinates. Here, the homography matrix is a matrix obtained by examining the plane in the space and the projection of the image (plane), and given a correspondence between two-dimensional points, it is generally 3 rows by 3 columns. It is a matrix.

  The imaging surface corner coordinate calculation unit 112 calculates the corner (corner) of the display unit 101 of the display device 1 in the camera image using the estimated homography matrix (mat). The imaging plane corner coordinate calculation unit 112 calculates the coordinates of the corners of the camera image coordinate system by performing projective transformation of the coordinates of the corners of the display video coordinate system using the estimated homography matrix. Here, the display video coordinate system is a coordinate system on the display video plane, and the camera image coordinate system is a coordinate system on the camera image plane. The imaging surface corner coordinate calculation unit 112 outputs the corner coordinates of the display unit 101 in the calculated camera image to the projection conversion unit 113.

  The projective transformation unit 113 displays the coordinates of the center of the camera image coordinate system based on the perspective transformation matrix (PMat) and the corner coordinates of the camera image coordinate system input from the imaging surface corner coordinate calculation unit 112. The coordinates obtained by projective transformation to the center of the are calculated as pointing coordinates. More specifically, based on the calculated corner coordinates in the camera image coordinate system and the coordinates of the corners of the display unit 101 in the display video coordinate system, the coordinates in the display video coordinate system are changed from the coordinates in the camera image coordinate system. A perspective transformation matrix (PMat) for transformation into coordinates is calculated.

FIG. 7 is an explanatory diagram for explaining an example of the projection conversion process in the projection conversion unit 113 of the pointing unit 11 of the display device 1 according to the present embodiment.
The imaging plane corner coordinate calculation unit 112 calculates a perspective transformation matrix PMat from the corner coordinates M of the display video coordinate system and the calculated corner coordinates m of the camera image coordinate system by Expression (1).

  Here, it is an example when M is (tx, ty, t). Moreover, it is an example when m is (x, y, 1).

  Next, the projective transformation unit 113 calculates the coordinates C of the display video coordinate system corresponding to the center coordinates c of the camera image coordinate system, using the perspective transformation matrix PMat obtained as shown in FIG. The calculated coordinate C is a point where the user intersects the optical axis of the camera 201 of the electronic device 2 and the display unit 101 of the display device 1, and the pointing position is determined based on the point.

FIG. 8 is an explanatory diagram illustrating an example of the projective transformation process in the projective transformation unit 113 of the pointing unit 11 of the electronic device 2 according to the present embodiment.
The projective transformation unit 113 estimates a point where the optical axis of the camera 201 and the display unit 101 of the display device 1 intersect. Specifically, the mapping unit 208 converts the center coordinate c of the camera image coordinate system to the coordinate C of the display video coordinate system based on the perspective conversion matrix PMat input from the conversion matrix generation unit 207. For example, assuming that the resolution of the camera 201 is 640 × 480 pixels, the center coordinates c of the camera image coordinate system are (320, 240, 1). In the matrix, the projective transformation unit 113 transforms the center coordinates c (320, 240, 1) of the camera image coordinate system into coordinates C (X, Y, Z) of the display video coordinate system using the perspective transformation matrix PMat. . The projective transformation unit 113 outputs the calculated coordinates C to the control unit 12.

Based on the measurement value measured by the motion sensor unit 211 of the electronic device 2, the control unit 12 moves the pointer to the display unit 101 when the pointer returns to the display region of the display unit 101 outside the display region of the display unit 101. The position where the point is pointed is calculated.
The attitude calculation unit 121 calculates the attitude of the electronic device 2 from the acceleration and angular velocity input from the motion sensor unit 211 of the electronic device 2 via the communication unit 110. With reference to FIG. 9, the posture calculation means of the electronic device 2 of the posture calculation unit 121 will be described in detail.

FIG. 9 is an explanatory diagram illustrating an example of a posture calculation unit of the electronic device 2 in the posture calculation unit 121 of the control unit 12 of the display device 1 according to the present embodiment.
The posture calculation unit 121 calculates the acceleration and angle in the three-axis (X, Y, Z) directions from the following equation (2) from the acceleration and angular velocity input from the motion sensor unit 211 of the electronic device 2 via the communication unit 110. Calculate using (6).

Here, a _x , a _y , and a _z represent accelerations measured by the acceleration sensor 2111 in the X, Y, and Z axis directions, respectively. Θ _yaw is calculated using equations (4), (5), and (6).

Here, M _x , M _y , and M _z represent output signals in the X, Y, and Z axis directions of the geomagnetic sensor 2112, respectively.
The posture calculation unit 121 outputs information (θ _pitch , θ _roll , θ _yaw ) representing the calculated angle of the electronic device 2 to the estimation unit 122.

Based on the information (θ _pitch , θ _roll , θ _yaw ) representing the posture of the electronic device 2 input from the posture calculation unit 121, the estimation unit 122, for example, the direction in which the camera 201 of the electronic device 2 is other than the display unit 101 The camera 201 of the electronic device 2 is facing the left-right direction of the display unit 101, the camera 201 of the electronic device 2 is facing the front direction of the display unit 101, and so on. . Specifically, the estimation unit 122 always estimates the movement of the electronic device 2 based on the output from the pointing unit 11 and the information representing the angle of the electronic device 2 input from the posture calculation unit 121. The estimation unit 122 includes a plane including the display area of the display unit 101 of the display device 1 based on information indicating the estimated movement of the electronic device 2, for example, information indicating the posture of the electronic device 2 (θ _pitch , θ _roll , θ _yaw ). The above pointing coordinates are output to the switching determination unit 123. The pointing coordinates will be described in detail with reference to FIG.

FIG. 10 is an explanatory diagram for explaining the switching process between the pointing coordinate based on the camera image and the pointing coordinate based on the estimation unit 122 in the switching determination unit 123 of the control unit 12 of the display device 1 according to the present embodiment.
In FIG. 10, when the pointer in the display area of the display unit 101 moves in the direction indicated by the dashed arrow, that is, when there are pointing coordinates that the pointer may indicate outside the display area of the display unit 101, The switching determination unit 123 switches from the pointing coordinates of the pointing unit 11 to the pointing coordinates of the estimation unit 122. At this time, the estimation unit 122 always estimates the pointing coordinates that may be indicated by the pointer from the information indicating the angle of the electronic device 2 from the posture calculation unit 121.

  When the pointer returns to the display area of the display unit 101 again, the switching determination unit 123 switches the pointing coordinates from that of the estimation unit 122 to that of the pointing unit, and the weighting unit 124 changes the weighting step by step. I will do it. Thereby, the estimation error of the pointing coordinate when the pointer returns to the display area of the display unit 101 can be reduced.

  Based on the information indicating the movement of the electronic device 2 input from the estimation unit 122 and the pointing coordinates input from the pointing unit 11, the switching determination unit 123 has a pointing coordinate in the display area of the display unit 101. Determine whether or not. When the pointing coordinates are not within the display area of the display unit 101, the switching determination unit 123 switches from the pointing coordinates input from the pointing unit 11 to the pointing coordinates estimated by the estimation unit 122.

  When the pointing coordinate is in the display area of the display unit 101 of the display device 1, the switching determination unit 123 switches from the pointing coordinate estimated by the estimation unit 122 to the pointing coordinate estimated by the pointing unit 11, and the weighting unit 124. Output pointing coordinates. At this time, the switching determination unit 123 initializes the measurement value of the motion sensor unit 211 used by the estimation unit 122. On the other hand, the switching determination unit 123 outputs the pointing coordinates to the output instruction unit 125 when the pointing coordinates input from the pointing unit 11 are present in the display area of the display unit 101.

When the pointing coordinates estimated by the estimation unit 122 are input from the switching determination unit 123, the weighting unit 124 (correction unit) corrects the pointing coordinates based on the pointing coordinates and the pointing coordinates input from the pointing unit 11. I do.
Specifically, when the pointing coordinates estimated by the estimation unit 122 are input from the switching determination unit 123, the weighting unit 124 performs weighted averaging (weighting) on the pointing coordinates and the pointing coordinates input from the pointing unit 11. ) To calculate the weighted pointing coordinates and cause the display unit 101 to display a pointer in the area represented by the pointing coordinates. Specifically, the pointing coordinates estimated by the estimating unit 122 and the pointing coordinates estimated by the pointing unit 11 are weighted in a stepwise manner from 0 to 1, and pointing is performed when the pointer is displayed on the display unit 101. Reduce the effects of coordinate estimation errors. The weighting unit 124 displays the weighted pointing coordinates on the display unit 101.

  The output instruction unit 125 causes the display unit 101 to display the pointer as it is in the area represented by the pointing coordinates input from the switching determination unit 123.

  As described above, according to the present embodiment, the pointing system S1 includes the display device 1 that displays video and the electronic device 2 that includes the imaging unit (camera 201) that images the display device 1 that displays video. The motion sensor unit 211 that measures the measurement value representing the attitude of the electronic device, and the optical axis of the imaging unit (camera 201) determines the display area of the display device 1 based on the measurement value measured by the motion sensor unit 211. A posture calculation unit 121 that calculates a point that intersects the plane including the pointing position, a determination unit (switch determination unit 123) that determines whether or not the pointing position is within the display area, and a determination unit (switch determination unit 123). A correction unit (weighting unit 124) that corrects the pointing position when it is determined that it is within the display area;

  Thus, the pointing system S1 corrects the pointing position obtained by the motion sensor unit 211, thereby improving the accuracy of the pointing position. Therefore, it is possible to improve the convenience of the user when pointing. Further, the pointing system S1 can display a pointer on the display unit 101 from the pointing position calculated from the image obtained by imaging with the imaging unit (camera 201) and the pointing position calculated from the motion sensor. Even when the pointer deviates from the display area of the unit 101, the movement of the pointer can be estimated and pointing can be performed.

  The correction unit (weighting unit 124) corrects the pointing position based on the comparison between the image obtained by the imaging unit (camera 201) and the video.

  Thus, the pointing system S1 corrects the pointing position obtained by the motion sensor unit 211, thereby improving the accuracy of the pointing position. Therefore, it is possible to improve the convenience of the user when pointing.

  The pointing system S1 also generates a conversion rule generation unit (imaging) that generates a conversion rule from the image coordinate system to the video coordinate system based on the video and the image obtained by the imaging unit (camera 201). The position of the pointer that causes the display unit 101 to display the point where the optical axis of the imaging unit (camera 201) intersects the display unit 101 included in the display device 1 is determined with reference to the surface corner coordinate calculation unit 112) and the conversion rule. And an estimation unit (pointing unit 11) that estimates as information to be performed.

  Thus, the pointing system S1 can estimate the corner of the display unit 101 and its coordinates in the captured image, and displays the pointer by projective conversion of the center coordinates of the image to the coordinates of the display unit 101 based on the estimated corner coordinates. The part 101 can be pointed. Furthermore, the pointing system S1 can improve the convenience of the user at the time of pointing. Further, the pointing system S1 can display a pointer on the display unit 101 from the pointing position calculated from the image obtained by imaging with the imaging unit (camera 201) and the pointing position calculated from the motion sensor. Even when the pointer deviates from the display area of the unit 101, the movement of the pointer can be estimated and pointing can be performed.

  In addition, the conversion rule generation unit (pointing unit 11) includes a corner detection unit 111 that detects a corner in the display unit 101 from an image acquired by the imaging unit (camera 201), and a display unit 101 in the image detected by the corner detection unit 111. An imaging surface corner coordinate calculation unit 112 that estimates the coordinates of the corner of the image, and a projective transformation that performs projective conversion of the center coordinates c of the image to the coordinates C of the display unit 101 based on the corner coordinates estimated by the imaging surface corner coordinate calculation unit 112. And a unit 113.

  Thus, the pointing system S1 can estimate the corner of the display unit 101 and its coordinates in the captured image, and displays the pointer by projective conversion of the center coordinates of the image to the coordinates of the display unit 101 based on the estimated corner coordinates. The part 101 can be pointed. Furthermore, the pointing system S1 can improve the convenience of the user at the time of pointing. Further, the pointing system S1 can display a pointer on the display unit 101 from the pointing position calculated from the image obtained by imaging with the imaging unit (camera 201) and the pointing position calculated from the motion sensor. Even when the pointer deviates from the display area of the unit 101, the movement of the pointer can be estimated and pointing can be performed.

  The motion sensor unit 211 includes an acceleration sensor 2111 that measures acceleration and a geomagnetic sensor 2112 that measures the direction of the earth's magnetic field. The control unit 12 estimates the movement of the pointer outside the display area of the display unit 101 from the acceleration measured by the acceleration sensor 2111 and the direction of the magnetic field measured by the geomagnetic sensor 2112, and returns to the display area of the display unit 101. At this time, the pointer is pointed on the display unit 101 based on the estimated pointer movement and the pointer position estimated from the image obtained by the imaging unit (camera 201).

  Thus, the pointing system S1 can estimate the corner of the display unit 101 and its coordinates in the captured image, and displays the pointer by projective conversion of the center coordinates of the image to the coordinates of the display unit 101 based on the estimated corner coordinates. The part 101 can be pointed. Furthermore, the pointing system S1 can improve the convenience of the user at the time of pointing. Further, the pointing system S1 can display a pointer on the display unit 101 from the pointing position calculated from the image obtained by imaging with the imaging unit (camera 201) and the pointing position calculated from the motion sensor. Even when the pointer deviates from the display area of the unit 101, the movement of the pointer can be estimated and pointing can be performed.

(Second Embodiment)
Hereinafter, a second embodiment of the present invention will be described in detail with reference to the drawings.
FIG. 11 is a schematic block diagram showing an example of the configuration of the display device 1a according to the second embodiment of the present invention.
The display device 1 a includes a display unit 101 and a screen extraction unit 13. The screen extraction unit 13 includes a screen acquisition unit 100, a screen feature point extraction unit 102, a screen division unit 103, a screen feature point count unit 104, an area selection unit 105, a screen compression unit 106, and a storage unit 107. The control unit 108 and the communication unit 109 are configured. Although the display device 1a has other functions of a known display device, illustration and description thereof are omitted.

  The screen acquisition unit 100 acquires a display video to be displayed on the display unit 101. The screen acquisition unit 100 causes the display unit 101 to display the acquired display video. The screen acquisition unit 100 outputs the acquired display video to the screen feature point extraction unit 102. The screen feature point extraction unit 102 extracts feature points from the display video. Specifically, the screen feature point extraction unit 102 compares, for example, luminance values for each pixel in the display video, extracts feature points based on the luminance values, and extracts feature amounts of the feature points. Here, the feature amount of a feature point is a unique property using luminance gradient information of a peripheral region including the feature point, and is represented by a multidimensional vector such as 64 dimensions or 128 dimensions. . The screen feature point extraction unit 102 causes the storage unit 107 to store the extracted feature points of the display video, the feature amounts of the feature points, and the display video. When the screen feature point extraction unit 102 completes the extraction of the feature points of the display video and the feature amount of the feature points, the screen feature point extraction unit 102 outputs an extraction completion signal to the screen division unit 103.

  When the extraction completion signal is input from the screen feature point extraction unit 102, the screen division unit 103 reads the display video from the storage unit 107. The screen dividing unit 103 divides the display video into a plurality of areas. 12 and 13 are explanatory diagrams illustrating an example of the operation of the screen dividing unit 103 of the display device 1a according to the present embodiment. The screen dividing unit 103 divides the display video G1 into, for example, four areas A1, A2, A3, and A4 as shown in FIG. Then, the screen dividing unit 103 further divides the divided area A1 into four sub areas A1a, A1b, A1c, and A1d as shown in FIG. Similarly, the screen dividing unit 103 divides each of the areas A2, A3, and A4 into four subareas. That is, the screen dividing unit 103 divides the display video G1 into four areas, and further divides each display area into four subareas to divide the display video into sixteen. The screen dividing unit 103 stores information indicating the divided subareas in the storage unit 107 and outputs the information to the screen feature point counting unit 104.

  When the information indicating the sub area is input from the screen dividing unit 103, the screen feature point counting unit 104 reads the feature point of the display video from the storage unit 107, and the feature point for each sub area based on the information indicating the sub area. Count the number of The screen feature point counting unit 104 outputs the number of feature points counted for each sub-area to the area selection unit 105. Based on the number of feature points for each sub-area input from the screen feature point counting unit 104, the area selection unit 105 ranks each sub-area with, for example, 1st to 16th ranks. The area selection unit 105 compares the ranks of the subareas in each area, and selects, for example, one subarea having the highest rank for each area, for example, a total of four subareas. The area selection unit 105 sets the selected four sub areas as effective areas. The area selection unit 105 stores information indicating the effective area in the storage unit 107 and outputs the information to the screen compression unit 106.

When the information indicating the effective area is input from the area selection unit 105, the screen compression unit 106 reads the display video from the storage unit 107, and compresses the display video to an arbitrary magnification, for example, 1/4. The screen compression unit 106 reads the feature points of the display image and the feature values of the feature points from the storage unit 107, and displays the feature points of the display image before compression and the features of the feature points with respect to the display image compressed at an arbitrary magnification. The amounts are associated and stored in the storage unit 107. The screen compression unit 106 reads information indicating the effective area from the storage unit 107, associates information indicating the effective area before compression with the display image compressed at an arbitrary magnification, and causes the storage unit 107 to store the information.
The storage unit 107 includes a display image, a feature point and a feature amount of the feature point, information indicating a sub area, information indicating an effective area, a compressed display image, a feature point of the display image, and a feature amount of the feature point, Information indicating the effective area of the compressed display image is stored.

The control unit 108 reads out the information indicating the effective area of the compressed display video, the compressed display video, the feature point of the display video, and the feature amount of the feature point from the storage unit 107, and transmits the information through the communication unit 109. To the electronic device 2. The control unit 108 causes the display unit 101 to display a pointer based on information indicating the pointing position received from the electronic apparatus 2 via the communication unit 109, for example, pointing coordinates.
The communication unit 109 sends the information indicating the effective area of the compressed display video input from the control unit 108, the compressed display video, the feature points of the display video, and the feature quantities of the feature points to the electronic device 2. Send. When the communication unit 109 receives information indicating the pointing position from the electronic device 2, for example, pointing coordinates, the communication unit 109 outputs the information to the control unit 108.

FIG. 14 is a schematic block diagram showing an example of the configuration of the electronic device 2a according to this embodiment.
The electronic device 2a includes a camera 201, an image feature point extraction unit 202, a communication unit 203, a control unit 210, a motion sensor unit 211, a threshold storage unit 212, a filter unit (smoothing unit) 213, and pointing Part 220. The motion sensor unit 211 includes an acceleration sensor 2111 and a geomagnetic sensor 2112. The pointing unit 220 includes an acquisition unit 204, a mapping unit 208, and a conversion rule generation unit 209. Also, the conversion rule generation unit 209 includes a feature point comparison unit 205, a corner estimation unit 206, and a conversion matrix generation unit 207. The control unit 210 includes an attitude calculation unit 2101, an estimation unit 2102, a switching determination unit 2103, a weighting unit 2104, and an output instruction unit 2105. The electronic device 2 has other functions of a general electronic device, but illustration and description thereof are omitted.

  The camera 201 is a visible light camera, captures a display image displayed on the display unit 101 of the display device 1a, and acquires a camera image. The camera 201 outputs the acquired camera image to the image feature point extraction unit 202. The image feature point extraction unit 202 extracts feature points from the camera image input from the camera 201. Specifically, for example, the image feature point extraction unit 202 compares the luminance values for each pixel in the camera image, extracts the feature points based on the luminance values, and extracts the feature amounts of the feature points. The image feature point extraction unit 202 outputs the extracted feature points of the camera image and the feature amounts of the feature points to the feature point comparison unit 205.

The communication unit 203 outputs to the acquisition unit 204 the feature points of the display image received from the display device 1a, the feature amounts of the feature points, and information indicating the effective area. The communication unit 203 transmits information indicating the pointing position input from the filter unit (smoothing unit) 213 to the display device 1a.
The acquisition unit 204 acquires information indicating the feature point of the display image, the feature amount of the feature point, and the effective area from the display device 1a via the communication unit 203. The acquisition unit 204 outputs the feature points of the acquired display video, the feature amounts of the feature points, and information indicating the effective area to the feature point comparison unit 205.

  The feature point comparison unit 205 includes a feature point of the camera image input from the image feature point extraction unit 202 and a feature amount of the feature point, a feature point of the display image input from the acquisition unit 204, and a feature amount of the feature point. And the feature value of the feature point of the camera image is compared with the feature value of the feature point of the display image in the effective region based on the information indicating the effective region, and the feature point of the display image in the effective region is compared A matching process for associating feature points is performed. Specifically, the feature point comparison unit 205 refers to the information indicating the effective area, and the feature amount of the feature point of the display image in the effective area matches the feature amount of the feature point of the camera image most frequently. A pair of feature points in which the feature points of the camera image are associated with the feature points of the display video is generated. The feature point comparison unit 205 outputs the generated feature point pair to the corner estimation unit 206. An example of the matching process of the feature point comparison unit 205 will be described in detail with reference to FIG.

FIG. 15 is an explanatory diagram illustrating an example of matching processing in the feature point comparison unit 205 of the pointing unit 220 of the electronic device 2a according to the present embodiment.
When the information indicating the effective area in the display video G2 represents A1a, A2d, A3b, and A4c in FIG. 13, when there are four feature points included in each sub-area, for example, feature point comparison The unit 205 associates the feature points of the camera image G3 with the feature points of the display image G2 so that the feature amounts of the feature points of the display image G2 and the feature points of the camera image G3 most closely match. FIG. 15 illustrates the feature points of the camera image G3 corresponding to the display video G2 connected by lines. The feature point comparison unit 205 outputs the feature points of the display video G2 connected by the lines in FIG. 15 and the feature points of the camera image G3 to the corner estimation unit 206 as feature point pairs.

  The corner estimation unit 206 repeatedly generates a subset of feature points input from the feature point comparison unit 205, for example, a set of feature points including arbitrary four feature point pairs. The corner estimation unit 206 estimates a homography matrix (mat) using a set of four feature points generated repeatedly and a least square method. Here, the homography matrix is to examine the plane in the space and the projection of the image (plane), and is generally a matrix of 3 rows and 3 columns when given a correspondence of two-dimensional points.

  The corner estimation unit 206 calculates the coordinates of the corners (corners) of the display unit 101 of the display device 1a in the camera image using the estimated homography matrix (mat). The corner estimation unit 206 calculates the corner coordinates of the camera image coordinate system by perspective-transforming the corner coordinates of the display video coordinate system using the estimated homography matrix. Here, the display video coordinate system is a coordinate system on the display video plane, and the camera image coordinate system is a coordinate system on the camera image plane. The corner estimation unit 206 outputs the calculated corner coordinates of the camera image coordinate system to the transformation matrix generation unit 207. With reference to FIG. 16, an example of the corner estimation means of the corner estimation unit 206 will be described in detail.

FIG. 16 is an explanatory diagram illustrating an example of a corner estimation unit in the corner estimation unit 206 of the pointing unit 220 of the electronic device 2a according to the present embodiment.
When the information indicating the effective area in the display video G4 shown in FIG. 16 represents A1a, A2d, A3b, and A4c in FIG. 13, there are, for example, four feature points included in each subarea. Then, the corner estimation unit 206 extracts any four feature point pairs with the feature points of the camera image G5 in FIG. 16 associated with the feature points of the display video G4 in FIG. 16 in the feature point comparison unit 205. A subset (set) composed of four pairs of feature points is repeatedly generated. The corner estimation unit 206 estimates a homography matrix using the generated subset and the least square method. The corner estimation unit 206 calculates the coordinates of the corner C2 of the camera image G5 in FIG. 16 corresponding to the corner C1 of the display video G4 in FIG. 16 by performing perspective transformation using the estimated homography matrix. The corner estimation unit 206 outputs the calculated coordinates of the corner C2 to the conversion matrix generation unit 207.

  The transformation matrix generation unit 207 calculates the perspective transformation matrix PMat from the corner coordinates M of the display video coordinate system and the corner coordinates m of the camera image coordinate system input from the corner estimation unit 206 by Expression (1).

The transformation matrix generation unit 207 performs the coordinate m (x, y, 1) of the corner of the camera image coordinate system in the camera image G6 shown in FIG. 7 and the coordinate M of the corner of the display video coordinate system in the display video G7 shown in FIG. A perspective transformation matrix (PMat) is calculated from (tx, ty, t). The transformation matrix generation unit 207 outputs the calculated perspective transformation matrix to the mapping unit 208.
The mapping unit 208 performs a mapping process based on the perspective transformation matrix input from the transformation matrix generation unit 207. The mapping process of the mapping unit 208 will be described in detail with reference to FIG.

  The mapping unit 208 estimates a point where the optical axis of the camera 201 intersects with the display unit 101 of the display device 1a. Specifically, the mapping unit 208 converts the center coordinate c of the camera image coordinate system to the coordinate C of the display video coordinate system based on the perspective conversion matrix input from the conversion matrix generation unit 207. For example, assuming that the resolution of the camera 201 is 640 × 480 pixels, the center coordinates c of the camera image coordinate system are (320, 240, 1). In the matrix, the mapping unit 208 converts the center coordinates c (320, 240, 1) of the camera image coordinate system into coordinates C (X, Y, Z) of the display video coordinate system using the perspective transformation matrix. The mapping unit 208 outputs the calculated coordinates C to the control unit 210.

  Since the operation of the motion sensor unit 211 of the electronic device 2a according to the present embodiment is the same as that of the motion sensor unit 211 of the electronic device 2 according to the first embodiment, the description thereof is omitted. The operation of the control unit 210 of the electronic device 2a according to the present embodiment is the same as that of the control unit 12 of the display device 1 according to the first embodiment, and a description thereof will be omitted.

The threshold storage unit 212 stores an acceleration threshold related to the acceleration sensor.
The control unit 210 compares the acceleration input from the acceleration sensor 2111 with the acceleration threshold value read from the threshold value storage unit 212. When the acceleration value is less than the threshold value, the control unit 210 causes the filter unit (smoothing unit) 213 to smooth the pointing coordinates output from the output instruction unit 2105 or the weighting unit 2104. On the other hand, when the acceleration value is equal to or greater than the threshold value, the control unit 210 converts the pointing coordinates input from the output instruction unit 2105 or the weighting unit 2104 without being smoothed by the filter unit (smoothing unit) 213 into the communication unit. The data is transmitted to the display device 1a via 203.

  The filter unit (smoothing unit) 213 performs smoothing, for example, a two-dimensional low-pass filter or movement, when a signal indicating that the pointing coordinates and acceleration values input from the control unit 210 are less than a threshold value is input. The average is used to reduce the vibration of the pointing position of the pointer due to the physiological vibration of the user. The filter unit (smoothing unit) 213 transmits the smoothed pointing coordinates to the display device 1a via the communication unit 203 as information indicating the pointing position. The filter unit (smoothing unit) 213 displays the pointing coordinate and the acceleration value input from the control unit 210 via the communication unit 203 without performing smoothing when a signal indicating that the acceleration value is equal to or greater than a threshold value is input. Pointing coordinates are transmitted to the device 1a as information indicating the pointing position.

  As described above, according to the present embodiment, the pointing system S1 includes the display device 1a that displays video and the electronic device 2a that includes the imaging unit (camera 201) that images the display device 1 that displays video. Then, based on the measurement value measured by the motion sensor unit 211 that measures the measured value representing the attitude of the electronic device 2a, the optical axis of the imaging unit (camera 201) is the display area of the display device 1a. A posture calculation unit 2101 that calculates a point that intersects a plane including a pointing position, a determination unit that determines whether or not the pointing position is within the display area (switch determination unit 2103), and a determination unit (switch determination unit 2103) And a correction unit (weighting unit 2104) that corrects the pointing position when it is determined that it is within the display area. That.

  Thus, the pointing system S1 corrects the pointing position obtained by the motion sensor unit 211, thereby improving the accuracy of the pointing position. Therefore, it is possible to improve the convenience of the user when pointing. Further, the pointing system S1 can display a pointer on the display unit 101 from the pointing position calculated from the image obtained by imaging with the imaging unit (camera 201) and the pointing position calculated from the motion sensor. Even when the pointer deviates from the display area of the unit 101, the movement of the pointer can be estimated and pointing can be performed.

  The correction unit (weighting unit 2104) corrects the pointing position based on the comparison between the image obtained by the imaging unit (camera 201) and the video.

  Thus, the pointing system S1 corrects the pointing position obtained by the motion sensor unit 211, thereby improving the accuracy of the pointing position. Therefore, it is possible to improve the convenience of the user when pointing.

  Further, the pointing system S1 includes a conversion rule generation unit 209 that generates a conversion rule from the image coordinate system to the video coordinate system based on the video and the image obtained by the imaging unit (camera 201). Referring to the conversion rule, estimation that estimates the point at which the optical axis of the imaging unit (camera 201) intersects with the display unit 101 included in the display device 1a as the information for determining the position of the pointer to be displayed on the display unit 101 Unit (pointing unit 220).

  Thus, the pointing system S1 can estimate the corner of the display unit 101 and its coordinates in the captured image, and displays the pointer by projective conversion of the center coordinates of the image to the coordinates of the display unit 101 based on the estimated corner coordinates. The part 101 can be pointed. Furthermore, the pointing system S1 can improve the convenience of the user at the time of pointing. Further, the pointing system S1 can display a pointer on the display unit 101 from the pointing position calculated from the image obtained by imaging with the imaging unit (camera 201) and the pointing position calculated from the motion sensor. Even when the pointer deviates from the display area of the unit 101, the movement of the pointer can be estimated and pointing can be performed.

  Also, the conversion rule generation unit 209 is detected by a corner detection unit (corner estimation unit 206) that detects a corner in the display unit 101 from an image acquired by the imaging unit (camera 201), and a corner detection unit (corner estimation unit 206). An imaging plane corner coordinate calculation unit (corner estimation unit 206) that estimates the corner coordinates of the display unit 101 in the obtained image and the corner coordinates estimated by the imaging plane corner coordinate calculation unit (corner estimation unit 206). A projection conversion unit (conversion matrix generation unit 207) that performs projective conversion of the center coordinates c into the coordinates C of the display unit 101;

  Thus, the pointing system S1 can estimate the corner of the display unit 101 and its coordinates in the captured image, and displays the pointer by projective conversion of the center coordinates of the image to the coordinates of the display unit 101 based on the estimated corner coordinates. The part 101 can be pointed. Furthermore, the pointing system S1 can improve the convenience of the user at the time of pointing. Further, the pointing system S1 can display a pointer on the display unit 101 from the pointing position calculated from the image obtained by imaging with the imaging unit (camera 201) and the pointing position calculated from the motion sensor. Even when the pointer deviates from the display area of the unit 101, the movement of the pointer can be estimated and pointing can be performed.

  The motion sensor unit 211 includes an acceleration sensor 2111 that measures acceleration and a geomagnetic sensor 2112 that measures the direction of the earth's magnetic field. The control unit 210 estimates the movement of the pointer outside the display area of the display unit 101 from the acceleration measured by the acceleration sensor 2111 and the direction of the magnetic field measured by the geomagnetic sensor 2112, and returns to the display area of the display unit 101. At this time, the pointer is pointed on the display unit 101 based on the estimated pointer movement and the pointer position estimated from the image obtained by the imaging unit (camera 201).

  Thus, the pointing system S1 can estimate the corner of the display unit 101 and its coordinates in the captured image, and displays the pointer by projective conversion of the center coordinates of the image to the coordinates of the display unit 101 based on the estimated corner coordinates. The part 101 can be pointed. Furthermore, the pointing system S1 can improve the convenience of the user at the time of pointing. Further, the pointing system S1 can display a pointer on the display unit 101 from the pointing position calculated from the image obtained by imaging with the imaging unit (camera 201) and the pointing position calculated from the motion sensor. Even when the pointer deviates from the display area of the unit 101, the movement of the pointer can be estimated and pointing can be performed.

  The motion sensor unit 211 includes an acceleration sensor 2111 that measures acceleration and a geomagnetic sensor 2112 that measures the direction of the earth's magnetic field. The control unit 12 estimates the movement of the pointer outside the display area of the display unit 101 from the acceleration measured by the acceleration sensor 2111 and the direction of the magnetic field measured by the geomagnetic sensor 2112, and returns to the display area of the display unit 101. At this time, the pointer is pointed on the display unit 101 based on the estimated pointer movement and the pointer position estimated from the image obtained by the imaging unit (camera 201).

  Thus, the pointing system S1 can estimate the corner of the display unit 101 and its coordinates in the captured image, and displays the pointer by projective conversion of the center coordinates of the image to the coordinates of the display unit 101 based on the estimated corner coordinates. The part 101 can be pointed. Furthermore, the pointing system S1 can improve the convenience of the user at the time of pointing. Further, the pointing system S1 can display a pointer on the display unit 101 from the pointing position calculated from the image obtained by imaging with the imaging unit (camera 201) and the pointing position calculated from the motion sensor. Even when the pointer deviates from the display area of the unit 101, the movement of the pointer can be estimated and pointing can be performed.

  The motion sensor unit 211 is a gyro sensor that measures acceleration and angular velocity, and the control unit 210 estimates the movement of the pointer outside the display area of the display unit 101 from the acceleration and angular velocity measured by the gyro sensor, and displays them. When returning to the display area of the unit 101, the pointer is pointed to the display unit 101 based on the estimated pointer movement and the position of the pointer estimated from the image obtained by the imaging unit (camera 201). Let

  Thus, the pointing system S1 can estimate the corner of the display unit 101 and its coordinates in the captured image, and displays the pointer by projective conversion of the center coordinates of the image to the coordinates of the display unit 101 based on the estimated corner coordinates. The part 101 can be pointed. Furthermore, the pointing system S1 can improve the convenience of the user at the time of pointing. Further, the pointing system S1 can display a pointer on the display unit 101 from the pointing position calculated from the image obtained by imaging with the imaging unit (camera 201) and the pointing position calculated from the motion sensor. Even when the pointer deviates from the display area of the unit 101, the movement of the pointer can be estimated and pointing can be performed.

  The pointing system S1 further includes a smoothing unit 213 that smoothes the movement of the pointer. The control unit 210 controls whether or not the smoothing unit 210 performs smoothing according to the acceleration.

  Thereby, pointing system S1 can prevent the vibration of the pointing position by a user's physiological vibration.

  In each embodiment, the pointing unit 11 or 220 may be included in the electronic devices 2 and 2a, or may be included in the display devices 1 and 1a. When the display device 1 or 1a includes the pointing unit 11 or 220, the display device 1 or 1a may be configured like the display device 1 or the electronic device 2 of the first embodiment. Further, when the electronic devices 2 and 2a include the pointing units 11 and 220, they may be configured as the display device 1a and the electronic device 2a in the second embodiment.

  In each embodiment, the motion sensor unit 211 includes the acceleration sensor 2111 and the geomagnetic sensor 2112, but a gyro sensor may be used instead of the acceleration sensor 2111 or the geomagnetic sensor 2112. In this case, the gyro sensor may measure the angle and acceleration of the electronic devices 2 and 2a.

  In each embodiment, feature points are extracted based on luminance values, but feature points may be extracted using SIFT, SURF, ORB, or the like. Furthermore, although the area selection unit 105 of the display device 1a selects four subareas, only one subarea may be selected, or one or more subareas may be selected. . The area selection unit 105 can perform high-speed processing calculation by selecting a small number, for example, one area, and can improve corner estimation accuracy in the corner estimation unit 206 by selecting a plurality of areas. The feature point comparison unit 205 extracts a plurality of feature point pairs and repeatedly generates a subset. However, it is sufficient that there are at least four pairs (one set) of feature point pairs. The feature point comparison unit 205 can improve the accuracy of calculating the coordinates C in the corner estimation unit 206 and the mapping unit 208 by using a plurality of pairs of feature points, and can improve the estimation of the pointing position.

  Furthermore, although the screen compression unit 106 compresses the display video at an arbitrary magnification, the display video may be compressed at a plurality of magnifications with different compression rates and stored in the storage unit 107. Thereby, even when the electronic device 2a is at a distance from the display device 1a, it is possible to appropriately extract feature points and calculate the pointing position in the camera image acquired by the camera 201.

  In addition, you may make it implement | achieve a part or all of the display apparatuses 1 and 1a and the electronic devices 2 and 2a in embodiment mentioned above with a computer. In that case, the program for realizing the control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by a computer system and executed. Here, the “computer system” is a computer system built in the display devices 1 and 1a and the electronic devices 2 and 2a and includes hardware such as an OS and peripheral devices.

  The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” is a medium that dynamically holds a program for a short time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, In such a case, a volatile memory inside a computer system serving as a server or a client may be included and a program that holds a program for a certain period of time.

  The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  Moreover, you may implement | achieve part or all of the display apparatuses 1 and 1a and the electronic devices 2 and 2a in embodiment mentioned above as integrated circuits, such as LSI (Large Scale Integration). Each functional block of the display devices 1 and 1a and the electronic devices 2 and 2a may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Further, in the case where an integrated circuit technology that replaces LSI appears due to progress in semiconductor technology, an integrated circuit based on the technology may be used.

  As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the above, and various design changes and the like can be made without departing from the scope of the present invention. It is possible to

S1 Pointing system 1, 1a Display device 2, 2a Electronic device 100 Screen acquisition unit 101 Display unit 102 Screen feature point extraction unit 103 Screen division unit 104 Screen feature point count unit 105 Area selection unit 106 Screen compression unit 107 Storage unit 108 Control unit 109 communication unit 110 communication unit 11 pointing unit (conversion rule generation unit)
111 Corner detection unit 112 Imaging surface corner coordinate calculation unit 113 Projection conversion unit 12 Control unit 121 Posture calculation unit (calculation unit)
122 estimation unit 123 switching determination unit (determination unit)
124 Weighting unit (correction unit)
125 Output instruction unit 13 Screen extraction unit 201 Camera (imaging unit)
202 Image feature point extraction unit 203 Communication unit 204 Acquisition unit 205 Feature point comparison unit 206 Corner estimation unit 207 Transformation matrix generation unit 208 Mapping unit 209 Conversion rule generation unit 210 Control unit 2101 Posture calculation unit (calculation unit)
2102 estimation unit 2103 switching determination unit (determination unit)
2104 Weighting unit (correction unit)
2105 Output instruction unit 211 Motion sensor unit 2111 Acceleration sensor 2112 Geomagnetic sensor 212 Threshold storage unit 213 Filter unit (smoothing unit)
220 Pointing part (estimator)

Claims (10)

A pointing system comprising: a display device that displays video; and an electronic device that includes an imaging unit that images the display device displaying the video,
A motion sensor unit for measuring a measurement value representing the posture of the electronic device;
Based on the measurement value measured by the motion sensor unit, a calculation unit that calculates a point where the optical axis of the imaging unit intersects a plane including the display area of the display device as a pointing position;
A determination unit that determines whether or not the pointing position is within the display area;
A correction unit that corrects the pointing position when the determination unit determines that it is within the display area;
A pointing system characterized by comprising: The pointing system according to claim 1, wherein the correction unit corrects the pointing position based on a comparison between an image obtained by the imaging unit and the video. A conversion rule generating unit that generates a conversion rule from the coordinate system of the image to the coordinate system of the video based on the video and the image obtained by imaging by the imaging unit;
With reference to the conversion rule, an estimation unit that estimates, as information for determining a position of a pointer to be displayed on the display unit, a point at which the optical axis of the imaging unit intersects the display unit included in the display device;
Further comprising
The pointing system according to claim 2, wherein the correction unit corrects the pointing position using the information estimated by the estimation unit. The conversion rule generation unit
A corner detection unit for detecting a corner in the display unit from the image acquired by the imaging unit;
An imaging surface corner coordinate calculation unit that estimates the coordinates of the corner of the display unit in the image detected by the corner detection unit;
A projective conversion unit for projectively converting the center coordinates of the image to the coordinates of the display unit based on the coordinates of the corners estimated by the imaging surface corner coordinate calculation unit;
The pointing system according to claim 3, further comprising: The motion sensor unit is
An acceleration sensor for measuring acceleration;
A geomagnetic sensor that measures the direction of the earth's magnetic field;
With
The correction unit estimates the movement of the pointer outside the display area of the display unit from the acceleration measured by the acceleration sensor and the direction of the magnetic field measured by the geomagnetic sensor, and moves the correction unit to the display area of the display unit. When returning, the pointing position is corrected based on the estimated movement of the pointer and the position of the pointer estimated from the image captured by the imaging unit. The pointing system according to claim 4. The motion sensor unit is a gyro sensor that measures acceleration and angular velocity,
The correction unit estimates the pointer movement outside the display area of the display unit from the acceleration and angular velocity measured by the gyro sensor, and returns the estimated pointer when the display unit returns to the display area of the display unit. 5. The pointing position is corrected based on the movement of the image and the position of the pointer estimated from the image obtained by the imaging unit. 5. The pointing system described in. A smoothing unit for smoothing the movement of the pointer;
The pointing system according to any one of claims 1 to 6, wherein the control unit controls the smoothing unit in accordance with the acceleration measured by the motion sensor unit. A pointing method in a pointing system, comprising: a display device that displays an image; and an electronic device that includes an imaging unit that images the display device that displays the image,
A first step of measuring a measurement value representing the attitude of the electronic device;
A second step of calculating, as a pointing position, a point where an optical axis of the imaging unit intersects a plane including a display area of the display device based on the measurement value measured in the first step;
A third step of determining whether the pointing position is within the display area;
A fourth step of correcting the pointing position when it is determined in the third step that the region is within the display area;
A pointing method characterized by comprising: A display device in a pointing system, comprising: a display device that displays a video; and an electronic device that includes an imaging unit that images the display device that displays the video,
Based on the measurement value measured by the motion sensor unit of the electronic device, a calculation unit that calculates a point where the optical axis of the imaging unit intersects a plane including the display area of the display device as a pointing position;
A determination unit that determines whether or not the pointing position is within the display area;
A correction unit that corrects the pointing position when the determination unit determines that it is within the display area;
A display device comprising: An electronic device in a pointing system comprising: a display device that displays an image; and an electronic device that includes an imaging unit that images the display device that displays the image,
A pointing unit that estimates a corner of the display unit included in the display device from an image obtained by imaging by the imaging unit, and calculates a position at which the pointer is pointed to the display unit from the estimated corner;
A motion sensor unit for measuring a measurement value representing the posture of the electronic device;
Based on the measurement value measured by the motion sensor unit, a calculation unit that calculates a point where the optical axis of the imaging unit intersects a plane including the display area of the display device as a pointing position;
A determination unit that determines whether or not the pointing position is within the display area;
A correction unit that corrects the pointing position when the determination unit determines that it is within the display area;
An electronic device comprising: