JP2020122696A - Position detection device, position detection unit, image display system, and position detection method - Google Patents

Abstract

To provide a position detection device, a position detection unit, an image display system, and a position detection method that can perform position detection with high accuracy.SOLUTION: A position detection device comprises: a first indicator that indicates a display surface on which a structured light pattern is displayed; a first imaging device that is provided in the first indicator and picks up an image of the structured light pattern displayed on the display surface; a second imaging device that picks up an image of the structured light pattern displayed on the display surface; and an image processing unit. The first imaging device acquires a first image obtained by picking up an image of an area including, of a plurality of projection points of the structured light pattern located around the first indicator, at least four points different from each other; the second imaging device acquires a second image obtained by picking up an image of the entirety of a display area of the structured light pattern on the display surface indicated by the first indicator; the image processing unit detects position information on the display surface indicated by the first indicator based on the first image and the second image.SELECTED DRAWING: Figure 1

Description

The present invention relates to a position detection device, a position detection unit, an image display system and a position detection method.

Conventionally, in position detection using structured light, one camera can recognize a three-dimensional shape, and the number of devices can be reduced as compared with the case of using a stereo camera. Position detection using structured light can also be applied as touch recognition technology for a screen onto which an image from a projector is projected. As a position detecting method using such structured light, there is a technique of projecting a random speckle pattern (for example, refer to Patent Document 1).

Japanese Patent Publication No. 2009-511897

However, in the case of the method of projecting a random speckle pattern, if the pattern pitch is made fine in order to improve the depth resolution, the data in the depth direction is repeated, and only a pattern having a coarse pitch can be projected. As a result, only a low-resolution image can be obtained, and the position detection accuracy deteriorates.

According to the first aspect of the present invention, a first indicator for indicating a display surface on which a structured light pattern is displayed, and the structured indicator provided on the first indicator and displayed on the display surface. A first image pickup device for picking up a light pattern, a second image pickup device for picking up the structured light pattern displayed on the display surface, and an image processing unit, wherein the first image pickup device is the first image pickup device. Of the plurality of projection points of the structured light pattern located around the pointer of No. 1, the first image obtained by capturing an area including at least three points different from each other is acquired, and the second image pickup apparatus sets the first image pickup apparatus to the first image pickup apparatus. A second image obtained by capturing the entire display area of the structured light pattern on the display surface indicated by the indicator is acquired, and the image processing unit, based on the first image and the second image, A position detection device that detects position information on the display surface indicated by a first indicator is provided.

In the above-mentioned first aspect, when a second indicator points to the display surface, the image processing unit controls the display area of the structured light pattern on the display surface pointed to by the second indicator. It is preferable to detect the position information on the display surface indicated by the second indicator using the third image obtained by capturing the entire image with the second image pickup device.

In the first aspect, the central axis of the pointing portion of the first pointer is preferably parallel to the optical axis of the first imaging device.

In the first aspect, the central axis preferably coincides with the optical axis.

According to a second aspect of the present invention, there is provided a position detection unit including the position detection device according to the first aspect, and a structured light display section that displays the structured light pattern toward the display surface. ..

According to a third aspect of the present invention, there is provided an image display system including the position detection unit according to the second aspect, and an image display device that displays an image on the display surface.

In the third aspect, the image processing unit is provided in the first indicator, the first indicator transmits the position information to the image display device, the image display device, It is preferable to control the image displayed on the display surface based on the position information transmitted from the first indicator.

In the third aspect, it is preferable that the first indicator further includes a data transfer unit that transfers data to the image display device by wireless communication.

According to the fourth aspect of the present invention, at least four different projection points among the plurality of projection points of the structured light pattern located around the first indicator pointing the display surface on which the structured light pattern is displayed. A first image acquisition step of acquiring a first image of a region including a projected point of a point; and a first image of the entire display region of the structured light pattern on the display surface indicated by the first indicator. A second image acquisition step of acquiring two images, and a detection step of detecting positional information on the display surface indicated by the first indicator based on the first image and the second image. There is provided a position detecting method comprising.

It is a side view which shows the structure of an image display system. FIG. 6 is a diagram showing a structured light pattern displayed on a screen. It is the figure which expanded the principal part of the electronic pen which points a screen. It is a figure which shows the 1st image imaged by the 1st camera. It is a figure which shows the 2nd image imaged by the 2nd camera. It is a figure which shows the structure of the electronic pen of this modification.

An embodiment of a position detecting device, a position detecting unit, an image display system, and a position detecting method of the present invention will be described below with reference to the drawings.

(Image display system)
FIG. 1 is a side view showing the configuration of the image display system.
As shown in FIG. 1, the image display system 1 of the present embodiment enables highly accurate detection of the position of the electronic pen (first indicator) P on the flat screen (display surface) SC. It is a device.

As shown in FIG. 1, the image display system 1 of the present embodiment includes a position detection unit 2 and a projector (image display device) 3 that displays a display image (image) on the screen SC.

(Position detection unit)
The position detection unit 2 has a structured light display unit 10 and a position detection device 20.
The structured light display unit 10 displays the structured light pattern LP on the screen SC. The structured light display unit 10 includes, for example, a light source that emits near-infrared light in a wavelength band of about 840 nm, a diffractive optical element that diffracts near-infrared light emitted from the light source and shapes the light into a desired pattern, and a diffractive optical element. And a lens system that projects the near-infrared light.

The projector 3 may be, for example, an optical scanning type projector, an LCD type projector, or a DMD type projector as long as it can project the display image V1 on the screen SC. May be. The projector 3 is installed at a position where it does not block the near-infrared light forming the structured light pattern LP emitted from the structured light display unit 10.

FIG. 2 is a diagram showing a structured light pattern displayed on the screen.
As shown in FIG. 2, the structured light pattern LP formed on the screen SC by the structured light display unit 10 has a lattice shape. Here, the pitch of the structured light pattern LP is not particularly limited, and is appropriately set according to the resolution of the camera of the position detection device 20 described later. In FIG. 2, the four points defining the four corners of the structured light pattern LP in the form of a lattice are indicated by L1, L2, L3, and L4.

In the present embodiment, by using near-infrared light as the structured light pattern LP, the structured light pattern LP displayed on the screen SC becomes hard to be visually recognized by human eyes, so that the display image by the projector 3 is deteriorated. There is no fear of causing it. The light source of the structured light display unit 10 is not limited to one that emits near infrared light, and may be a light source that emits visible light. In this case, it is desirable that the structured light pattern LP is given a change in brightness that is difficult for human eyes to visually recognize, or that the structured light pattern LP is periodically displayed at a frequency that is difficult for human eyes to visually recognize. By doing so, it is possible to reduce the deterioration of the display image by the projector 3 due to the overlapping with the structured light pattern LP.

(Position detection device)
As shown in FIG. 1, the position detection device 20 includes a second camera (second imaging device) 21 that captures an image of the screen SC, an image processing circuit (image processing unit) 22, and an electronic pen P. There is.

The second camera 21 is, for example, a NIR camera compatible with near infrared light (NIR light). The second camera 21 includes a light receiving unit having an image sensor (not shown) such as a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor, and a lens system, and an image from the image sensor. And a processing unit that processes a signal.

The second camera 21 is electrically connected to the image processing unit 22 by, for example, wiring. The second camera 21 acquires the second image G2 including the entire display area of the structured light pattern LP on the screen SC designated by the electronic pen P, and outputs the second image G2 to the image processing unit 22 (second image acquisition step). ..
The second image G2 is used to define position information on the screen SC as described later.

The second camera 21 is calibrated in advance with respect to the structured light pattern LP emitted from the structured light display unit 10. That is, the position detection device 20 captures the structured light pattern LP displayed on the screen SC by the structured light display unit 10 with the second camera 21, and captures only the structured light pattern LP projected on the screen SC. The processed image and the captured image of the structured light pattern LP when the object is present between the screen SC and the structured light display unit 10 are processed by the image processing unit 22, whereby the position information of the object ( The coordinate value) can be calculated. The coordinate origin of the position detection unit 2 can be set arbitrarily. For example, the camera origin of the second camera 21 may be the coordinate origin of the unit.

The position detection unit 2 of the present embodiment has highly accurate position information regarding the coordinates of the portion where the structured light pattern LP is projected on the screen SC, that is, the position A1 overlapping the structured light pattern LP shown in FIG. Can be detected.

Here, a case will be considered in which the pointing position of the object is detected when the screen SC is pointed by a pointer whose outer shape is smaller than the lattice shape of the structured light pattern LP. An indicator having an outer shape smaller than the lattice shape of the structured light pattern LP may indicate a portion where the structured light pattern LP is not projected, that is, a place A2 that does not overlap the structured light pattern LP shown in FIG. , It becomes difficult to detect the pointing position of the object.

For example, when giving a presentation, the electronic pen P of the present embodiment can be used as an indicator that indicates a predetermined area on the screen SC.

In the position detection unit 2 of the present embodiment, the position of an indicator smaller than the grid shape of the structured light pattern LP, such as the pen tip of the electronic pen P, can be detected with high accuracy by adopting the following configuration. ..

The electronic pen P of the present embodiment includes a main body 30, a pen tip (pointing portion) 31 provided at the tip of the main body 30, a first camera (first imaging device) 32, and a data transfer unit 33. I have it. The first camera 32 and the data transfer unit 33 are housed in the main body unit 30. It should be noted that at least the vicinity of the pen tip 31 of the main body portion 30 is formed of a translucent member, more preferably a transparent member, so that the first camera 32 can capture an image.

The pen tip 31 is provided so as to project from the main body section 30. The outer shape of the pen tip 31 is smaller than the grid shape of the structured light pattern LP.
In the present embodiment, a detection element 31a that determines the contact of the pen tip 31 with the screen SC is provided at the root of the pen tip 31. As such a detection element 31a, for example, a pressure sensor or the like can be used. The detection element 31a is electrically connected to the first camera 32. The method for determining the contact of the pen tip 31 with the screen SC is not limited to pressure detection.

The first camera 32 has the same configuration as the second camera 21, and captures the structured light pattern LP displayed on the screen SC designated by the pen tip 31.
When the detection element 31a determines that the pen tip 31 is in contact with the screen SC, the first camera 32 captures an image by being supplied with power from a power source (not shown). Therefore, the electronic pen P drives the first camera 32 only when the pen tip 31 comes into contact with the screen SC, so that power consumption can be reduced.

FIG. 3 is an enlarged view of a main part of the electronic pen for pointing the screen. FIG. 3 shows the main configuration of the first camera 32 provided near the pen tip 31 of the electronic pen P.
As shown in FIG. 3, the electronic pen P uses a first camera 32 provided near the pen tip 31 to acquire a peripheral image of the pen tip 31 that indicates the screen SC. More specifically, the first camera 32 images the area designated by the pen tip 31 and its peripheral area in the structured light pattern LP displayed on the screen SC, and acquires the area as a later-described first image G1. ..

As shown in FIG. 3, the first camera 32 has an image sensor 32a and a lens 32b. In the present embodiment, the central axis 31C of the pen tip 31 of the electronic pen P is parallel to the optical axis 32C of the first camera 32. More specifically, the central axis 31C coincides with the optical axis 32C. Since the optical axis 32C of the first camera 32 passes through the center of the image sensor 32a, the pen tip 31 of the electronic pen P is located at the center of the image acquired by the image sensor 32a. Since the pen tip 31 is located at the center of the image in this way, the position of the pen tip 31 can be easily specified.

In the present embodiment, the first camera 32 has an angle of view at which at least four different projection points among the plurality of projection points of the structured light pattern LP projected on the screen SC can be recognized. Here, the four different projection points refer to four points that are not aligned on a straight line in the structured light pattern LP in the form of a grid.

The first camera 32 acquires the first image G1 including at least four different points among the plurality of projection points of the structured light pattern LP located around the pen tip 31 of the electronic pen P that points the screen SC. (First image acquisition step).

FIG. 4 is a schematic diagram showing a first image taken by the first camera. In FIG. 4, the four projection points in the first image G1 are indicated by reference numerals LP1, LP2, LP3, and LP4, respectively. It should be noted that reference numerals L1 and L2 in FIG. 4 correspond to two points at the four corners of the structured light pattern LP shown in FIG.
As described above, in the electronic pen P, since the pen tip 31 is located at the center of the image sensor 32a, the central portion G1c in the first image G1 is missing as shown in FIG.

The image data of the first image G1 acquired by the first camera 32 is transmitted to the data transfer unit 33. The data transfer unit 33 transmits the data regarding the first image G1 to the image processing unit 22 by wireless communication such as Bluetooth (registered trademark) or wireless LAN (Local Area Network).

The image processing unit 22 is a CPU (Central Processing Unit), a RAM (Random Access Memory) used for temporary storage of various data, a mask ROM (Read Only Memory), and a flash memory. , FeRAM (Ferroelectric Random Access Memory: ferroelectric memory) and the like, and a processing device for image analysis (neither is shown). The image processing unit 22 is electrically connected to a control unit that controls driving of the projector 3.

As shown in FIG. 4, the structured light pattern LP, which is originally rectangular, is in a deformed state in the first image G1. This is because the first image G1 was taken by the first camera 32 provided on the electronic pen P from a direction different from the projection direction of the structured light pattern LP.

FIG. 5 is a schematic diagram showing a second image captured by the second camera.
As shown in FIG. 5, the structured light pattern LP is not deformed in the projection direction of the structured light pattern LP, that is, in the second image G2 captured by the second camera 21 that images the screen SC from the front direction. It has a rectangular shape. Therefore, the position information calculated from the structured light pattern LP in the deformed state included in the first image G1 shown in FIG. 4 is the unstructured structured light included in the second image G2 shown in FIG. It is different from the position information on the screen SC acquired from the pattern LP.

In the present embodiment, the image processing unit 22 causes the coordinates of the four projection points LP1, LP2, LP3, LP4 on the screen SC acquired from the second image G2 and the three projection points acquired from the first image G1. A projective transformation matrix is calculated using the coordinates of LP1, LP2, LP3, and LP4. The projective transformation matrix has an undeformed structure included in the second image G2 obtained by capturing the screen SC from the front direction of the coordinate system defined by the deformed structured light pattern LP included in the deformed first image G1. It can be converted into the coordinate system on the screen SC which is defined by the converted light pattern LP.

In the electronic pen P of the present embodiment, the pen tip 31 is located at the central portion G1c of the first image G1. Therefore, the image processing unit 22 can convert the coordinates of the central portion G1c of the first image G1 into the coordinates of the pen tip 31 on the screen SC shown in FIG. 5 by converting the coordinates with the projective conversion matrix.

As described above, in the position detection device 20 of the present embodiment, the image processing unit 22 displays on the screen SC indicated by the pen tip 31 of the electronic pen P based on the above-described first image G1 and second image G2. The coordinates can be accurately detected (detection step).

Next, in the image display system 1 according to the present embodiment, for example, a case where the position of a finger (second indicator) that points on the screen SC is detected will be described. Unlike the pen tip 31 of the electronic pen P, the human finger is larger than the size of the lattice portion of the structured light pattern LP. Therefore, the finger pointing on the screen SC overlaps the place A1 (see FIG. 2) where the structured light pattern LP is projected.

In the position detection unit 2 and the position detection device 20 of the present embodiment, the entire display area of the structured light pattern LP on the screen SC pointed by the finger is imaged by the second camera 21 (third image). , Position information (coordinates) on the screen SC designated by the finger is detected. In the image captured by the second camera 21, the structured light pattern LP has a part of the structured light pattern LP projected on the finger in front of the screen SC, so that the shape of the region designated by the finger is changed. Is changing.

As described above, since the second camera 21 is calibrated in advance for the structured light pattern LP, the image processing unit 22 captures the entire display area of the structured light pattern LP on the screen SC with the second camera 21. The position information (coordinates) of the finger on the screen SC can be detected by comparing the image thus obtained with the image deformed by the finger as described above. According to the position detection unit 2 of this embodiment, the position of the finger pointing on the screen SC can be detected.

The image processing unit 22 outputs the position information of the electronic pen P or the finger on the screen SC detected as described above to the control unit of the projector 3.

According to the position detection unit 2 and the position detection device 20 of the present embodiment, since the electronic pen P having the first camera 32 is provided, it is possible to indicate on the screen SC with a thin pointing body such as the pen tip 31. However, the pointing position can be detected with high accuracy. Further, the position detection unit 2 and the position detection device 20 can detect the position of the finger pointing on the screen SC by using the image captured by the second camera. That is, according to the position detection unit 2 and the position detection device 20 of the present embodiment, both rough input detection using the finger on the screen SC and high-precision input detection using the electronic pen P on the screen SC are compatible. be able to.

Therefore, according to the image display system 1 of the present embodiment including the position detection unit 2 and the position detection device 20, by detecting the position of the electronic pen P or the finger on the screen SC with high accuracy, the image is displayed on the screen SC. The presentation can be smoothly progressed by switching the images and enlarging or reducing the images. Further, an image or a character drawn on the screen SC may be displayed on the screen SC by the movement of the presenter's electronic pen P or finger.

(Modification)
Next, an image display system according to the modified example will be described. The configuration of the electronic pen is different between this modification and the above embodiment. Therefore, the configuration of the electronic pen will be mainly described below. In the following description, the same components and members as those in the above embodiment are designated by the same reference numerals, and detailed description thereof will be omitted.

FIG. 6 is a diagram showing the configuration of the electronic pen of this modification.
As shown in FIG. 6, the electronic pen P1 includes a main body unit 30, a pen tip 31, a first camera 32, an image processing unit 42, and a data transfer unit 43. The first camera 32, the image processing unit 42, and the data transfer unit 43 are housed in the main body unit 30. That is, in this modification, the image processing unit 42 is provided in the electronic pen P1. The image processing unit 42 has the same configuration as the image processing unit 22 of the above embodiment.

The image processing unit 42 is electrically connected to the first camera 32, and the image data of the first image G1 acquired by the first camera 32 is output to the image processing unit 42. In this modification, the second camera 21 is connected to a data transfer unit (not shown), and transfers the generated second image G2 to the image processing unit 42 provided in the electronic pen P1.

The image processing unit 42 calculates the position information on the screen SC designated by the pen tip 31 of the electronic pen P1 based on the first image G1 and the second image G2 transmitted from the first camera 32 and the second camera 21. be able to. The electronic pen P1 of the present modification outputs the position information of the pen tip 31 calculated by the image processing unit 42 from the data transfer unit 43 to the control unit of the projector 3. The data transfer unit 43 transfers data to the control unit of the projector 3 by wireless communication.

According to the electronic pen P1 of the present modification, since only the coordinate data, which is the position information of the electronic pen P1 calculated by the image processing unit 42, is transferred from the data transfer unit 43 to the projector 3, the image data is transferred. The data transfer amount can be reduced as compared with the configuration of the embodiment. Therefore, according to the image display system using the electronic pen P1 of the present modification, more real-time operation can be performed.

In the above-described embodiment, the structured light pattern LP has a grid shape as an example, but the shape of the structured light pattern LP is not limited to the grid shape as long as it can be used for touch recognition. The pattern may be such that (dots) are evenly scattered.

Further, in the above embodiment, the central axis 31C of the pen tip 31 of the electronic pen P coincides with the optical axis 32C of the first camera 32, but the central axis 31C may be parallel to the optical axis 32C. When the central axis 31C and the optical axis 32C are parallel to each other, the pen tip 31 is always located at a predetermined position shifted from the center of the first image G1 by a predetermined amount. Therefore, the image processing unit 22 can convert the coordinates of the predetermined position of the first image G1 into the coordinates of the pen tip 31 on the screen SC by the projective conversion matrix.

Further, for example, an electronic pen having a pen tip containing ink or the like may be used. In this way, characters and pictures can be drawn on the display surface with the electronic pen. Further, by detecting the position of the pen tip when drawing a character or a picture, the input information of the image or the character may be acquired to create the document data.

Further, in the above embodiment, the case where the light emitted from the light source is shaped by the diffractive optical element and the structured light pattern LP is displayed on the screen SC as the structured light display unit 10 has been described as an example. You may comprise a structured light display part using the projector which projects the structured light pattern LP which consists of light on the screen SC.

Further, although the screen SC is given as an example of the display surface in the above-described embodiment, the display surface is not limited to this, and may be, for example, a wall, glass, a whiteboard, or the like. The display surface does not have to be flat, and may be a spherical surface or an uneven surface. Further, the indicator for indicating the display surface is not limited to the electronic pen P or the finger, and may be, for example, a pointing stick, a magnet attracted to the display surface, or the like. Further, the use of the image display system 1 is not limited to presentations, and can be used for various purposes such as store guides for department stores and the like, introduction/search of products handled, and the like.

DESCRIPTION OF SYMBOLS 1... Image display system, 2... Position detection unit, 3... Projector (image display device), 10... Structured light display part, 20... Position detection device, 21... 2nd camera (2nd imaging device), 22... Image Processing circuit (image processing unit), 22, 42... Image processing unit, 31... Pen tip (instruction portion), 31C... Central axis, 32C... Optical axis, 32... First camera (first imaging device), 33, 43 ... data transfer unit, F... finger, G1... first image, G2... second image, LAN... wireless, LP... structured light pattern, LP1... projection point, P, P1... electronic pen (first indicator) , SC... Screen (display surface).

Claims (9)

A first indicator for indicating a display surface on which the structured light pattern is displayed;
A first imaging device provided on the first indicator, for imaging the structured light pattern displayed on the display surface;
A second imaging device for imaging the structured light pattern displayed on the display surface;
An image processing unit,
The first imaging device acquires a first image of a region including at least four different points among a plurality of projection points of the structured light pattern located around the first indicator,
The second image capturing device acquires a second image capturing the entire display area of the structured light pattern on the display surface indicated by the first indicator,
The position detecting device, wherein the image processing unit detects position information on the display surface indicated by the first pointer based on the first image and the second image. When the second indicator points to the display surface, the image processing unit performs the second imaging on the entire display area of the structured light pattern on the display surface instructed by the second pointer. The position detection device according to claim 1, wherein position information on the display surface indicated by the second indicator is detected by using a third image captured by the device. The position detecting device according to claim 1, wherein a central axis of a pointing portion of the first pointing body is parallel to an optical axis of the first imaging device. The position detection device according to claim 3, wherein the central axis coincides with the optical axis. A position detecting device according to any one of claims 1 to 4,
And a structured light display section that displays the structured light pattern toward the display surface. A position detection unit according to claim 5;
An image display system comprising: an image display device that displays an image on the display surface. The image processing unit is provided in the first indicator,
The first indicator transmits the position information to the image display device,
The image display system according to claim 6, wherein the image display device controls an image displayed on the display surface based on the position information transmitted from the first indicator. The image display system according to claim 7, wherein the first indicator further includes a data transfer unit that transfers data to the image display device by wireless communication. An image of an area including at least four different projection points of the plurality of projection points of the structured light pattern located around the first indicator that indicates the display surface on which the structured light pattern is displayed is imaged. A first image acquisition step of acquiring a first image;
A second image acquisition step of acquiring a second image of the entire display area of the structured light pattern on the display surface indicated by the first indicator,
A position detecting method for detecting position information on the display surface indicated by the first indicator based on the first image and the second image.

Images