JP5201096B2 - Interactive operation device - Google Patents

Description

The present invention relates to a system, a program, and a recording medium for operating information projected by a viewer with a fingertip.
The present invention is particularly useful when the content projected on the projection table is projected on the projection table by the viewer giving an instruction with a fingertip.

Recently, projectors that are remarkably widespread can project a large amount of content on a wall surface or a display board (hereinafter referred to as a “projection stand”), and thus are increasingly used as advertising media or communication tools.
A content viewer can interact with the projected content interactively at a museum, for example, to obtain more art information and enjoy a full art appreciation.
Here, a method for interactively operating the displayed content will be described.

<< A. Cursor operation
This is a method of interactively operating content viewed by a viewer by operating a cursor using a pointing device typified by a mouse or a trackball.
Alternatively, it is a method of interactively manipulating content using a racer pointer, depending on how the viewer moves when irradiating a laser spot light.
For example, Patent Document 1 discloses a technique for seamlessly dragging and dropping a digital object between a computer display screen and a virtual extended display screen of a desk or wall using a pointing device. .
Patent Document 2 discloses a technique for operating a presentation screen such as screen switching, line drawing, and screen partial enlargement by instructing the screen on the projection screen with the light of a laser pointer. Has been.
<< B. Touch panel instructions >>
In this method, when a viewer touches a touch panel displaying content, the content is interactively operated according to the touched position.
For example, Patent Document 3 discloses a technique for selecting an area by changing the area size according to the length of time for which a point on the display is kept pressed with a fingertip.

JP 2001-175374 A (paragraph number “0125” to paragraph number “0135”, FIGS. 4 to 7) JP 2001-125738 A (paragraph “0065”, paragraph “0067” to paragraph “0069”, FIG. 6) Japanese Patent Laying-Open No. 2006-113715 (paragraph “0015” to “0016”, FIG. 4)

However, << A. In the “cursor operation” method, a viewer cannot perform an interactive operation unless using a pointing device such as a mouse.
In addition, << B. In the method of “touch panel instruction”, a device for detecting that a viewer has touched a hand must be provided in a device for displaying content.

The present invention has been made in order to solve the above-described points, and the object of the present invention is that the viewer can project and display without using a tool and without setting a device on the projection table. for the content, it is to provide a dialogue operation equipment.

This invention solves the said subject by the means as described in each following aspect.
That is, the first invention of the present invention is :
A projector for projecting an image based on the content projection screen data to the projection stage, and an infrared camera to generate a captured image by capturing the projection stage, the interaction device is connected,
And a content storage area for storing the content configuration data,
A display screen holding area for holding the content projection screen data,
Storage means comprising :
Interprets the content member data, a display viewing means for creating the content projection screen data,
Captured image acquisition means for acquiring the captured image from the infrared camera;
And the fingertip detection means for detecting the fingertip of the viewer from each frame of the captured image,
And the fingertip position calculating means for calculating the coordinates of the fingertip,
And calculates the movement distance of the finger using the coordinates of the fingertip of each frame consecutive, when the moving distance during a predetermined time is determined to zero, the fingertip stationary detection information including coordinates of the fingertip Fingertip stationary detection means for creating
Based on the fingertip stationary detection information, and generates the input event information, an input emulation means to pass to the display viewing means,
With
The display browsing means receives the input event information delivered from the input emulation means, updates the content projection screen data as appropriate ,
The fingertip detecting means is one of a plurality of sides constituting the outline of the content projection screen, and one of the sides in contact with the captured image of the hand is a bottom side, a left side, a right side, or an upper side. It is determined whether or not it is a side, and when the determined side is the bottom side, the top of the hand is determined as the fingertip, and when the determined side is the left side, the rightmost part of the hand is the fingertip. determined, the determined sides is determined that the fingertip of the leftmost portion of the hand when it is right, it is determined that the fingertip of the bottom of the hand when the determined sides are the upper side,
This is an interactive operation device characterized by the above.

In this way, it is possible to detect the movement and position of the viewer's fingertip, generate input event information, and process and display the content being projected and viewed.

According to a second aspect of the present invention, the fingertip detection means according to the first aspect is
The fingertip detection means includes
If the side of the content projection screen that touches the handprint image is not found, the straight line that passes through the maximum brightness point is parallel to the side of the hand that has been moved and extended based on the maximum brightness point of the distance image. Of the contours of the handprint image divided into two, without searching for the contour on the side where the hand has moved, the farthest part is the fingertip.
This is an interactive operation device characterized by the above.

According to the present invention,
(1) An information operation can be performed on the projected content without providing an information operation instruction detection device on the projection table.
(2) The viewer can give an instruction to operate the information with a fingertip without using a tool.
Therefore, according to the present invention, there is an effect that the viewer can easily operate the information simply by pointing the projected content with the fingertip.

FIG. 1 is a diagram illustrating an overview of a fingertip operation type information providing system 1 according to an embodiment of the present invention. Example 1 FIG. 2 is an example of a content projection screen based on a dialogue instruction. FIG. 3 is a diagram for explaining a rough operation procedure of the operation and processing of the fingertip operation type information providing system 1. FIG. 4 is an example of fingertip detection. FIG. 5 is a diagram for explaining the detailed flow of (1) initial setting processing. FIG. 6 is a diagram for explaining an example of setting the projection range. FIG. 7 is a diagram for explaining the detailed flow of (3) fingertip detection processing. FIG. 8 is a diagram for explaining an example of the binarized image creation process. FIG. 9 illustrates the method of Otsu. FIG. 10 is a flowchart of (3-6) fingertip determination processing. FIG. 11 is a detailed configuration diagram of the information processing apparatus 100. FIG. 12 is a diagram for explaining the reflection of a shadow. (Example 2) FIG. 13 is a diagram for explaining processing for ignoring the fine movement of the fingertip. (Example 3) FIG. 14 is a diagram for explaining fingertip detection. Example 4

Hereinafter, embodiments of the present invention will be described in more detail with reference to the drawings.

FIG. 1 is a diagram illustrating an overview of a fingertip operation type content providing system 1 according to an embodiment of the present invention.
The fingertip operation type content providing system 1 includes a projection table 400, an infrared light source 500, a projector 300 (= projector), and an interactive operation device 100.
The interactive operation device 100 is an existing personal computer and has a dedicated program described later.
The interactive operation device 100 connects the infrared camera 106 and the projector 300.
The infrared light source 500 is an LED light source device that irradiates the projection table 400 with infrared light.
The infrared camera 106 is an electronic camera that captures an image reflected by infrared light.
The projector 300 is a projector that projects an image based on the content projection screen data created by the interactive operation device 100 onto the projection table 400.

Here, the reason for using the infrared light source 500 and the infrared camera 106 is that the image of the fingertip of the viewer 900 irradiated with infrared light is captured, but the projector 300 projects onto the projection table 400 with visible light. This is because an image (= content projection screen) is not captured.

FIG. 2 is an example of the content projection screen 403.
FIG. 2A shows an example of a content projection screen 403 displaying content and commentary.
The meaning of the exemplified content projection screen is that the explanation text of the content designated by the fingertip is displayed superimposed on the content.
FIG. 2B is an example of a content projection screen 403 including a partially enlarged image of content.
The meaning of the exemplified content projection screen is that an image obtained by enlarging the content portion on which the fingertip is placed is displayed superimposed on the content.

Next, a rough operation and information processing procedure of the fingertip operation type information providing system 1 will be described with reference to FIG.
(1) << Initial Setting Process >> The system administrator uses the captured image of the projection table generated by the infrared camera of the interactive operation device 100 by capturing the projection table 400 to perform a perspective transformation process on the captured image. Initial setting processing for setting a transformation matrix relational expression, a background image of a captured image, and the like is performed. (Details will be described later)
(2) << Content Projection Process >> The interactive operation device 100 projects the content projection screen 103 to be viewed by the viewer on the projection table 400 using the projector 300.
(3) << Interactive Operation >> The viewer performs an interactive operation on the content projection screen 403 projected on the projection table 400 using a fingertip.
(4) << Fingertip Detection Processing >> The dialog operation device 100 captures the fingertip of the viewer who performs the dialog operation with the infrared camera 106 and detects the position and operation of the viewer's fingertip from the captured image. Here, in order to detect the fingertip movement of the viewer, the fingertip position of each frame of the captured image may be compared. That is, it is only necessary to measure the amount of change in the fingertip position of each frame to determine whether the frame is stationary or moving.
(5) << Projection processing of interactively operated content >> The interactive operation device 100 uses the detected fingertip motion as a fingertip motion determination condition (for example, fingertip still motion if the fingertip rest motion time is 0.5 seconds or longer). And performing an interactive operation in accordance with the fingertip movement determination condition. For example, the result of the operation desired by the viewer, such as the explanation of the selected content or the partial enlarged image of the selected content, is displayed on the projector. 300 is used to project onto the projection table 400.
Here, the dialogue operation apparatus 100 may superimpose a cursor on a position corresponding to the fingertip of the content projection screen 403.
(6) << Content browsing >> The viewer browses the content projection screen 403 projected on the projection table 400.

The viewer's fingertip will be described with reference to FIG.
FIG. 4A is an example for explaining the fingertip of the hand extending from the bottom side 1031 of the captured image display screen 103 of the hand.
An example in which a captured image of a hand extending from the bottom of the display screen is displayed on the screen 103 is shown.
In the case of a hand extending from the base 1031, the meaning of the photographed image shown as an example indicates that the uppermost portion 1035 of the hand is detected as a fingertip.

FIG. 4B illustrates an example of a fingertip of a hand extending from the left side 1032 of the content projection screen.
An example in which a captured image of a hand extending from the left side of the display screen is displayed on the screen 103 is shown.
In the case of a hand extending from the left side 1032, the meaning of the illustrated fingertip indicates that the rightmost portion 1036 of the hand is detected as the fingertip.

5 and 6 are used to explain the detailed flow of (1) << Initial Setting Process >>. FIG. 5 is a diagram illustrating the detailed flow of (1) << Initial Setting Process >>.
(1-1) << Projection range instruction >> The system administrator uses four captured vertices of a quadrangle as a projection range, using a captured image of the projection table generated by the infrared camera of the interactive operation device 100 capturing the projection table 400. Enter the coordinates. For example, the system administrator displays a photographed image of the projection table on the display unit of the interactive operation device 100 and inputs the four vertex coordinates of the projection range by operating, for example, a cross cursor with a keyboard.

ここで、ｘ_p は、撮影画像のＸ座標値である。ｙ_p は、撮影画像のＹ座標値である。
ｈは、カメラ原点と画像平面の距離を変数とする値である。
Ｘ_mは、コンテンツ投影画面のＸ座標値である。Ｙ_m は、コンテンツ投影画面のＹ座標値である。
Ｃ_ijは、二次元画像透視変換関係式の行列要素である。ここで、ｉ，ｊ＝１、２，３である。

ここで、Ｃ_a 、Ｃ_b 、・・、Ｃ_h は、算出した撮影画像透視変換関係式の行列要素である。 (1-2) << Calculation of Coordinate Transformation Matrix >> The interactive operation apparatus 100 accepts a projection range instruction input by the system administrator, and has already mathematically converted a coordinate transformation matrix relational expression (for example, a coordinate transformation matrix relational expression). For the following mathematical formula 1), the four vertex coordinate values [X _pk , Y _pk ] (k = 1, 2, 3, 4) of the received projection range instruction and the four standards of the content projection screen Substituting the vertex coordinate values [X _mk , Y _mk ] (k = 1, 2, 3, 4) (for example, [0, 0] [0, 1024] [768, 0] [768, 1024]) , Twelve equations are solved and these are solved to calculate eight matrix elements C _ij (i, j = 1, 2, 3) of the coordinate transformation matrix, and the captured image perspective transformation matrix relational expression (the following formula) Equation 2) is derived. Photographed image perspective transformation matrix equation is a matrix equation which coordinates of the photographed image [X _p, Y _p] with coordinates [X _m, Y _m] Content projection screen and perspective transformation.

Here, _xp is the X coordinate value of the captured image. y _p is the Y-coordinate value of the captured image.
h is a value having the distance between the camera origin and the image plane as a variable.
X _m is the X coordinate value of the content projection screen. Y _m is the Y coordinate value of the content projection screen.
C _ij is a matrix element of the two-dimensional image perspective transformation relational expression. Here, i, j = 1, 2, and 3.

Here, C _a , C _b ,..., _Ch are matrix elements of the calculated captured image perspective transformation relational expression.

The captured image perspective transformation relational expression can be calculated using the opencv perspective transformation matrix computation function cvGetPerspectiveTransform. Here, opencv is an open source computer vision library developed and released by Intel (registered trademark).

Here, a display screen used in the operation of “projection range instruction” will be described with reference to FIG.
FIG. 6A illustrates a projection range instruction screen 1031. On the exemplified projection range instruction screen 1031, a photographed image of the projection base 400 photographed by the infrared camera 104 of the interactive operation device 100 is displayed.
In addition, the projection range instruction screen 1031 shows a state where four vertices p, q, r, and s of the projection range are input.
At this time, the vertex angles of the four vertices p, q, r, and s in the projection range are not necessarily a right angle.
FIG. 6B illustrates an image display screen 1032 obtained by perspective-transforming the projection range of the captured image.
The illustrated perspective transformation image display screen 1032 is obtained by perspective transformation of the projection range of the photographic image using the photographic image perspective transformation relational expression.
Here, the four vertices P, Q, R, and S (= the coordinates of the perspective transformation image display screen 1033 correspond to [0, 0] [0, 1024] [768, 0] [768, 1024]. ) Is a point in which the four vertices p, q, r, and s of the projection range are perspective-transformed. At this time, the vertex angles of the four vertices P, Q, R, and S of the rectangular area are right angles.

Returning to FIG. 5, the detailed flow of (1) << Initial Setting Process >> will be continued.
(1-3) << Background Image Extraction >> The interactive operation device 100 extracts an image in the projection range from the photographed image of the projection table and sets it as a background image. When this background image is extracted, the image in the projection range is perspective-transformed into the coordinates of the content projection screen using the derived captured image perspective transformation matrix relational expression (= Formula 2).

The detailed flow of (4) << fingertip detection process >> will be described with reference to FIGS.
FIG. 7 is a diagram for explaining the detailed flow of (4) << fingertip detection process >>.
The fingertip detection process is performed for each frame of the captured image.
(4-1) << Perspective Conversion Processing of Captured Image >> The dialogue operation apparatus 100 uses the derived captured image perspective transformation matrix relational expression (= Formula 2) to convert the captured image of the hand into an image in the coordinate system of the content screen. Transform to perspective.
(4-2) << Background Removal >> The interactive operation apparatus 100 subtracts the pixel value of the background image obtained by perspective transformation from the pixel value of the photographed hand image obtained by perspective transformation to obtain the background image of the photographed image obtained by perspective transformation. Remove it.
(4-3) << Binarization Processing of Captured Image >> The interactive operation apparatus 100 applies a method of Otsu to a perspective-transformed captured image that has been subjected to background image removal processing, calculates a threshold value, and performs binary processing The binarized image is generated by performing the binarization process.

Here, the method of Otsu will be described with reference to FIG. Otsu's method is to divide each pixel value into two regions with a temporary threshold value t for the pixel value histogram of the grayscale image, and to distribute the variance σ ² _a in each region and between the two regions A function for calculating the variance σ ² _b and calculating the variance within the region and the variance ratio between the regions (function of the temporary threshold t) W (t) = σ ² _a / σ ² _b is the minimum This is a method using t as a threshold value.
In the image binarization process using the Otsu method, a binarization threshold value can be calculated based on the Otsu method and a binarized image can be obtained using the OpenCV binarization function cvThreshold.

In the binarization process, using the method of Otsu is one example. For example, other values may be used as threshold values, and a plurality of binarized images may be generated using a plurality of threshold values.

Returning to FIG. 7, the detailed flow of (4) << Fingertip detection process >> will be continued.
(4-4) << Distance Image Creation Processing >> The interactive operation device 100 obtains the shortest distance from the white pixel of the binarized image to the black pixel, and is a distance configured by pixels having pixel values associated with the distance. Create an image and hold the maximum pixel value of the distance image.
(4-5) << Outline Pixel Detection Process >> The interactive operation device 100 scans the black pixel area of the binarized image from the end, and extracts black pixels using the outline extraction algorithm using four connected pixels. Thus, the extracted black pixel is set as a contour pixel.
Here, the contour extraction algorithm using four connected pixels is a scan of a binarized image from the end, with the detected untracked black pixel as the starting contour pixel, and the four connected pixels centered on this pixel as an inverse. A black pixel is detected by tracking clockwise and is used as a contour pixel, and the black pixel is detected by 4-connected pixel tracking with this pixel as the next center, and the process is repeated until the start point is returned. The contour extraction algorithm may use 8-connected pixels.
(4-6) << Hand-Shape Image Extraction Processing >> The dialogue operation apparatus 100 compares the luminance of the pixel of the distance image (= the region surrounded by the contour pixel) with the retained maximum luminance value and agrees. The region is extracted as a handprint image of the viewer.

Here, an example of “fingertip detection processing” will be described with reference to FIG.
FIG. 8A illustrates a binarized image 1961.
The illustrated binarized image 1961 is an image including a large white pixel region 19611 and a small white pixel region 19612.
FIG. 8B illustrates a distance image 1962.
The illustrated distance image 1962 is an image including a distance image 19621 corresponding to the large white pixel region 19611 and a distance image 19622 corresponding to the small white pixel region 19612. Here, the distance image 19621 corresponding to the large white pixel region 19611 is an image having the pixel 1038 having the maximum pixel value.
FIG. 8C illustrates an outline pixel.
The illustrated contour pixel is a pixel that forms a large white pixel region contour 19631 and a small white pixel region contour 19632. Here, the outline 19631 of the white pixel region has a pixel 10381 corresponding to the pixel 1038 having the maximum pixel value.
FIG. 8D illustrates the maximum contour portion.
The exemplified maximum contour portion is a white pixel region contour portion 19631 corresponding to the distance image 19621 having the maximum pixel value.

Returning to FIG. 7, the detailed flow of (4) << Fingertip detection process >> will be continued.
(4-7) << Fingertip Determination Process >> The interactive operation device 100 performs a fingertip determination process using the viewer's handprint image to determine the fingertip. (Details will be described later)
(4-8) << Fingertip Position Calculation >> The interactive operation device 100 calculates the fingertip coordinates of successive frames.
(4-9) << Fingertip Stillness Detection Process >> The interactive operation device 100 compares the fingertip coordinates of successive frames to calculate the fingertip movement distance, and compares it with the motion determination condition that defines the fingertip stillness determination time. It is determined whether the fingertip movement distance during the fingertip stillness determination time is zero. When it is determined that the fingertip moving distance during the fingertip stillness determination time is zero, fingertip stillness detection information including fingertip coordinates is created.

FIG. 10 is a flowchart of the fingertip determination process.
(1) The interactive operation device 100 determines whether or not the side (= the captured image frame) of the content projection screen with which the captured image of the hand is in contact is the base 1031. If the side with which the hand image is in contact is the lower side 1031, the process proceeds to the next step.
If the side with which the hand image is in contact is not the base 1031, the process proceeds to step S 120. (Step S110)
(2) The dialogue operation apparatus 100 determines that the top 1035 of the hand is the fingertip, holds “side with hand = bottom”, and ends. (Step S115)
(3) The interactive operation device 100 determines whether or not the side of the content projection screen that is in contact with the captured image of the hand is the left side 1032. If the side with which the hand image is in contact is the left side 1032, the process proceeds to the next step.
If the side with which the hand image is in contact is not the left side 1032, the process proceeds to step S 130. (Step S120)
(4) The dialogue operation apparatus 100 determines that the rightmost part 1036 of the hand is a fingertip, holds “side with hand = left side”, and ends. (Step S125)
(5) The interactive operation device 100 determines whether or not the side of the content projection screen that is in contact with the captured image of the hand is the right side. If the side on which the hand image is in contact is the right side, the process proceeds to the next step.
If the side on which the captured image of the hand is in contact is not the right side, the process proceeds to step S140. (Step S130)
(6) The interactive operation device 100 determines that the leftmost part of the hand is the fingertip, holds “side with hand = right side”, and ends. (Step S135)
(7) The interactive operation device 100 determines whether or not the side of the content projection screen that is in contact with the captured image of the hand is the upper side. If the side with which the hand image is in contact is the upper side, the process proceeds to the next step.
If the side on which the hand image is in contact is not the upper side, the process proceeds to step S160. (Step S140)
(6) The dialogue operation apparatus 100 determines that the lowermost part of the hand is the fingertip, holds “side where the hand touches = upper side”, and ends. (Step S145)
(7) The position of the fingertip is determined as follows according to the value of the side that the held bill touches.
・ If “side touched by hand = bottom side”, the top of the hand is determined as the fingertip.
-If "side with hand = left side", the rightmost part of the hand is determined as the fingertip.
・ If “side touched by hand = right side”, the leftmost part of the hand is determined as the fingertip.
-If "side touched by hand = upper side", the lowermost part of the hand is determined as the fingertip.
finish.

FIG. 11 is a detailed configuration diagram of the interactive operation device 100.
The interactive operation apparatus 100 includes a CPU 101, a display unit 102, a keyboard 103, an infrared camera unit 106, a projector connection unit 107 (= projector connection means), a network communication unit 108, and a storage unit 109. And a program.
The CPU 101, the display unit 102, the keyboard unit 103, the camera unit 106, the projector connection unit 107, the network communication unit 108, and the storage unit 109 are connected by BUS199.

The CPU 101 is a central processing unit.
The display unit 102 is a liquid crystal display device or an organic EL display device.
The projector connection means 107 is a projector adapter.
The network communication unit 108 is a LAN adapter.

The infrared camera unit 106 is an infrared electronic camera.
The infrared camera unit 106 images the hand of the viewer 900 irradiated with the infrared light from the infrared light source 500 and generates a captured image of the hand.

The storage unit 109 is a semiconductor memory or a magnetic memory.
The storage unit 109 includes a display screen holding area 1091, an initial setting information storage area 1092, and a content storage area 1093, and stores the operating system 185 and a dedicated program.
The display screen holding area 1091 holds the content projection screen data 198.
The initial setting information storage area 1092 stores a captured image perspective transformation relational expression 193 and background image data 195.
The content storage area 1093 stores content configuration data 197 and content material data 192 for creating content projection screen data.
The content configuration data 197 is data representing the configuration of the entire content (for example, how many pages are formed) and how to use the content material data (for example, which material is disposed at which position on which page). It is.
The content material data 192 is photo or text data.

The operating system 185 manages and controls the hardware (for example, the display unit 102, the keyboard 103, the projector connection unit 107, the network communication unit 108, the storage unit 109, etc.) of the interactive operation device 100, It is basic software that provides services (such as dedicated programs) that allow these hardware to be used.
The operating system 185 has a mouse function library 191 including a mouse left click emulation function and a cursor display emulation function.
The operating system 185 receives the mouse left click mouse event (= input event information) issued by the mouse left click emulation function, and stores it in the temporary storage area. At this time, the active process reads the held mouse event with reference to the temporary storage area, and processes the process according to the event.

In addition, an initial setting unit 110, a fingertip detection unit 130, a fingertip position calculation unit 140, a fingertip stationary detection unit 150, an input emulation module 155, a display screen creation unit 160, and a projection display unit 170 are provided. . Each of these means is realized by each dedicated program and functions by the CPU 101 interpreting and executing the dedicated program.

The initial setting means 110 includes a projection range instruction function that receives an input of a projection range of a captured image of a projection table generated by an infrared camera shooting the projection table 400;
A coordinate transformation matrix calculation function for calculating a photographed image perspective transformation matrix relational expression representing a perspective transformation relationship between the coordinates of the photographed image and the coordinates of the content projection screen;
A background image extraction function that extracts an image in the projection range from a photographed image of the projection table and uses it as a background image.
Details of the initial setting means are described in the section describing the detailed flow of (1) << Initial setting process >> in FIG.

The fingertip detection unit 130 includes a captured image perspective conversion function that perspectively converts a captured image of the viewer's hand created by the imaging unit 106 into a perspective converted captured image;
A background removal function for removing a background image from a perspective transformation photographed image;
A captured image binarization function for generating a binarized image from a perspective transformation captured image with the background image removed;
A distance image creation function for creating a distance image from a binarized image;
A contour pixel detection function for detecting a contour pixel from a perspective transformation captured image from which a background image has been removed;
A maximum contour extraction function that extracts a handprint image using a distance image and a contour pixel;
A fingertip determination function for determining a fingertip using information on the side of the content projection screen that the handprint image touches;
Is provided.

The details of the fingertip detection means 130 are described in the section describing the detailed flow of ( 4 ) << Fingertip detection processing >> in FIG.

The fingertip position calculation means 140 calculates the fingertip coordinates.
The fingertip stillness detection means 150 calculates the fingertip movement distance using the fingertip coordinates of successive frames, and if the fingertip movement distance during a predetermined time is determined to be zero, the fingertip stillness detection information including the fingertip coordinates Create

The input emulation module 155 generates input event information based on the fingertip stillness detection information and delivers it to the display browsing means.
In particular, the input emulation module 155 accepts a fingertip stationary detection information, call the mouse left click emulation function, a function of Ru to issue the mouse event.

The display browsing unit 160 interprets the content configuration data 196 and creates content projection screen data 198 using the content material data 192.
In addition, the display browsing unit 160 receives a mouse event delivered through the operating system 185, and updates the content projection screen data as appropriate. Specifically, the display browsing unit 160 reads out the mouse event held in the temporary storage area by the operating system 185, and appropriately updates the content projection screen data. (The input emulation means in claim 1 is realized by the cooperative operation of the input emulation module 155 and the operating system 185).

Next, prevention of erroneous extraction of shadows will be described.
Since it is empirically known that there is a case where there is no viewer's hand on the projection stand (= state where nothing is reflected), the shadow shot image may be mistakenly extracted as a hand shot image. The following processing is performed.
<< Binarization Processing of Captured Image >> The interactive operation device 100 uses a pixel value histogram of a grayscale image and a pixel value histogram as a temporary threshold value t for a perspective-transformed captured image subjected to background image removal processing. Dividing into two regions, calculating the variance (= intra-class variance) σ ² _a in each region and the variance (= inter-class variance) σ ² _b between the ^two regions, A function for calculating the sum of variances between regions (= total variance) (a function of a temporary threshold t) If w = σ ² _a + σ ² _b is equal to or greater than a predetermined value, the method of Otsu is applied. Then, a binarized image is generated.

FIG. 12 is a diagram for explaining the dispersion of the pixel value histogram of the grayscale image.
FIG. 12A is an example of a captured image in a state where nothing is shown.
It shows that the total variance of the pixel value histogram of the captured image in a state where nothing is reflected is small.
FIG. 12B is an example of a captured image in which a hand is shown.
This shows that the total variance of the pixel value histogram of the photographed image in which the hand is reflected is large.

Next, a countermeasure against the slight movement fluctuation of the coordinates of the stationary fingertip will be described.
Since the fingertip coordinates calculated from the captured image of the hand when the viewer stops the fingertip are empirically known to fluctuate as if the fingertip is moving finely, Process.
<< Fingertip Minor Fluctuation Determination Process >> The interactive operation apparatus 100 calculates the fingertip movement direction vector using the fingertip coordinates calculated from each successive frame (= still image of the captured image), and the direction vector fluctuation value. If the condition satisfies a predetermined condition (for example, the direction vector variation angle θ is used to define the direction vector variation value as cos θ and the predetermined condition is θ <25 °), Hold the coordinates of the fingertip of the first frame.

図１３の（ａ）は、指先が移動しているときのｃｏｓθの例である。
指先が移動しているときの数式３の値は大きい。言い換えるとθの値は小さい。例えば、θが２５°未満である場合には、指先が移動しているときである。
図１３の（ｂ）は、指先が静止しているときのｃｏｓθの例である。
指先が静止しているときの数式３の値は小さい。言い換えるとθの値は大きい。例えば、θが２５°以上である場合には、指先が静止しているときである。 FIG. 13 is a diagram for explaining minute fluctuations of the fingertip.
In the case of the direction vectors ν ₁ , ν ₂ , ν ₃ between the four consecutive frames of the image obtained by photographing the moving fingertip, the variation value of the direction vector can be expressed as Equation 3.

FIG. 13A shows an example of cos θ when the fingertip is moving.
The value of Equation 3 when the fingertip is moving is large. In other words, the value of θ is small. For example, when θ is less than 25 °, the fingertip is moving.
FIG. 13B is an example of cos θ when the fingertip is stationary.
The value of Equation 3 is small when the fingertip is stationary. In other words, the value of θ is large. For example, when θ is 25 ° or more, the fingertip is stationary.

Next, a fingertip determination process when the handprint image does not touch the side of the content projection screen will be described.
When the viewer's cuffs cannot be photographed, the handprint image does not touch the side of the content projection screen, so the fingertip determination process is performed as follows.
<< Fingertip Determination Process >> In the fingertip determination process, in the fingertip determination process, when the handprint image does not touch the side of the content projection screen, the side of the side where the held handprint touches with respect to the highest luminance point of the distance image The fingertip is determined without searching for the outline of.
FIG. 14 is an example of a photographed image of a hand that cannot photograph the viewer's cuffs.
The illustrated hand image shows a state in which the viewer's cuffs are on the left side of the screen and the captured image of the hand is away from the left side.
Since the viewer's hand has moved from the left side, the fingertip is determined without searching for the left contour on the basis of the highest luminance point 1038 of the distance image.

1 Fingertip Operation Type Content Providing System 100 Interactive Operation Device 103 Captured Image Display Screen 106 Infrared Camera 107 Projector Connection Unit 110 Initial Setting Unit 130 Fingertip Detection Unit 140 Fingertip Position Calculation Unit 150 Fingertip Stillness Detection Unit 155 Input Emulation Module 160 Display Browsing Means 170 Projection display means 191 Mouse function library 300 Projector, projector 400 Projection table 403 Content projection screen 500 Infrared light source 900 Viewer

Claims (2)

A projector for projecting an image based on the content projection screen data to the projection stage, and an infrared camera to generate a captured image by capturing the projection stage, the interaction device is connected,
And a content storage area for storing the content configuration data,
A display screen holding area for holding the content projection screen data,
Storage means comprising :
Interprets the content member data, a display viewing means for creating the content projection screen data,
Captured image acquisition means for acquiring the captured image from the infrared camera;
And the fingertip detection means for detecting the fingertip of the viewer from each frame of the captured image,
And the fingertip position calculating means for calculating the coordinates of the fingertip,
And calculates the movement distance of the finger using the coordinates of the fingertip of each frame consecutive, when the moving distance during a predetermined time is determined to zero, the fingertip stationary detection information including coordinates of the fingertip Fingertip stationary detection means for creating
Based on the fingertip stationary detection information, and generates the input event information, an input emulation means to pass to the display viewing means,
With
The display browsing means receives the input event information delivered from the input emulation means, updates the content projection screen data as appropriate ,
The fingertip detecting means is one of a plurality of sides constituting the outline of the content projection screen, and one of the sides in contact with the captured image of the hand is a bottom side, a left side, a right side, or an upper side. It is determined whether or not it is a side, and when the determined side is the bottom side, the top of the hand is determined as the fingertip, and when the determined side is the left side, the rightmost part of the hand is the fingertip. determined, the determined sides is determined that the fingertip of the leftmost portion of the hand when it is right, it is determined that the fingertip of the bottom of the hand when the determined sides are the upper side,
An interactive operation device characterized by that. The interactive operation device according to claim 1,
The fingertip detection means includes
If the side of the content projection screen that touches the handprint image is not found, the straight line that passes through the maximum brightness point is parallel to the side of the hand that has been moved and extended based on the maximum brightness point of the distance image. Of the contours of the handprint image divided into two, without searching for the contour on the side where the hand has moved, the farthest part is the fingertip.
An interactive operation device characterized by that.

Images