JP4575829B2 - Display screen position analysis device and display screen position analysis program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a display screen position analysis device capable of stably analyzing a position on a display screen pointed by an operator without depending on the position of the operator or requiring any calibration. <P>SOLUTION: This display screen position analysis device 1 is provided with an image input means 10 for inputting a frame image of an operator taken by two or more cameras, a distance image generation means 11 generating a distance image showing a distance from the display screen based on a parallax, a fingertip detection means 14 detecting a fingertip position giving a minimum distance within a distance image prediction area, a viewpoint detection means (means detecting a distance between eyebrows) 16 detecting a viewpoint position of the operator from the frame image, and a screen position analysis means (cursor position analysis means) 18 analyzing a position on the display screen based on the fingertip position, the viewpoint position, and the distance image. The fingertip detection means 14 has a prediction area setting means (prediction area setting part) 14c setting the prediction area. <P>COPYRIGHT: (C)2007,JPO&amp;INPIT

Description

  The present invention relates to a technique for analyzing a position on a display screen selected by an operator, and in particular, a display screen position analyzing apparatus and a display screen position analyzing program for analyzing a position on a display screen pointed to by an operator. About.

  Conventionally, there are pointing devices such as a mouse, a trackball, a joystick, and a tablet for an operator to select a display screen. In order to select the display screen, the operator must hold and operate these pointing devices. With these pointing devices, it is not possible to directly select the position on the display screen intended by the operator, and the cursor on the display screen must be moved to the intended position by operating the pointing device. The operation requires training and familiarity. Further, for example, a mouse requires a flat desk or the like for movement, and the environment in which the operator operates is limited.

  There is a touch panel as a pointing device that allows an operator to directly select an intended location on a display screen. However, with a touch panel, the operator needs to touch the display screen directly, which is convenient when the viewing distance is short like a computer, but when the viewing distance is large, such as when watching TV, the operator You cannot touch the display screen as it is, and you must move your body.

  Therefore, a technique for setting the position on the display screen by detecting the movement and position of the body of the finger or the operator has been studied. For example, as a technique for detecting the movement of the operator's body, a technique for detecting an operation by irradiating the operator with a laser beam has been developed. In addition, a technique is disclosed in which an operator is photographed by a camera, the position of the operator's fingertip is detected from the photographed video, and the position of the cursor on the display screen is set from the position (Non-patent Document 1). And Non-Patent Document 2). In this technique, since the position of the cursor is determined by the position of the fingertip of the operator, the operator can select by moving the cursor to the vicinity of the position by pointing to the position to be selected.

Furthermore, there is a method for setting the position of the cursor based on the position of the operator and assuming the position of the operator and the position of the detected fingertip. In this method, since the position of the cursor is set based on the position of the operator, the cursor can be moved on the display screen at a position close to the extension line of the fingertip as viewed from the operator.
Lee Y, two others, “Real-time recognition of hand movement using silhouette images and its application”, Proceedings of Human Interface Symposium, 1997, Vol. 13, p. 665-672 Harakawa, Kenichi, “System for operating a computer with the movement of a finger”, Image Lab, April 1, 2000, Vol. 11, no. 4, p. 29-33

  However, in the technique of irradiating the operator with laser light, there is a possibility that the operator's body and eyes may be damaged by the laser light, and viewing of the display screen may be disturbed. In the techniques of Non-Patent Document and Non-Patent Document 2, the cursor position is set based only on the position of the fingertip. Therefore, depending on the position of the operator, the cursor moves away from the position pointed by the operator. It was not practical. Furthermore, in the method of detecting the position of a part of the operator's body, there is a problem that the position of the cursor becomes unstable because it may not be detected or erroneous detection may occur.

  Further, in the method of assuming the position of the operator, calibration is required to assume the position of the operator. Therefore, calibration must be performed every time the operator moves while viewing the display screen, which is not practical.

  The present invention has been made to solve the above-described problems of the prior art, does not depend on the position of the operator, does not require calibration, and extends the fingertip as viewed from the operator stably. It is an object of the present invention to provide a display screen position analysis device and a display screen position analysis program that can analyze the position on the display screen on a line.

In order to solve the above-described problem, the on-display-position analyzing apparatus according to claim 1 inputs a video obtained by photographing a subject including an operator by at least two cameras, and based on a frame image constituting the video. A display screen position analysis device for analyzing the position on the display screen of the display device indicated by the fingertip of the operator, wherein the image input means, the background image storage means, the distance image generation means, and the background difference processing It means, and the fingertip detection unit, and the viewpoint detecting unit, and configured to include a screen position analyzer.

According to this configuration, the position analysis device on the display screen stores the background image that is the background image of the operator photographed by the camera by the background image storage unit. Further, the position analysis device on the display screen inputs a frame image constituting a video obtained by photographing the operator with at least two cameras by the image input means in time series, and the disparity of the frame image by the distance image generation means. Based on, a distance image that is an image indicating the distance of the photographed subject from the display screen is generated.

Further, the display screen position analyzing apparatus includes a difference processing unit that generates a difference image between a background image stored in the background image storage unit and a frame image input from the image input unit; A mask image generation unit that masks pixels whose difference indicated by the difference image is smaller than a threshold value and generates a mask image indicating a mask area not photographed by the operator in the frame image, and a difference indicated by the difference image A background image selection unit that determines that the frame image is the background image and stores the frame image in the background image storage unit when the amount of change is less than or equal to a predetermined amount. Further, the position analysis device on the display screen is a part in which the distance from the display screen to the subject is the shortest in the prediction area of a predetermined size based on the distance image generated by the distance image generation unit by the fingertip detection unit. The fingertip position which is the position in the frame image is detected. Further, the position analysis device on the display screen predicts the fingertip position in the distance image based on the fingertip position detected for the frame image input before the frame image by the prediction area setting unit of the fingertip detection unit. A predicted position is calculated, and a predicted area including the predicted position in the distance image is set. As a result, the position analysis device on the display screen tracks the fingertip position and sets the prediction region, so that the fingertip position can be detected stably. Further, the position analysis device on the display screen detects the fingertip position from a non-mask area that is an area excluding the mask area indicated by the mask image generated by the mask image generation unit by the fingertip determination unit of the fingertip detection unit. Based on the distance at the fingertip position and the average value of the distances in the non-mask area, it is determined whether or not the fingertip position indicates the position of the fingertip.

  Further, the display screen position analysis device detects a viewpoint position that is a position between the eyes of the operator in the frame image from the frame image input from the image input means by the viewpoint detection means, A display screen indicated by the fingertip based on the fingertip position detected by the fingertip detection means by the analysis means, the viewpoint position detected by the viewpoint detection means, and the distance indicated by the distance image at the fingertip position and the viewpoint position Analyze the top position.

  Thereby, the position analysis device on the display screen detects the fingertip position and the viewpoint position based on the video imaged of the operator, and the fingertip position, the viewpoint position, and the subject generated using the parallax are detected. Based on the distance image indicating the distance from the display screen, the three-dimensional position of the position between the fingertip of the operator and the eyes can be analyzed. Furthermore, the display screen position analysis device analyzes the position where the extension line passing through the fingertip from between the eyes of the operator intersects the display screen based on the three-dimensional position, and the position on the display screen The position on the display screen of the display device indicated by the fingertip of the operator can be analyzed.

  As a result, the position analysis device on the display screen detects the fingertip position from the background image and the part where the change has occurred, that is, the area where the operator is photographed, based on the difference image between the background image and the frame image. The fingertip position can be detected stably. Further, the average value of the distance from the subject to the display screen in the non-mask area and the distance of the fingertip position are respectively the average value of the distance from the operator's display screen and the display screen of the subject corresponding to the detected fingertip position. In order to indicate the distance from the display screen, the position analysis device on the display screen determines whether the fingertip position is closer to the operator's fingertip based on how close the subject corresponding to the detected fingertip position is to the display screen than the operator. It can be determined whether or not the corresponding position.

  As a result, even if the background changes, the position analysis device on the display screen stores the changed background image in the background image storage means, so that only the area where the operator is photographed can be set as a non-mask area. The fingertip can be detected stably.

Further, the display screen position analysis program according to claim 2 inputs a video obtained by photographing a subject including an operator by at least two cameras, and based on frame images constituting the video, the operator's An image input means comprising a background image storage means for storing a background image that is a background image of the operator photographed by the camera in order to analyze the position on the display screen of the display device indicated by the fingertip. distance image generating means, and the background difference processing means, the fingertip detection means, the viewpoint detecting means, and be made to function as on-screen position analyzing means.

  According to such a configuration, the position analysis program on the display screen inputs, in a time series, the frame images constituting the video obtained by photographing the operator with at least two cameras by the image input means, and the distance image generating means Based on the parallax of the frame image, a distance image that is an image indicating the distance of the photographed subject from the display screen is generated.

Further, the display screen position analysis program includes a difference processing unit that generates a difference image between a background image stored in the background image storage unit and a frame image input from the image input unit; A mask image generation unit that masks pixels whose difference indicated by the difference image is smaller than a threshold value and generates a mask image indicating a mask area not photographed by the operator in the frame image, and a difference indicated by the difference image A background image selection unit that determines that the frame image is the background image and stores the frame image in the background image storage unit when the amount of change is less than or equal to a predetermined amount. Further, the position analysis program on the display screen is a part in which the distance from the display screen to the subject is the shortest in the prediction area of a predetermined size based on the distance image generated by the distance image generation unit by the fingertip detection unit. The fingertip position which is the position in the frame image is detected. Further, the position analysis program on the display screen predicts the fingertip position in the distance image based on the fingertip position detected for the frame image input before the frame image by the prediction area setting unit of the fingertip detection unit. A predicted position is calculated, and a predicted area including the predicted position in the distance image is set. Further, the display screen position analysis program detects the fingertip position from a non-mask area that is an area excluding the mask area indicated by the mask image generated by the mask image generation unit by the fingertip determination means of the fingertip detection means. Based on the distance at the fingertip position and the average value of the distances in the non-mask area, it is determined whether or not the fingertip position indicates the position of the fingertip.

  Further, the display screen position analysis program detects a viewpoint position that is a position between the eyes of the operator in the frame image from the frame image input from the image input means by the viewpoint detection means, A display screen indicated by the fingertip based on the fingertip position detected by the fingertip detection means by the analysis means, the viewpoint position detected by the viewpoint detection means, and the distance indicated by the distance image at the fingertip position and the viewpoint position Analyze the top position.

  As a result, the position analysis program on the display screen detects the fingertip position and the viewpoint position based on the video captured by the operator, and the fingertip position, the viewpoint position, and the subject generated using the parallax are detected. Based on the distance image indicating the distance from the display screen, the three-dimensional position of the position between the fingertip of the operator and the eyes can be analyzed. Further, the position analysis program on the display screen analyzes the position where the extension line passing through the fingertip from between the eyes of the operator intersects the display screen based on the three-dimensional position, and the position on the display screen is determined. The position on the display screen of the display device indicated by the fingertip of the operator can be analyzed. Further, the display screen position analysis program tracks the fingertip position and sets the prediction region, so that the fingertip position can be detected stably.

  The display screen position analysis apparatus and the display screen position analysis program according to the present invention have the following excellent effects.

According to the first or second aspect of the invention, since the operator analyzes the position on the display screen indicated by the fingertip of the operator based on the video image of the operator, the operator uses a device such as a remote controller. The position on the display screen can be designated by pointing with no operation. Also, to detect the position between the fingertip of the operator and the position of the eyes, it is possible to analyze the position on the display screen on the extension line from between the eyes of the operator to the fingertips without depending on the position of the operator This eliminates the need for calibration even when the position of the operator changes.

  Furthermore, there is no need to irradiate the operator with laser light or to attach a detector or the like for detecting the position to the operator's body, which may harm the operator or cause the operator to display the screen. Do not give discomfort when watching. Further, since the fingertip position can be stably detected by tracking the fingertip position, the position on the display screen pointed by the operator can be stably analyzed.

According to the first or second aspect of the present invention, since the fingertip position is detected only from the area where the operator is photographed, a thing other than the operator near the display screen is detected as the fingertip. The position on the display screen pointed by the operator can be analyzed stably. Further, the operator determines whether the fingertip position is a position corresponding to the fingertip of the operator based on how close the subject corresponding to the detected fingertip position is to the display screen compared to the operator. When the position on the display screen is not pointed and the arm is lowered, the position is not detected, and the position on the display screen can be analyzed only when the finger is pointing.

According to the invention described in claim 1 or 2 , since the background image can be automatically stored in the background image storage means, it is necessary for the operator to register the background image in advance or when the background changes. Disappears.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[Configuration of cursor position analyzer]
With reference to FIG. 1 and FIG. 2, the configuration of a cursor position analysis device (display screen position analysis device) 1 according to an embodiment of the present invention will be described. FIG. 1 is an explanatory diagram for explaining an outline of a cursor position analyzing apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram showing the configuration of the cursor position analysis apparatus according to the embodiment of the present invention.

  As shown in FIG. 1, the cursor position analysis apparatus 1 captures an image of an operator U taken by a stereoscopic camera C having a plurality of cameras installed in the vicinity of a display screen of a display device D such as a television receiver. On the display screen of the display device D, which is input from the outside, calculates the position of the eyebrow (viewpoint) between the fingertip and eyes of the operator U shown in the video, and is shown on the extension line from the eyebrow to the fingertip And the cursor S is displayed at this position. As shown in FIG. 2, here, the cursor position analysis apparatus 1 includes an image input unit 10, a distance image generation unit 11, a background difference processing unit 12, a background image storage unit 13, and a fingertip detection unit 14. A face detection unit 15, an eyebrow detection unit 16, an eyebrow position setting unit 17, a cursor position analysis unit 18, a cursor display unit 19, and a video input unit 20.

  The image input means 10 inputs the frame images constituting the video of the operator U taken by the stereoscopic camera C in time series. The frame image input by the image input means 10 is captured by a plurality of cameras, and a plurality of frame images captured at the same time from different positions are input simultaneously. The frame image input here is output to the distance image generation unit 11, the background difference processing unit 12, and the face detection unit 15.

  The distance image generation unit 11 generates a distance image indicating the distance from the display screen of the subject imaged in the frame image based on the frame image input from the image input unit 10. The distance image generated here is output to the fingertip detection unit 14 and the cursor position analysis unit 18.

  Here, frame images taken by a plurality of cameras at different positions are input to the image input unit 10, and the distance image generation unit 11 calculates the parallax of these frame images, and calculates the distance based on the parallax. By calculating, an image indicating a distance from a camera installed near the display screen is generated as a distance image. Here, since the stereoscopic camera C is installed in the vicinity of the display screen, the distance image generation unit 11 generates a distance image indicating the distance from the display screen by generating an image indicating the distance from the camera. Can do. This parallax calculation can be performed by, for example, a block matching method. Hereinafter, a method for generating a distance image by the block matching method will be described.

  First, the distance image generation unit 11 sets a block centered on each pixel for each frame image. This block is a collection of a predetermined number of pixels such as 8 × 8 pixels. Then, the distance image generation means 11 performs matching between the block of the frame image of interest and the block of another frame image. Then, as a result of matching, the shift of the center pixel of the two matching blocks is the parallax value between the frame images. At this time, the shift can be made smaller than the interval of one pixel by the sub-pixel processing. Then, the distance image generation unit 11 sets the parallax value as the value of the pixel. Further, the distance image generation unit 11 similarly calculates the parallax value for the next pixel of the frame image of interest. In this way, the distance image generation unit 11 calculates the parallax value for all the pixels of the frame image of interest, and generates a parallax image using this as the pixel value.

  Further, the distance image generation unit 11 converts the parallax value of the parallax image into a pixel value corresponding to the distance to generate a distance image. At this time, calibration is performed in advance to obtain an internal parameter (camera parameter) of the camera, and the distance image generation unit 11 converts the parallax value into a distance using the camera parameter. As described above, the distance image generation unit 11 can generate a distance image indicating the distance from the display screen of the subject corresponding to the pixel of each pixel of the frame image of interest.

  Note that the stereoscopic camera C is installed at a position away from the display screen, and the distance image generation unit 11 calculates the distance from one camera based on the parallax, and further, the tertiary in the real space corresponding to each pixel. A distance image indicating a distance from the display screen may be calculated by calculating the original position. Further, the stereoscopic camera C only needs to include at least two cameras, and if the number is two or more, the distance image generation unit 11 can generate a distance image from the frame image based on the parallax.

  The background difference processing means 12 obtains a difference image between a background image stored in a background image storage means 13 to be described later, which is an image of a background without an operator U, and a frame image input from the image input means 10. And generating a mask image indicating a background region. Here, the background difference processing means 12 includes a difference processing unit 12a, a mask image generation unit 12b, and a background image selection unit 12c.

  The difference processing unit 12 a generates a difference image between the background image stored in the background image storage unit 13 and the frame image input from the image input unit 10. Here, the difference processing unit 12a calculates a difference between the frame image and the background image for each pixel to obtain a difference image. The difference image generated here is output to the mask image generation unit 12b and the background image selection unit 12c. In addition, as a signal used when the difference processing unit 12a calculates a difference, a luminance signal, a color difference signal, a color signal, an I signal, or a combination of these signals can be used.

  The mask image generation unit 12b generates a mask image indicating a region where the operator U is not photographed in the frame image input from the image input unit 10 based on the difference image generated by the difference processing unit 12a. It is. The mask image generation unit 12b can generate a mask image using a pixel whose difference indicated by the difference image is smaller than a threshold as a mask. As this threshold value, for example, the variance or maximum value of each pixel value when a background image is taken can be used. The mask image generated here is output to the closest block determination unit 14 b of the fingertip detection unit 14.

  The background image selection unit 12c determines whether the frame image input from the image input unit 10 is an image of the background based on the difference image generated by the difference processing unit 12a. Here, the frame image determined as the background image is stored in the background image storage means 13. As a result, even if the background image is not stored in the background image storage unit 13 in advance, the background image selection unit 12c can automatically select an image to be the background image from the frame images.

  Here, the background image selection unit 12c may determine that the background image is captured when the total difference (change amount) indicated by the difference image generated by the difference processing unit 12a is smaller than a predetermined value. it can. Here, the background image selection unit 12c accumulates the image determined as the background image in the background image accumulation unit 13 and reduces the influence of camera noise, and the background accumulated in the background image accumulation unit 13 A background image which is an image having an average value of pixel values of a plurality of frame images obtained by shooting the pixel values as pixel values is generated and stored in the background image storage unit 13.

  The background image storage unit (background image storage unit) 13 stores a background image that is an image of the background, and is a general storage unit such as a semiconductor memory or a hard disk. Here, the background image accumulating unit 13 accumulates the image determined as the background image by the background image selecting unit 12c and the background image generated by the background image selecting unit 12c. The background image storage unit 13 may store a background image in advance, or may store the latest image determined as a background image by the background image selection unit 12c as a background image. . The background image accumulated here is read out when the difference processing unit 12a generates a difference image, and is read out when the background image selection unit 12c generates a background image.

  The fingertip detection unit 14 is captured in the frame image input from the image input unit 10 based on the distance image generated by the distance image generation unit 11 and the mask image generated by the background difference processing unit 12. The position of the fingertip of the operator U is detected. Here, the fingertip detection unit 14 includes a candidate block search unit 14a, a closest block determination unit 14b, a fingertip determination unit 14c, and a prediction region setting unit 14d.

  The candidate block search unit 14a includes, in the distance image generated by the distance image generation means 11, a non-mask area that is an area excluding the mask area indicated by the mask image generated by the background difference processing means 12, and the prediction described later The prediction region set by the region setting unit 14d is blocked, and a predetermined number of short blocks indicated by the distance image are extracted. The extracted block is output to the closest block determination unit 14b.

  The block is composed of a plurality of pixels such as an 8 × 8 pixel area. The non-mask area is an area where the image of the operator U exists in the frame image. Therefore, by extracting a block having a short distance for this region of the distance image, the candidate block search unit 14a removes the influence of the background and extracts a block including a region corresponding to the fingertip of the operator U. Can do. The predicted area is an area where a block corresponding to the fingertip is predicted to exist with a high probability based on the movement of the fingertip. By extracting a block having a short distance from the predicted area, the position of the fingertip can be tracked and a block including an area corresponding to the fingertip can be extracted stably. Here, the candidate block search unit 14a extracts a predetermined number of blocks in ascending order of distance.

  The closest block determination unit 14b selects one block among the blocks extracted by the candidate block search unit 14a as a block indicating the fingertip position. Here, in the closest block determination unit 14b, based on the position of the fingertip of the previous frame image by the prediction region setting unit 14d described later, a prediction region predicted to include a block corresponding to the fingertip in this distance image When the block extracted by the candidate block search unit 14a is included in the prediction area, the closest block determination unit 14b selects the block having the shortest distance from the block in the prediction area. If it is selected and not included in the prediction region, the block having the shortest distance outside the prediction region is selected. The block selected here is output to the fingertip determination unit 14c.

  The fingertip determination unit (fingertip determination means) 14c compares the block distance selected by the closest block determination unit 14b with the average value of the block distances corresponding to the image of the operator U, and the selected block is It is determined whether the block corresponds to the fingertip. Here, the fingertip determination unit 14c calculates the average value of the distances of the regions other than the mask region indicated by the mask image generated by the background difference processing means 12, and selects this average value and the closest block determination unit 14b. It is determined whether or not the difference from the distance of the determined block is greater than a predetermined value. If it is greater than the predetermined value, the closest block determination unit 14b determines that this block is a block corresponding to the fingertip, and uses the position information (fingertip position) of this block on the distance image as the prediction region setting unit 14d. And a cursor position analyzing unit 18, a detection start signal indicating that a block corresponding to the fingertip has been detected is output to the eyebrow position setting unit 17. If it is not greater than the predetermined value, the fingertip determination unit 14c determines that this block is not a block corresponding to the fingertip, and outputs a non-detection signal to the cursor display means 19.

  That is, the non-mask area is an area corresponding to the image of the operator U, and the average value of the distance of this area can be regarded as indicating the distance from the camera that captured the frame image to the operator U. . Then, the predetermined value is set to, for example, the length of the arm, and if the portion closest to the camera in the prediction region is closer than the average value of the distance in the region other than the mask region, the operator It can be seen that U raises his arm and points his fingertip toward the display screen. On the other hand, if it is not close to a predetermined value or more, it can be seen that the operator U does not point his fingertip at the display screen and does not need to display a cursor on the display screen. In addition, it is good also as using the distance in the position between the eyebrows detected by the eyebrow detection means 16 mentioned later instead of the average value of the distance in areas other than a mask area | region.

  The prediction region setting unit (prediction region setting unit) 14d is a region in which a block corresponding to the fingertip is predicted to exist in the distance image of the next frame based on the fingertip position input from the fingertip determination unit 14c. The area is set. Here, the prediction area setting unit 14d determines the speed at which the fingertip moves on the distance image based on the position (fingertip position) of the block corresponding to the fingertip detected based on the current frame and the distance image one frame before. And the fingertip position in the distance image of the next frame is estimated from the speed. For example, a general Kalman filter can be used for estimation of the fingertip position. Then, the prediction region setting unit 14d sets a region having a predetermined size centered on the fingertip position as the prediction region. The prediction region set here is output to the candidate block search unit 14a and the closest block determination unit 14b.

  The face detection unit 15 detects the face of the operator U from the frame image input from the image input unit 10. This face detection is pre-processing for detecting the eyebrows between the eyes of the operator U from the frame image at high speed by the eyebrows detecting means 16 described later, and the detection of the eyebrows can be processed in real time. The cursor position analysis device 1 may not include the face detection unit 15, and the eyebrow detection unit 16 may directly detect the eyebrow interval from the frame image input from the image input unit 10. The face area detected here is output to the eyebrow detection means 16. If no face is detected, the face detection means 15 notifies the eyebrow position setting means 17 that no face has been detected. Thus, since the face detection unit 15 limits the detection range and detects the position between the eyebrows by the eyebrow detection unit 16 described later, the processing speed can be improved.

  The face detection means 15 can use a general face detection method. For example, the face detection means 15 can use a method based on color information. According to this method, an area where a color close to a preset face color exists in the frame image is determined as a face area. This method includes a method using an RGB space for obtaining a color difference, a method using only a color difference, and a method using an I axis.

  In addition, the face detection unit 15 may use a face image recognition algorithm included in the OpenCV library as a face detection method. This OpenCV is a software library related to Computer Vision that is open sourced to developers by Intel (registered trademark). The face recognition algorithm included in OpenCV was proposed by Paul Viola et al. And improved by Rainer Lienhar et al. In this face recognition algorithm, a face is detected by using a detector that has learned from hundreds of samples. The detector has a hierarchical structure and can detect an object at high speed.

  Here, for example, the face detection unit 15 may specify a range for detecting a face based on the position between the eyebrows of the previous frame selected by the eyebrow candidate point selection unit 16b of the eyebrow detection unit 16 described later. Good.

  The eyebrow detection means (viewpoint detection means) 16 detects the position between the eyebrows from the area of the face image detected by the face detection means 15. Here, the eyebrow detection means 16 detects the position between the eyebrows using a ring filter. The eyebrow detection means 16 includes a ring filter unit 16a and an eyebrow candidate point selection unit 16b. Here, the case where the eyebrow detection unit 16 uses a ring filter will be described. However, the eyebrow detection unit 16 is not limited to this method, and can be realized by various methods for detecting the position between the eyebrows.

  The ring filter unit 16a extracts an image candidate between the eyebrows from the face image region input from the face detection means 15 by a one-dimensional ring filter. Here, since the ring filter unit 16a extracts candidates based on one-dimensional information, a plurality of candidates including regions other than the eyebrows are extracted. The eyebrow candidate extracted here is output to the eyebrow candidate point selection unit 16b.

  The eyebrow candidate point selection unit 16b selects one eyebrow candidate as an image between the eyebrows of the operator U from the plurality of eyebrow candidates extracted by the ring filter unit 16a, and detects the position of this candidate. . Here, the eyebrow candidate point selection unit 16b matches the eyebrow candidate with a two-dimensional template between eyebrows stored in advance in a storage unit (not shown) to detect the position of the most matching candidate. Information on the position between the eyebrows detected here is output to the position setting means 17 between the eyebrows.

  The eyebrow position setting means 17 is for setting the position between the eyebrows, which is the position between the eyebrows of the operator U in the frame image. Here, the eyebrow position setting means 17 inputs either the notification that the face is not detected from the face detection means 15 or the information on the position between the eyebrows from the eyebrow candidate point selection unit 16b, and sets the position between the eyebrows. . The eyebrow position setting means 17 locks (fixes) the position between the eyebrows while the fingertip detection means 14 continues to detect the fingertip based on the detection start signal input from the fingertip determination unit 14c of the fingertip detection means 14. ) The position between the eyebrows set here is output to the cursor position analyzing means 18. Hereinafter, a method in which the eyebrow position setting unit 17 sets the eyebrow position will be described.

  When the detection start signal is input in the previous frame and the current frame from the fingertip determination unit 14c of the fingertip detection unit 14, the interbrow position setting unit 17 sets the interbrow position set in the previous frame to the current position. The position between the eyebrows of the frame. Further, the eyebrow position setting means 17 receives information indicating the position between the eyebrows from the eyebrow candidate point selection unit 16b, and starts detection in the previous frame and the current frame from the fingertip determination unit 14c of the fingertip detection means 14. When no signal is input, the position between the eyebrows input from the eyebrow candidate point selection unit 16b is set as the position between the eyebrows.

  Further, the eyebrow position setting means 17 also determines the position between the eyebrows set in the previous frame even when the amount of change in the position between the eyebrows input from the eyebrow candidate point selection unit 16b is not less than a predetermined time and not more than a predetermined value. The position between the eyebrows of the current frame. By setting in this way, the eyebrow position setting means 17 locks the position between the eyebrows of the operator U while the operator U performs a series of operations for specifying the position at which the cursor S is displayed on the display screen. And the movement of the cursor S can be stabilized. Moreover, the position setting means 17 between eyebrows can lock the position between the eyebrows of the operator U and can stabilize the movement of the cursor S even when the position between the eyebrows of the operator U is slightly changed. .

  Further, the eyebrow position setting unit 17 sets the center position of the frame image as the eyebrow position when the notification that the face is not detected from the face detection unit 15 is input. In addition, when the information about the position between the eyebrows from the eyebrow candidate point selection unit 16b is not input, the eyebrow position setting unit 17 determines that the eyebrow position is detected by the eyebrow detection unit 16 in the past. If the position is not detected, the position of the center of the face area detected by the face detection means 15 is set as the eyebrow position.

  The cursor position analysis means (on-screen position analysis means) 18 includes a distance image generated by the distance image generation means 11, a fingertip position detected by the fingertip determination unit 14 c, and an eyebrow position set by the eyebrow position setting means 17. Based on the above, the position on the display screen of the display device D shown on the extension line from the eyebrow of the operator U to the fingertip is analyzed and set as the position of the cursor S. The cursor position analysis means 18 includes a three-dimensional position calculation unit 18a and a cursor position calculation unit 18b.

  The three-dimensional position calculation unit 18a operates based on the distance image generated by the distance image generation unit 11, the fingertip position detected by the fingertip determination unit 14c, and the interbrow position set by the interbrow position setting unit 17. The three-dimensional position between the fingertip of the person U and the eyebrows is calculated. At this time, calibration is performed in advance to obtain camera parameters. Then, the three-dimensional position calculation unit 18a, based on the camera parameters, the fingertip position and the eyebrow position, and the distance from the display screen indicated by the distance image at the fingertip position and the eyebrow position to the fingertip and the eyebrow position, A three-dimensional position between the eyebrows can be calculated. The three-dimensional position between the fingertip and the eyebrows calculated here is output to the cursor position calculation unit 18b.

  The cursor position calculation unit 18b displays the display device D indicated by the fingertip of the operator U, which is a position for displaying the cursor S, based on the three-dimensional position between the fingertip and the eyebrows calculated by the three-dimensional position calculation unit 18a. Analyzes the position on the screen. Here, it is assumed that the positional relationship between the camera and the display screen is registered in advance in the cursor position analysis apparatus 1. Then, the cursor position calculation unit 18b calculates an equation of a straight line passing through the two points based on the three-dimensional position between the fingertip and the eyebrows, calculates an intersection of the straight line and the display screen, and displays the intersection on the display screen. The position at is the position of the cursor S. Thus, the cursor position calculation unit 18b can analyze the position on the display screen that is on the extension line of the fingertip as viewed from the operator U. The position of the cursor S analyzed here is output to the cursor display means 19. The data indicating the positional relationship between the camera and the display screen is a fixed value unless the positional relationship between the display screen and the camera changes.

  The cursor display means 19 inputs a video to be displayed on the display screen from the video input means 20 to be described later, and the cursor S is positioned at the position calculated by the cursor position calculation unit 18b for each of the frame images constituting the video. These images are added, converted into a displayable output format, and output to the display device D.

  The video input means 20 inputs video for display on the display screen of the display device D. The video input here is output to the cursor display means 19.

  As described above, the cursor position analyzing apparatus 1 can display the cursor S at a position on the display screen on the extension line of the fingertip as viewed from the operator U, based on the video image of the operator U. Therefore, the operator U can easily set the position of the cursor S without having a device such as a remote controller. The cursor position analyzing apparatus 1 also detects the position of the eyebrows (viewpoint) of the operator U, so that the cursor S is displayed on the extension line of the fingertip as viewed from the operator U regardless of the position of the operator U. be able to.

  Further, according to the cursor position analyzing apparatus 1, it is not necessary to perform calibration for correction when the position of the operator U changes. Further, according to the cursor position analyzing apparatus 1, there is no need to emit a laser beam or to attach a detector or the like for detecting the position to the operator U, which causes harm to the operator U. In addition, the operator U is not discomforted by attaching the detector.

  In addition, the cursor position analyzing apparatus 1 stably moves the cursor S by locking the eyebrow detection that is likely to cause erroneous detection while the fingertip position setting unit 17 continuously detects the fingertip. Can be made. Furthermore, the cursor position analysis apparatus 1 sets the prediction area by tracking the fingertip position by the prediction area setting unit 14d of the fingertip detection unit 14, and detects the next fingertip position from the prediction area. It is possible to prevent and stably detect the fingertip position.

  Here, the cursor position analysis device 1 converts the parallax value into a distance from the display screen by the distance image generation unit 11 and generates a distance image in which each distance is associated with each pixel. However, the cursor position analyzing apparatus 1 of the present invention may generate a distance image indicating the distance from the display screen to the subject corresponding to each pixel of the frame image by the distance image generating unit 11. A parallax image in which a parallax value is associated with each pixel may be generated and used as a distance image, and the parallax value may be converted into a distance by the three-dimensional position calculation unit 18 a of the cursor position analysis unit 18.

  Further, here, the cursor position analyzing apparatus 1 sets the prediction area by the fingertip detection unit 14 and detects the fingertip position from the distance image based on the prediction area and the mask area. The cursor position analysis apparatus 1 of the invention may not set the prediction region. At this time, the cursor position analysis device 1 does not include the prediction region setting unit 14d, and the candidate block search unit 14a of the fingertip detection unit 14 blocks the non-mask region of the distance image and extracts a predetermined number of blocks having a short distance. The nearest block determination unit 14b selects the block having the shortest distance from the extracted block, and the fingertip determination unit 14c determines whether the selected block is a block corresponding to the fingertip.

  The cursor position analyzing apparatus 1 can also realize each means as a function program in a computer, and can also operate the cursor position analyzing program by combining the function programs.

[Operation of cursor position analyzer]
Next, the operation of the cursor position analyzing apparatus 1 according to the present invention will be described with reference to FIGS. First, referring to FIG. 3, the cursor position analysis device 1 inputs a video image of the operator U from the outside, and the position of the display screen of the display device D indicated by the fingertip of the operator U indicated in the video image An operation for displaying the cursor S on the screen will be described. FIG. 3 is a flowchart showing the operation of the cursor position analyzing apparatus according to the present invention.

  The cursor position analyzing apparatus 1 externally displays a frame image of a video obtained by photographing the operator U with the stereoscopic camera C by the image input means 10 and a video frame image to be displayed on the display screen by the video input means 20. Input (step S10). Here, the stereoscopic camera C has a plurality of cameras, and the cursor position analysis apparatus 1 inputs a plurality of images taken from different positions by the image input means 10.

  Subsequently, the cursor position analysis apparatus 1 uses the distance image generation operation described later by the distance image generation unit 11 to detect the subject imaged in the frame image from the frame image input from the image input unit 10 in step S10. A distance image indicating the distance from the display screen is generated (step S11), and the process proceeds to step S19.

  After inputting the frame image in step S10, the cursor position analyzing apparatus 1 uses the difference processing unit 12a of the background difference processing unit 12 to store the background image stored in the background image storage unit 13 and the image input unit in step S10. A difference image from the frame image input from 10 is generated (step S12). Subsequently, the cursor position analyzing apparatus 1 generates a mask image indicating a region where the operator U has not been photographed based on the difference image generated in step S12 by the mask image generation unit 12b of the background difference processing unit 12. (Step S13).

  Further, the cursor position analyzing apparatus 1 determines that the frame image input from the image input unit 10 in step S10 is based on the difference image generated in step S13 by the background image selection unit 12c of the background difference processing unit 12. It is determined whether the image is a photographed image (step S14). When the image is a background image (Yes in step S14), the cursor position analysis apparatus 1 uses the background image selection unit 12c to input the frame image input from the image input unit 10 in step S10 and the background image storage unit. A background image is generated based on the background image stored in 13, and the frame image and the background image are stored in the background image storage means 13 (step S15), and the process proceeds to step S16. If it is not a background image (No in step S14), the process proceeds to step S19.

  After the frame image is input in step S10, the cursor position analyzing apparatus 1 detects the face image of the operator U from the frame image input from the image input unit 10 in step S10 by the face detection unit 15 ( Step S16). Then, the cursor position analysis apparatus 1 extracts the eyebrow candidates using the ring filter by the ring filter unit 16a of the eyebrow detection unit 16 (step S17). Furthermore, the cursor position analyzing apparatus 1 selects a candidate between the eyebrows from the candidate between eyebrows extracted in step S17 by the candidate point selecting unit 16b between eyebrows, and detects the position between the eyebrows ( The process proceeds to step S18) and step S19.

  Then, the cursor position analysis apparatus 1 detects the fingertip position based on the distance image generated in step S11 and the mask image generated in step S13 by the fingertip detection operation described later by the fingertip detection unit 14 (step S13). S19), the process proceeds to step S20. Then, the cursor position analysis apparatus 1 determines whether the fingertip position is detected by the fingertip determination unit 14c of the fingertip detection unit 14 (step S20). That is, the cursor position analysis apparatus 1 calculates the average value of the distance indicated by the distance image in the area other than the mask area indicated by the mask image generated in step S13 by the fingertip determination unit 14c. Then, it is determined whether the difference from the distance indicated by the distance image of the fingertip position detected in step S19 is greater than a predetermined value.

  If the position of the fingertip is not detected (No in step S20), the cursor position analysis device 1 uses the cursor display means 19 to place the cursor S on the frame image input from the video input means 20 in step S10. The image is output as it is without being added (step S21), and the process returns to step S10, and the operation after the operation of inputting the frame image of the video image of the operator U and the video image to be displayed on the display screen is performed.

  When the position of the fingertip is detected (Yes in step S20), the cursor position analysis apparatus 1 detects the fingertip position detected in step S19 and the previous frame by the prediction region setting unit 14d. Based on the fingertip position, a prediction area of the distance image of the next frame is set (step S22). Then, the cursor position analyzing apparatus 1 sets the position between the eyebrows by the position setting operation between the eyebrows described later by the position setting means 17 for the eyebrows (step S23).

  Then, the cursor position analysis apparatus 1 includes the distance image generated in step S11, the fingertip position detected in step S19, and the eyebrow position detected in step S23 by the three-dimensional position calculation unit 18a of the cursor position analysis unit 18. Based on the above, the three-dimensional position between the fingertip of the operator U and the eyebrows is calculated (step S24). Then, the cursor position analysis apparatus 1 calculates the intersection of the straight line passing through the two points and the display screen based on the three-dimensional position between the fingertip and the eyebrows calculated in step S24 by the cursor position calculation unit 18b. The position of the intersection point on the display screen is calculated as the cursor position (step S25).

  The cursor position analyzing apparatus 1 adds the image of the cursor S to the position of the cursor calculated in step S25 in the frame image input from the video input unit 20 in step S10 by the cursor display unit 19 and outputs it. Then (step S26), returning to step S10, the operation after the operation of inputting the frame image of the video obtained by photographing the operator U and the frame image of the video to be displayed on the display screen is performed.

  With the above operation, the cursor position analyzing apparatus 1 can display the cursor at a position on the display screen on the extension line of the fingertip as viewed from the operator U, based on the video image of the operator U.

(Distance image generation operation)
Next, referring to FIG. 4 (refer to FIGS. 1 and 3 as appropriate), the distance at which the cursor position analyzing apparatus 1 generates a distance image by the distance image generating unit 11 based on the frame image input from the image input unit 10. The image generation operation (step S11 in FIG. 3) will be described. FIG. 4 is a flowchart showing the distance image generation operation of the cursor position analysis apparatus according to the present invention.

  First, the cursor position analyzing apparatus 1 sets a block centered on each pixel in the frame image input from the image input means 10 in step S10 of FIG. 3 (step S30). Then, the cursor position analysis device 1 uses the distance image generation unit 11 to match the block of the frame image of interest with the block of the other frame image, and to determine the pixel shift at the center of the most matched block, The parallax value of this pixel is set (step S31).

  Then, the cursor position analysis apparatus 1 determines whether matching has been completed for all pixels of the frame image of interest by the distance image generation unit 11 (step S32). If all the pixels have not been completed (No in step S32), the cursor position analysis apparatus 1 sets the next pixel by the distance image generation unit 11 (step S33), and returns to step S31. The operation after the operation of matching the block of this pixel of the frame image of interest with the block of another frame image is performed.

  When all the pixels are completed (Yes in step S32), the cursor position analysis apparatus 1 converts the disparity value calculated by the matching in step S31 into a distance by the distance image generation unit 11 and converts the distance image. Is generated (step S34), and the operation is terminated.

(Fingertip detection operation)
Next, referring to FIG. 5 (refer to FIG. 1 and FIG. 3 as appropriate), the cursor position analyzing apparatus 1 is detected by the distance image generating unit 11 by the candidate block searching unit 14a and the closest block determining unit 14b of the fingertip detecting unit 14. A fingertip detection operation (step S19 in FIG. 3) for detecting the fingertip position from the generated distance image will be described. FIG. 5 is a flowchart showing the fingertip detection operation of the cursor position analysis apparatus according to the present invention.

  First, the cursor position analysis apparatus 1 is shown by the mask image generated in step S13 of FIG. 3 among the distance images generated in step S11 of FIG. 3 by the candidate block search unit 14a of the fingertip detection unit 14. The mask area and the prediction area set in step S22 of FIG. 3 are blocked (step S40), and a predetermined number of blocks having a short distance indicated by the distance image are extracted (step S41).

  Then, the cursor position analysis apparatus 1 determines whether the block extracted in step S41 is included in the prediction region by the closest block determination unit 14b (step S42). Then, when it is included in the prediction region (Yes in step S42), the cursor position analysis apparatus 1 selects the block with the shortest distance among the blocks in the prediction region by the closest block determination unit 14b. The position of the block is set as the fingertip position (step S43), and the operation is terminated.

  In addition, when it is not included in the prediction region (No in step S42), the cursor position analysis apparatus 1 selects the block with the shortest distance among the blocks outside the prediction region by the closest block determination unit 14b. Then, the position of the block is set as the fingertip position (step S44), and the operation is terminated.

(Brow position setting operation)
Next, referring to FIG. 6 (refer to FIG. 1 and FIG. 3 as appropriate), the inter-eyebrow position setting operation (step S23 in FIG. 3) in which the cursor position analyzer 1 sets the inter-brow position by the inter-brow position setting means 17 will be described. . FIG. 6 is a flowchart showing an eyebrow position setting operation of the cursor position analysis apparatus according to the present invention.

  First, the cursor position analyzing apparatus 1 inputs a detection start signal generated when it is determined by the eyebrow position setting means 17 that a fingertip has been detected in step S20 of FIG. 3 for the current frame and the previous frame. It is determined whether it has been performed (step S50). If it is input (Yes in step S50), the cursor position analysis apparatus 1 sets the inter-eyebrow position set in the previous frame by the inter-brow position setting unit 17 as the inter-brow position of the current frame ( Step S52), the operation is terminated.

  If no input has been made (No in step S50), the cursor position analyzing apparatus 1 detects the position of the eyebrows in step S18 of FIG. It is determined whether information on the position between the eyebrows is input from the point selection unit 16b (step S51). If it has not been input (No in step S51), the process directly proceeds to step S55. In addition, when input (Yes in step S51), the cursor position analysis apparatus 1 analyzes the position information and the amount of change in position between the eyebrows by using the position setting means 17 between the eyebrows. It is determined whether the amount is not less than a predetermined value for a predetermined time (step S53).

  When the fluctuation amount is not less than the predetermined time and not more than the predetermined value (Yes in step S53), the cursor position analyzing apparatus 1 uses the inter-eyebrow position setting unit 17 to determine the inter-eyebrow position set in the previous frame. The position between the eyebrows of the current frame is set (step S52), and the operation is terminated. If the fluctuation amount is not less than the predetermined time and not more than the predetermined value (No in step S53), the cursor position analyzing apparatus 1 determines the position between the eyebrows detected in step S18 in FIG. The position is set between the eyebrows (step S54), and the operation ends.

  When the information on the position between the eyebrows is not input (No in step S51), the cursor position analyzing apparatus 1 determines whether the position between the eyebrows has been previously detected by the position setting means 17 between the eyebrows ( Step S55). If it is detected (Yes in step S55), the cursor position analyzing apparatus 1 sets the position between the eyebrows detected last by the interbrow position setting means 17 as the position between the eyebrows (step S56).

  If not detected (No in step S55), the cursor position analyzing apparatus 1 determines whether the face position setting unit 17 notifies that the face is not detected in step S16 of FIG. (Step S57).

  When notified (Yes in step S57), the cursor position analyzing apparatus 1 uses the position between the eyebrows position setting means 17 to set the position of the center of the face area detected in step S16 in FIG. Set (step S58), the operation is terminated. If not notified (No in step S57), the cursor position analysis apparatus 1 sets the center position of the frame image as the position between the eyebrows by means of the eyebrow position setting means 17 (step S59), and ends the operation. To do.

It is explanatory drawing for demonstrating the outline | summary of the cursor position analysis apparatus which is embodiment in this invention. It is the block diagram which showed the structure of the cursor position analysis apparatus which is embodiment in this invention. It is the flowchart which showed operation | movement of the cursor position analysis apparatus in this invention. It is the flowchart which showed the distance image generation operation | movement of the cursor position analysis apparatus in this invention. It is the flowchart which showed the fingertip detection operation | movement of the cursor position analysis apparatus in this invention. It is the flowchart which showed the eyebrow position setting operation | movement of the cursor position analyzer in this invention.

Explanation of symbols

1 Cursor position analyzer (Position analyzer on display screen)
DESCRIPTION OF SYMBOLS 10 Image input means 11 Distance image generation means 12 Background difference processing means 13 Background image storage means (background image storage means)
14 fingertip detection means 14c fingertip determination unit (fingertip determination means)
14d Prediction region setting unit (Prediction region setting means)
16 Eyebrow detection means (viewpoint detection means)
18 Cursor position analysis means (on-screen position analysis means)

Claims (2)

A display for inputting a video obtained by photographing a subject including an operator by at least two cameras, and analyzing a position on a display screen of a display device indicated by a fingertip of the operator based on a frame image constituting the video A position analysis device on the screen,
Image input means for inputting the frame images in time series;
Background image storage means for storing a background image that is an image of the background of the operator photographed by the camera;
Based on the parallax of the input frame image from the image input unit, the captured the object, a distance image generating means for generating a distance image is an image showing the distance from the display screen,
A difference processing unit that generates a difference image between the background image stored in the background image storage unit and the frame image input from the image input unit, and masks pixels whose difference indicated by the difference image is smaller than a threshold, A mask image generation unit configured to generate a mask image indicating a mask area not photographed by the operator in the frame image; and a change amount of a difference indicated by the difference image is equal to or less than a predetermined amount, the frame image is A background difference processing unit having a background image selection unit that determines that the background image is to be stored in the background image storage unit;
Based on the distance image generated by the distance image generation unit, and the fingertip detection means for detecting the fingertip position is a position in the frame image of the portion in which the distance in the prediction region of a predetermined size is the shortest,
Viewpoint detection means for detecting a viewpoint position that is a position between the eyes of the operator in the frame image from the frame image input from the image input means;
Indicated by the fingertip based on the fingertip position detected by the fingertip detection means, the viewpoint position detected by the viewpoint detection means, and the distance indicated by the distance image at the fingertip position and the viewpoint position. On-screen position analyzing means for analyzing the position on the display screen,
With
The fingertip detection means includes
Based on the detected fingertip position on the frame image input prior to those the frame image, and calculates a predicted position predicted the fingertip position in the range image, in the distance image, wherein including the predicted position A prediction area setting means for setting a prediction area;
The fingertip position is detected from a non-mask area that is an area excluding the mask area indicated by the mask image generated by the mask image generation section, and the distance at the fingertip position and the average value of the distance in the non-mask area And a fingertip determination means for determining whether or not the fingertip position indicates the position of the fingertip,
The on-screen position analyzing means analyzes the position on the display screen indicated by the fingertip when the fingertip detecting means determines that the fingertip position is a position corresponding to the fingertip. Upper position analysis device. In order to analyze a position on the display screen of the display device indicated by the fingertip of the operator, based on a frame image constituting the image, by inputting a video image of a subject including the operator by at least two cameras A computer comprising background image storage means for storing a background image that is a background image of the operator photographed by the camera ;
Image input means for inputting the frame images in time series,
A distance image generating means for generating a distance image which is an image indicating a distance from the display screen of the photographed subject based on the parallax of the frame image input from the image input means;
A difference processing unit that generates a difference image between the background image stored in the background image storage unit and the frame image input from the image input unit, and masks pixels whose difference indicated by the difference image is smaller than a threshold, A mask image generation unit configured to generate a mask image indicating a mask area not photographed by the operator in the frame image; and a change amount of a difference indicated by the difference image is equal to or less than a predetermined amount, the frame image is A background difference processing unit having a background image selection unit that determines that the background image is to be stored in the background image storage unit;
The distance image based on the distance image generated by the generating means, the fingertip detection means for detecting the fingertip position the distance a predetermined size of the prediction region is located in the frame image of the portion having the shortest,
Viewpoint detection means for detecting a viewpoint position that is a position between the eyes of the operator in the frame image from the frame image input from the image input means;
Indicated by the fingertip based on the fingertip position detected by the fingertip detection means, the viewpoint position detected by the viewpoint detection means, and the distance indicated by the distance image at the fingertip position and the viewpoint position. Function as on-screen position analyzing means for analyzing the position on the display screen,
The fingertip detecting means ;
Based on the detected fingertip position on the frame image input prior to those the frame image, and calculates a predicted position predicted the fingertip position in the range image, in the distance image, wherein including the predicted position Prediction area setting means for setting the prediction area ;
The fingertip position is detected from a non-mask area that is an area excluding the mask area indicated by the mask image generated by the mask image generation section, and the distance at the fingertip position and the average value of the distance in the non-mask area And the fingertip determination means for determining whether or not the fingertip position indicates the position of the fingertip,
The on-screen position analyzing means analyzes the position on the display screen indicated by the fingertip when the fingertip detecting means determines that the fingertip position is a position corresponding to the fingertip. Top position analysis program.

Images