JP2017187944A - Information processing apparatus, method of determining pointed position, computer program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing apparatus which determines a position pointed by a pointer stably.SOLUTION: An information processing apparatus 100 includes: an image acquisition unit 300 for successively acquiring images captured by a stereo camera 208 at a predetermined time interval; a pointing area detection unit 303 for detecting an area of a pointer image to be used for pointing by a user, from the acquired image; a distance acquisition unit 302 for acquiring a distance from the stereo camera 208, for each of pixels in the pointer image area; and a fingertip position determination unit 304 for determining a current pointed position, on the basis of the distance and a pointed position which is a position determined from an image acquired in the past and to be pointed by the pointer.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an information processing apparatus having a user interface for operating a virtual space. The user interface is hereinafter referred to as “UI”.

  In technical fields such as AR (Augmented Reality) and MR (Mixed Reality), a UI using a head mounted display (hereinafter referred to as “HMD”) is realized. For example, the user can perform a gesture operation on a virtual object displayed by the HMD using an indicator such as a finger. In the UI that enables the manipulation of the virtual object by the gesture, the area of the indicator image is detected from the image taken by the camera mounted on the HMD, and one point (indicated position) in the detected indicator image area is detected. Is an operation pointer. The user can perform various gesture operations on the virtual object by operating the pointer.

  Generally, the pointer is provided at the center of gravity of the detected indicator image area or the tip of the indicator image area (for example, a fingertip). For example, the system of Patent Literature 1 determines the fingertip position of the hand region that serves as a pointer from the direction of entry into the angle of view of the hand. The fingertip position becomes the designated position. Such a pointer is used not only as a cursor indicating the operation input position but also as feedback means for making it easier for the user to grasp the operation status. The pointer is used as a base point for performing hit determination between the virtual object displayed by the HMD and the indicator (hand) in the UI.

JP 2011-22945 A

  In Patent Document 1, the position of the pointer (instructed position) can be determined stably and accurately because the fingertip position of the hand region is not stable due to variations in distance accuracy, hand shake, hand tilt, hand shape, and the like. There are things that cannot be done. When the position of the pointer cannot be accurately determined, not only the operability is deteriorated but also the hit determination between the virtual object and the hand performed inside the information processing apparatus cannot be performed accurately.

  In order to solve the above problems, it is a main object of the present invention to provide an information processing apparatus that stably determines a position indicated by a pointer.

  An information processing apparatus of the present invention that solves the above-described problem is for an image acquisition unit that continuously acquires images captured by a predetermined imaging unit at a predetermined time interval, and for a user to give an instruction from the acquired image An indicator area detecting means for detecting an area of the indicator image, a distance acquiring means for acquiring a distance from the imaging means for each pixel in the indicator image area, and an image acquired in the past Storage means for storing an instruction position that is a position pointed to by the indicator, and determination means for determining a current instruction position based on the instruction position and the distance stored in the storage means. It is characterized by providing.

  According to the present invention, the designated position can be stably and accurately determined by using the designated position determined in the past as an element for determining the current designated position.

(A), (b) is explanatory drawing of a system. (A), (b) is explanatory drawing of an information processing apparatus. Explanatory drawing of a fingertip position determination process. The flowchart showing a fingertip position determination process. (A)-(e) is an illustration figure of the fingertip position for every shape of a hand. (A), (b) is explanatory drawing of a fingertip position determination process.

  Hereinafter, embodiments will be described in detail with reference to the drawings. However, the components described in the present embodiment are merely examples, and are not intended to limit the scope of the present invention only to them.

(Constitution)
FIG. 1 is an explanatory diagram of a system including an information processing apparatus using a head mounted display (HMD) as a user interface. The HMD 101 is equipped with a stereo camera and can capture RGB images. The information processing apparatus 100 and the HMD 101 are connected by a communication cable 105. The information processing apparatus 100 acquires an RGB image from the HMD 101 via the communication cable 105. Further, the information processing apparatus 100 transmits an object (virtual object) to be displayed on the HMD 101 to the HMD 101 via a communication cable. The HMD 101 is attached to the head of the user 106. The HMD 101 displays the virtual objects 103 a and 103 b acquired from the information processing apparatus 100 in the display area 104.

  In the present embodiment, a case where the user 106 operates the virtual objects 103a and 103b displayed in the display area 104 will be described. The information processing apparatus 100 presents a pointer 107 serving as a base point for a gesture operation to the user 106. The user 106 operates the virtual objects 103a and 103b to be operated using the pointer 107. In the present embodiment, an example in which the information processing apparatus 100 and the HMD 101 are connected by wire using a USB (Universal Serial Bus) or the like will be described, but may be configured by wireless connection. For wireless connection, near field communication such as NFC (Near Field Communication) and bluetooth (registered trademark), and wireless USB can be used.

  FIG. 1B is an explanatory diagram of the pointer 107 presented to the user 106. The pointer 107 is displayed in association with the position of the hand 102 of the user 106. The pointer 107 according to the present embodiment is a point that is in a hand area included in an image captured by a stereo camera mounted on the HMD 101 and is located at a position farthest from the HMD 101. As shown in FIG. 1B, in the present embodiment, the x- and y-axes are set on a two-dimensional plane parallel to the display area 104 of the HMD 101, and the z-axis is set in a direction orthogonal to the display area 104, so that three-dimensional Represents coordinates (position) in space. The z-axis direction is depth information (distance information), and the direction away from the HMD 101 (depth increases, distance increases) is a positive direction. Note that the coordinate system of the HMD 101 and the coordinate system of the three-dimensional space are appropriately calibrated, and the coordinate system (hx, hy, hz) of the HMD 101 is 1 in the three-dimensional coordinates (x, y, z) of the real space. One-to-one correspondence.

  In the present embodiment, an example will be described in which the user's 106 hand and its fingertip are used as an indicator for a user to give an instruction to the virtual objects 103a and 103b (operated by the user). In addition to this, an operating instrument such as a stylus pen or a insertion rod can be used as the indicator. The pointer 107 is presented at an indicated position that is a predetermined point in the image area of the indicator. That is, the pointer 107 is presented at the position pointed to by the indicator. Moreover, HMD101 should just be a wearable display apparatus provided with the imaging device.

  FIG. 2 is an explanatory diagram of the information processing apparatus 100. 2A is a hardware configuration diagram of the information processing apparatus 100, and FIG. 2B is a functional block diagram of the information processing apparatus 100. The HMD 101 transmits the RGB stereo image captured by the stereo camera 208 to the information processing apparatus 100.

The information processing apparatus 100 includes a CPU (Central Processing Unit) 200, a RAM (Random
The computer includes an access memory (201) and a read only memory (ROM) 202. The information processing apparatus 100 includes a display interface (I / F) 204 and an input I / F 205 that are interfaces with the HMD 101, and a storage I / F 206 that is an interface with the storage 207. The CPU 200, the RAM 201, the ROM 202, the display I / F 204, the input I / F 205, and the storage I / F 206 are connected via a bus 203 so that they can communicate with each other.

The CPU 200 reads the computer program from the ROM 202 and executes it using the RAM 201 as a work area, thereby controlling the operation of the information processing apparatus 100 and the display of the HMD 101. The ROM 202 stores a startup program, various programs such as an operating system, device drivers, and the like. The RAM 201 provides a temporary storage area that can be accessed at high speed.
The display I / F 204 converts a signal representing the virtual object generated by the CPU 200 into a signal that can be processed by the HMD 101. The input I / F 205 receives an RGB image captured by the stereo camera 208 mounted on the HMD 101 as an input signal and converts it into a signal that can be processed by the information processing apparatus 100.
The storage I / F 206 writes and reads data to and from the storage 207. The storage 207 stores information related to virtual objects generated and output by the information processing apparatus 100. The storage 207 is a mass storage device such as a disk device, flash memory, network drive, or USB drive. In addition, a plurality of storage devices may be connected in parallel.

  The information processing apparatus 100 detects a hand region from the RGB image captured by the stereo camera 208, and determines a fingertip position as an instruction position that is a pointer position. The RGB image captured by the stereo camera 208 is temporarily stored in the RAM 201 and is appropriately read out and processed by the CPU 200. The RGB image may be accumulated in the storage 207.

  The information processing apparatus 100 having such a configuration includes an image acquisition unit 300, a region detection unit 301, a distance acquisition unit 302, an instruction region detection unit 303, a fingertip position determination unit 304, a display control unit 305, a data reading unit 306, and a storage. It functions as the unit 307. These functions are realized by the CPU 200 executing a computer program. For example, at least a part of these functions may be realized by hardware.

  The image acquisition unit 300 continuously acquires right and left RGB images (RGB stereo images) input from the stereo camera 208 at predetermined time intervals and stores them in the RAM 201 as needed. The RGB stereo image is composed of two RGB images captured from two viewpoints. There is a shift (parallax) between the two RGB images, which are associated with the right eye and the left eye, respectively. The stored RGB stereo image is sequentially processed by the information processing apparatus 100. In the present embodiment, it is assumed that the period at which the image acquisition unit 300 acquires RGB stereo images matches the frame rate of the video captured by the stereo camera 208. However, it is also possible to set a predetermined time not limited to the frame rate as an image acquisition cycle. In this embodiment, the RGB image associated with the left eye is processed as a master image, and the RGB image associated with the right eye is processed as a slave image, but this is not restrictive.

  The area detection unit 301 detects the skin color area by using the skin color information of the RGB stereo image stored in the RAM 201. The skin color area is an area for detecting an image area of the hand 102 of the user 106 that is a recognition target. The area detection unit 301 stores the detected skin color area in the RAM 201.

  The distance acquisition unit 302 acquires the distance (depth) from the HMD 101 of each pixel in the skin color area based on the information on the skin color area stored in the RAM 201. The distance acquisition unit 302 searches for corresponding points of the two RGB images (master image and slave image) constituting the stereo image, and calculates the distance using triangulation based on the parallax. Here, a “corresponding point” is a point that appears in an image captured from each viewpoint when a point in the space is simultaneously captured from two viewpoints. Called. Details of the distance calculation method will be described later. The distance acquisition unit 302 stores distance information representing the calculated distance of each pixel in the RAM 201.

  In this embodiment, the stereo camera 208 that captures an RGB image is used for distance acquisition. However, the stereo camera 208 is not necessarily a visible image, and distance acquisition is performed using an infrared stereo image captured by an infrared stereo camera. May be. Further, the distance acquisition may be performed by using a distance image sensor instead of the stereo camera 208.

  The instruction area detection unit 303 detects a skin color area whose distance from the HMD 101 is in a range of 20 [cm] to 70 [cm] as a hand area based on the skin color area and distance information of the master image stored in the RAM 201. . The “hand region” is a region of the user's hand image in the master image. The instruction area detection unit 303 labels the detected hand area and stores it in the RAM 201. In the present embodiment, the hand region is defined as “skin color region whose distance is in the range of 20 [cm] to 70 [cm]” in consideration of the range in which the entire hand is reflected in the stereo camera 208 and the range in which human hands can reach. It is the result. When the distance is shorter than 20 [cm], the captured stereo image is only a skin color region, and it is difficult to determine whether it is a hand. Also, the range that human hands can reach (the range in which the hand 102 of the user 106 is shown) is often 70 [cm] or less, and conversely, if it is 70 [cm] or more, the skin color of the background becomes noise. The possibility increases. For this purpose, the distance from the HMD 101 in the above range is used as a detection criterion for the hand region. This is an example, and when the above-described problem does not occur, the skin color area as the hand area may be set to other than 20 [cm] to 70 [cm].

The fingertip position determination unit 304 determines the fingertip position from the hand region labeled by the instruction region detection unit 303 and the distance information acquired by the distance acquisition unit 302 and stores information representing the fingertip position in the RAM 201. Details of the fingertip position determination process will be described later.
As described above, in this embodiment, the fingertip position (instructed position) is presented to the user 106 as the display position of the pointer 107. The fingertip position (instructed position) is a position in real space corresponding to a pixel having distance information farthest from the HMD 101 among the pixels included in the hand region. For example, when the pixel (m, n) in the stereo image (more precisely, the master image) is farthest from the HMD 101, the fingertip position in the HMD coordinate system corresponding to the pixel (m, n) is (hx, hy, hz). The fingertip position in the real space is (x, y, z). Since each coordinate system shown in FIG. 1 is appropriately calibrated, a point (m, n) in the image, a point (hx, hy, hz) in the HMD coordinate system, and a point (x, y, z) can be converted into each other using a conversion formula.

The display control unit 305 generates the pointer 107 according to the information indicating the fingertip position stored in the RAM 201 and outputs the pointer 107 to the HMD 101. As a result, the pointer 107 is superimposed and displayed in the display area 104 of the HMD 101 in addition to the image captured by the stereo camera 208.
The user 106 can focus or select the virtual objects 103a and 103b by moving the pointer 107 to the positions of the virtual objects 103a and 103b. At this time, the information processing apparatus 100 performs hit determination using the three-dimensional coordinates of the pointer 107 and the three-dimensional coordinates of the virtual objects 103a and 103b. The hit determination is to determine whether or not the virtual objects 103a and 103b and the pointer 107 are in contact with each other.

  The data reading unit 306 reads data representing the virtual objects 103 a and 103 b from the storage unit 307 and sends the data to the display control unit 305. The display control unit 305 generates virtual objects 103 a and 103 b based on this data and outputs them to the HMD 101. The storage unit 307 is realized by the storage 207. Thereby, the virtual objects 103 a and 103 b generated by the information processing apparatus 100 are displayed on the display area 104 of the HMD 101. From this state, the user 106 uses the hand 102 to start a gesture operation on the virtual objects 103a and 103b.

(processing)
FIG. 3 is an explanatory diagram of fingertip position determination processing. Since the distance information acquired from the stereo image includes noise, if the fingertip position is determined only with the three-dimensional coordinates of one pixel farthest from the HMD 101, the reliability of the determined fingertip position is low. turn into. Therefore, conventionally, a predetermined number of pixels are selected in order of increasing distance from the HMD 101, and the fingertip position is determined by averaging the three-dimensional coordinates of each pixel.

For example, when the fingertip position is determined by a conventional method from an image (frame F) captured by the stereo camera 208 at time t, the following is performed.
The information processing apparatus 100 first selects the hand region 500a and the plurality of pixels 400a in the hand region 500a from the image of the frame F. The information processing apparatus 100 determines the fingertip position 401 from the average of the positions of the plurality of pixels 400a. In this manner, the fingertip position 401 can be determined by selecting and averaging a plurality of pixels 400a from the hand region 500a in an arbitrary frame (spatial average). When the hand region 500b and a plurality of pixels 400b in the hand region 500b are selected from the image of the frame F + 1 after a predetermined time from time t, the information processing apparatus 100 determines the fingertip position 402. The fingertip position 402 becomes a jump position that is not continuous with the fingertip position 401 obtained from the frame F.

  This is caused by noise included in the distance information acquired from the RGB stereo image. The generation of noise does not always enable the distance to be acquired stably for each frame. Therefore, a plurality of selected pixels are not stable, and as a result, there is a high possibility that the fingertip position becomes unstable. In this embodiment, in order to determine the fingertip position stably, not only a spatial average but also a temporal average is performed. As a general method of performing temporal averaging, there is a method of storing a past fingertip position determined based on an image before time t and determining a fingertip position based on the current image from the average. is there. However, when the frame rate of the RGB stereo image input to the information processing apparatus 100 is not sufficient, the response speed (response) to the operation is reduced.

  The information processing apparatus 100 according to the present embodiment stably determines such an unstable fingertip position while suppressing a decrease in response. The information processing apparatus 100 selects the hand region 500b and the plurality of pixels 400b in the hand region 500b from the image of the frame F + 1 ′ after a predetermined time from the time t, and the plurality of pixels 400b and the fingertip position at the time t Based on 401, the current fingertip position 403 is determined. FIG. 4 is a flowchart showing such fingertip position determination processing by the information processing apparatus 100.

The data reading unit 306 acquires virtual object data from the storage unit 307. The display control unit 305 generates a virtual object based on the virtual object data acquired by the data reading unit 306, and outputs the virtual object to the HMD 101 (S100). The HMD 101 superimposes and displays the virtual object acquired from the display control unit 305 on the image captured by the stereo camera 208. The HMD 101 transmits an RGB stereo image captured by the stereo camera 208 to the information processing apparatus 100 according to a predetermined time interval, for example, a frame rate.
The image acquisition unit 300 continuously acquires RGB stereo images from the HMD 101 at predetermined time intervals (S101). The image acquisition unit 300 stores the acquired RGB stereo image in the RAM 201 for each frame.

  The region detection unit 301 analyzes the RGB stereo image stored in the RAM 201 and detects the presence or absence of a skin color region in the master image and the slave image (S102). When the skin color area is not detected (S102: N), the display control unit 305 performs pointer display control (S110). In this case, there is no skin color area in the RGB stereo image, that is, the fingertip is not imaged.

  When the skin color area is detected (S102: Y), the area detection unit 301 stores data representing the detected skin color area in the RAM 201. The distance acquisition unit 302 analyzes the skin color area of the RGB stereo image stored in the RAM 201, and acquires the distance from the HMD 101 of each pixel in the skin color area in the master image (S103). For example, when acquiring the distance of the skin color region pixel (m, n) in the master image, the distance acquisition unit 302 searches for the stereo corresponding point (m1, n1) corresponding to the pixel (m, n) from the slave image. To do. The distance acquisition unit 302 acquires the distance from the HMD 101 to the pixel (m, n) by triangulation from the searched stereo correspondence point and the parallax between the master image and the slave image. The distance acquisition unit 302 performs such distance acquisition for all skin color region pixels in the master image, and stores the result in the RAM 201.

  The instruction area detection unit 303 detects a skin area whose distance from the HMD 101 is in the range of 20 [cm] to 70 [cm] as a hand area based on the skin color area and distance information of the master image stored in the RAM 201. (S104). The designated area detection unit 303 stores the detected hand area in the RAM 201 together with label information obtained by labeling the hand area.

  The fingertip position determination unit 304 selects N pixels in order of increasing distance from the HMD 101 based on the label information and distance information of the hand area stored in the RAM 201 (S105). Here, N is an integer of 2 or more. The fingertip position determination unit 304 calculates three-dimensional coordinates in the three-dimensional space corresponding to each of the selected N pixels from the conversion formula, and stores the calculation result in the RAM 201. In the example of FIG. 3, the three-dimensional coordinates (p1 to pN) of each of the N pixels 400b are stored in the RAM 201.

  The fingertip position determining unit 304 determines whether or not the fingertip position P of the previous frame is stored in the RAM 201 (S106). When the fingertip position P of the previous frame is not stored (S106: N), the fingertip position determination unit 304 determines the fingertip position by averaging N three-dimensional coordinates (p1 to pN) stored in the RAM 201. And stored in the RAM 201 (S107).

  When the fingertip position P of the previous frame is stored (S106: Y), the fingertip position determination unit 304 determines the fingertip position for each of the N three-dimensional coordinates (p1 to pN) stored in the RAM 201. The distance (d1 to dN) with P is acquired and stored in the RAM 201 (S108). For example, in the example of the image of frame F + 1 ′ in FIG. 3, the fingertip position determination unit 304 determines the distance (d1) from the three-dimensional coordinates (p1 to pN) of each of the N pixels 400b relative to the fingertip position 401 of the previous frame. ~ DN).

The fingertip position determination unit 304 weights each selected three-dimensional coordinate (p1 to pN) in consideration of the corresponding distance (d1 to dN). The fingertip position determining unit 304 determines the fingertip position 403 (see FIG. 3) by averaging the weighted N three-dimensional coordinates, and stores the determined fingertip position 403 in the RAM 201 (S109).
A specific fingertip position determination process will be described. For simplicity of explanation, N = 2. The two selected three-dimensional coordinates P1 (x1, y1, z1), P2 (x2, y2, z2), and distances d1 and d2 from the fingertip position P of the previous frame are used. At this time, the determined three-dimensional coordinates x, y, z of the fingertip position are as follows.
x = (x1 / d1 + x2 / d2) / (1 / d1 + 1 / d2)
y = (y1 / d1 + y2 / d2) / (1 / d1 + 1 / d2)
z = (z1 / d1 + z2 / d2) / (1 / d1 + 1 / d2)
By obtaining the same in the case of N> 2, the time t
The final fingertip position at can be determined.

  In the example of FIG. 3, the fingertip position 401 is determined from the image of the frame F that is the previous frame, and the fingertip position 403 is determined from the image of the frame F + 1 ′ that is the current frame. The fingertip position 401 and the fingertip position 403 are both the tips of the middle fingers of the hand areas 500a and 500b, and are substantially the same position. On the other hand, when N three-dimensional coordinates are simply averaged, the fingertip position 402 is determined. While the fingertip position 401 is the tip of the middle finger of the hand area 500a, the fingertip position 402 is determined in the middle of the middle finger of the hand area 500b, and the position in the hand area is moving. Even when the hand is not moved in this way, the fingertip position is likely to be blurred simply by averaging the three-dimensional coordinates due to the influence of noise resulting from the distance acquisition of the RGB stereo image. However, using the fingertip position determination process according to the present embodiment makes it possible to reduce blurring and determine the fingertip position stably.

  When the fingertip position is stored in the RAM 201, the display control unit 305 instructs the HMD 101 to display the pointer 107 at the fingertip position (S110). When the information processing apparatus 100 displays the pointer on the HMD 101, the information processing apparatus 100 determines whether to end the system (S111). In this embodiment, the information processing apparatus 100 determines to end the system when an instruction to end the system is input by an operation of a physical button or GUI, or when a predetermined time elapses without detecting a recognition target. When the system is not terminated (S111: N), the information processing apparatus 100 returns to the process of S101 and acquires the RGB stereo image of the next frame from the HMD 101. When ending the system (S111: Y), the information processing apparatus 100 ends all the processes.

  As described above, the information processing apparatus 100 according to the present embodiment determines the fingertip position (instructed position) based on the average of the three-dimensional coordinates of the weighted N pixels. According to this, even when there is an influence of noise (temporal fluctuation) due to distance acquisition of a stereo image, the fingertip position (instructed position) can be determined stably. As a result, when the fingertip position (indicated position) is displayed as a gesture operation pointer, blurring of the display position of the pointer is suppressed, so that the operability for the user is improved. Further, even when the fingertip position (instructed position) is used for hit determination, blurring is suppressed, so that stable hit determination can be performed.

In the present embodiment, the average position of the three-dimensional coordinates of the weighted N pixels and the fingertip positions of a plurality of frames determined in the past are stored, and the average position may be determined as the fingertip position. Good. In an environment where a sufficiently high frame rate stereo image can be acquired, the influence on the operation responsiveness by storing the fingertip positions of a plurality of frames can be ignored.
Further, based on the fingertip position Ps obtained by simply averaging the three-dimensional coordinates (p1 to pN) of N pixels and the fingertip position P of the immediately preceding frame, an intermediate position between the fingertip position Ps and the fingertip position P is determined. A new fingertip position may be determined. In this case, the amount of calculation can be reduced as compared with the method described in the present embodiment.

[Modification 1]
The fingertip position determination process requires at least two frames of RGB stereo images. However, when the frame rate is extremely low, it takes time to determine the fingertip position, and the responsiveness of the operation is lowered. In the first modification, the fingertip position is determined without using a plurality of frames as much as possible. The hardware configuration and functional blocks of the information processing apparatus 100 are the same as those in FIG. However, the indication area detection unit 303 also recognizes the shape of the hand.

  FIG. 5 is an illustration of fingertip positions (pointer display positions) for each hand shape. 5A and 5B show the same hand shape, and FIG. 5B is inclined. 5 (a) and 5 (b), the hand is open and the gap between the fingers is closed. FIG. 5C shows a pointing shape with an index finger raised. FIG. 5D shows a shape in which the index finger and the middle finger are raised. In FIG. 5E, the hand is opened and the finger is opened.

  5A and 5B, the shapes of the hand areas 600 and 601 detected by the instruction area detection unit 303 are the same. However, when the hand is shaken or the like and tilts randomly at short time intervals, the fingertip position jumps from the fingertip position 700 that is the tip of the middle finger to the fingertip position 701 that is the tip of the index finger. In the hand shapes 603 and 604 as shown in FIGS. 5D and 5E, when the hand is shaken, the fingertip positions 703a and 703b or the fingertip positions 704a to 704e are frequently switched.

  The shape of the hand as shown in FIG. 5C is less affected by noise (temporal fluctuation) resulting from the acquisition of the distance of the stereo image compared to the shape of the other hand. This is because the width (size) of a region (a region on the fingertip side) for selecting a pixel as a fingertip candidate when calculating the distance of each pixel in the hand region 602 from a stereo image is different from other shapes. This is because the range of the positions of the selected N pixels is not diffused because it is small. In this case, even if the fingertip position 702 is determined by simply averaging the three-dimensional coordinates of the N pixel positions, it is unlikely that the fingertip position will be a jump position. That is, when the shape of the hand region 602 is a pointing shape as shown in FIG. 5C, the fingertip position 702 can be determined stably.

  The information processing apparatus 100 performs a fingertip position determination process similar to that in FIG. However, when the hand region is detected in the process of S104, the pointing region detection unit 303 also recognizes the shape of the hand. When the detected hand shape is the shape of the hand region 602 in FIG. 5C, the indication region detection unit 303 performs pattern matching to determine that the detected hand region shape is the pointing shape. .

  When the pointing area detection unit 303 determines that the shape of the hand is a specific shape such as a pointing shape, the fingertip position determination unit 304 determines whether or not the fingertip position P of the previous frame is stored in the RAM 201. In step S106, it is determined that the fingertip position P is not stored. For this purpose, the fingertip position determination unit 304 determines the fingertip position by averaging the three-dimensional coordinates of the N pixels. That is, this is equivalent to determining the fingertip position only from the latest frame image (stereo image) regardless of the past fingertip position. The processing in the case where the instruction area detection unit 303 determines that the shape of the hand 102 is other than the pointing shape is the same as that in the first embodiment.

  As described above, when the shape of the hand is a specific shape such as a pointing hand, the information processing apparatus 100 determines the fingertip position only with the latest frame image (stereo image). Therefore, responsiveness can be improved as compared with the case where the fingertip position is always determined from two frame images.

[Modification 2]
In the second modification, when the fingertip position is frequently changed due to shaking of the hand 102 as shown in FIGS. 5A, 5B, 5D, and 5E, the following processing is performed to stabilize the position. Allows determination of fingertip position. The hardware configuration and functional blocks of the information processing apparatus 100 are substantially the same as those shown in FIG. 2 except for the fingertip position determination process by the fingertip position determination unit 304. A description of the same configuration is omitted.

  In order to easily understand the effect of the fingertip position determination process in the second modification, the case where the shape of the hand is FIG. 5D will be described. FIG. 6 is an explanatory diagram of the fingertip position determination process based on such a hand shape.

  In the second modification, the number of pixels to be selected is made larger than in the case described with reference to FIG. This is because if the fingertip position is determined from the average of the three-dimensional coordinates of more pixels, the dependency on the position (three-dimensional coordinates) of a specific pixel decreases. However, if the number of pixels to be simply selected is increased, the fingertip position tends to approach the center of the hand region. For example, as shown in FIG. 6A, in the case of the detected hand region 800 and the selected N pixels 900a, the fingertip position 901 is determined at a position close to the center (center of gravity) of the hand region 800. If the fingertip position 901 that is biased toward the position of the center of gravity is used as the position of the pointer, the operability is degraded. This is because the virtual object is hidden in the hand region 800 (hand 102) during the operation, and the pointing position accuracy is lowered. In other words, the pointer presented to the user 106 is desirably the tip position of the hand region 800 as much as possible from the viewpoint of operability.

  In the second modification, the x and y coordinates of the fingertip position are acquired from the three-dimensional coordinates of N pixels, and the z-coordinate of the fingertip position is acquired from the three-dimensional coordinates of M (<N) pixels. The method for obtaining each three-dimensional coordinate is as described above. The fingertip position is determined by weighting and averaging each three-dimensional coordinate. It differs from the above example in that the x and y coordinates are obtained using N three-dimensional coordinates and the z coordinates are obtained using M three-dimensional coordinates smaller than N. Pixels for determining the fingertip position are selected in order of increasing distance from the HMD 101. This method is effective because the distance (depth, z-coordinate) from the HMD 101 of the pixels near the fingertip serving as a fingertip position candidate is substantially the same. That is, it can be assumed that there is little difference in the z coordinate of the three-dimensional coordinates of the selected pixel, but there is a high possibility that the x and y coordinates are greatly different. Therefore, the x and y coordinates are obtained by averaging the three-dimensional coordinates of more pixels, and the z coordinate is obtained by averaging the three-dimensional coordinates of a smaller number of pixels than x and y. The fingertip position determination unit 304 of the present embodiment determines the fingertip position in this way.

  In the process of S105 in FIG. 4, the fingertip position determination unit 304 selects N pixels for x and y coordinate acquisition, and M (M <N) in order of increasing distance from the selected N pixels. Pixels are selected for z-coordinate acquisition. In the process of S108, the fingertip position determination unit 304 determines the distance (d1 to dM to the fingertip position P of the previous frame with respect to the three-dimensional coordinates (p1 to pM to pN) of N pixels and M pixels, respectively. dN). In the process of S109, the fingertip position determination unit 304 obtains the three-dimensional coordinates of the N pixels by weighted average for the x and y coordinates, and the weighted average of the three-dimensional coordinates of the M pixels for the z coordinate. Get by. That is, the fingertip position determination unit 304 determines two coordinates among the three-dimensional coordinates based on the N pixels, and determines the remaining one coordinate among the three-dimensional coordinates based on the M pixels. As a result, the fingertip position determination unit 304 determines the fingertip position.

  The fingertip position determined by such fingertip position determination processing will be described with reference to FIG. In the case of the hand region 800, the pixel 900a selected for the x and y coordinates, and the pixel 900b selected for the z coordinate, the fingertip position determining unit 304 determines the fingertip position 902. Compared with the fingertip position 901 in FIG. 6A, the fingertip position 902 is closer to the tip of the hand region 800. When the fingertip position is determined by this method, the fingertip position is not necessarily determined within the hand region 800. However, this is preferable from the viewpoint of operability, and operability is improved.

  As described above, even when the hand is tilted or has a specific shape, the fingertip position (instructed position) can be determined stably.

  With the information processing apparatus 100 as described above, the operability of the gesture operation using the pointer according to the designated position can be improved, and the hit determination for the virtual object can be accurately performed.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (16)

Image acquisition means for continuously acquiring images picked up by predetermined image pickup means at predetermined time intervals;
An instruction area detecting means for detecting an area of an image of an indicator for a user to give an instruction from the acquired image;
Distance acquisition means for acquiring the distance from the imaging means for each pixel in the area of the image of the indicator;
Storage means for storing an indicated position, which is a position indicated by the indicator, determined from an image acquired in the past;
Determining means for determining a current indicated position based on the indicated position and the distance stored in the storing means;
Information processing device. The determining means selects a predetermined number of pixels in order of increasing distance, and determines the current indicated position based on the indicated position stored in the storing means and the position of each selected pixel. Features
The information processing apparatus according to claim 1. The determining means weights the position of each selected pixel according to the distance between the position of each selected pixel and the indicated position stored in the storage means, and the selected the weighted selection The present indication position is determined according to the position of each pixel,
The information processing apparatus according to claim 2. The storage means stores a plurality of the indicated positions determined from a plurality of images acquired in the past,
The determining means determines the current indicated position based on an average position of the weighted average positions of the selected pixels and a plurality of the indicated positions stored in the storing means. And
The information processing apparatus according to claim 3. The determining means determines the current indicated position as an intermediate position between the weighted average position of the selected pixels and the indicated position stored in the storing means,
The information processing apparatus according to claim 3. The determining means determines the current designated position based on the distance without using the designated position determined from an image acquired in the past when the designated position is not saved in the storage means. Characterized by the
The information processing apparatus according to claim 1. The indication area detecting means detects an area in the skin color area of the image acquired by the image acquisition means that is within a predetermined reference range as the image area of the indicator. Characterized by the
The information processing apparatus according to claim 1. The indication area detection unit is configured to select an area of the skin color area of the image acquired by the image acquisition unit that is within a range of 20 [cm] to 70 [cm] from the imaging unit. It is detected as a region of
The information processing apparatus according to claim 7. The determining means, when the indicated area detecting means determines that the detected area of the image of the indicator has a specific shape, without using the indicated position determined from an image acquired in the past, the distance Determining the current indicated position based on:
The information processing apparatus according to claim 1. The determining means, when the pointing area detecting means determines that the detected area of the image of the pointing object is a pointing shape, without using the pointing position stored in the storing means, Determining the current indication position based on:
The information processing apparatus according to claim 9. The determining means selects M pixels (M <N) in order of further distance from the selected N pixels (N is an integer equal to or greater than 2), and based on the N pixels, the current unit Two coordinates of the three-dimensional coordinates of the designated position are determined, and the remaining one of the three-dimensional coordinates of the current designated position is determined based on the M pixels. ,
The information processing apparatus according to claim 1. The determining means includes
The N pixel positions are weighted according to the distance between the N pixel positions and the designated position stored in the storage unit, and the N pixel positions are weighted. Determining the two coordinates of the three-dimensional coordinates of the current indicated position;
The remaining one coordinate of the three-dimensional coordinates of the current designated position is determined by averaging the remaining one coordinate of the three-dimensional coordinates of the M pixels. ,
The information processing apparatus according to claim 11. Characterized by comprising display control means for displaying a pointer at a position corresponding to the current indicated position of a predetermined display means,
The information processing apparatus according to claim 1. A method executed by an information processing apparatus connected to a predetermined imaging means,
Continuously obtaining images taken by the imaging means at predetermined time intervals;
Detecting a region of an image of a pointer for a user to give an instruction from the acquired image;
Obtaining a distance from the imaging means for each pixel in the area of the indicator image;
A step of determining a current indicated position based on an indication position determined by an image obtained in the past and indicated by the indicator and the distance.
How to determine the indicated position. A computer connected to a predetermined imaging means;
Image acquisition means for continuously acquiring images taken by the imaging means at predetermined time intervals;
An instruction area detecting means for detecting an area of an image of an indicator for a user to give an instruction from the acquired image;
Distance acquisition means for acquiring the distance from the imaging means for each pixel in the area of the image of the indicator;
Storage means for storing an indicated position, which is a position indicated by the indicator, determined from images acquired in the past;
Determining means for determining a current indicated position based on the indicated position and the distance stored in the storing means;
Computer program to function as.   A computer-readable storage medium storing the computer program according to claim 15.