JP2010210490A - Device and method for measuring movement, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To eliminate unnaturalness in movement of a cursor or the like in a depth direction with respect to a three-dimensional application in an input device of an information apparatus. <P>SOLUTION: This device for measuring movement basically includes a compound-eye imaging means having a lens array with a plurality of lenses arranged in a roughly plane and imaging elements for obtaining a compound image which is aggregation of single-eye images imaged by the plurality of lenses of the lens array with respect to a subject moving in a vertical direction relative to the lens array, a shift amount calculation means that fetches in a time series manner the compound-eye images obtained by the compound-eye imaging means and sequentially calculates out image shift amounts between the single-eye images of the compound-eye images, and a movement amount calculation means that calculates out a movement amount of the subject on the basis of an image shift amount at a first clock time and an image shift amount at a second clock time different from the first clock time which are calculated out by the shift amount calculation means. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a technique for measuring movement of a subject such as a user's finger in a direction perpendicular to an imaging apparatus. Specifically, the present invention relates to a movement measuring apparatus and method suitable for directly reflecting movement of a cursor or the like in the depth direction on three-dimensional information on a display as an input device of an information device, and a program.

  As input devices for information devices such as personal computers, a mouse, a touch pad, a touch panel, a cross key, and the like are widely used. The user moves the cursor on the display via these input devices and operates the displayed information.

  On the other hand, in order to make effective use of the display area with limited display, and because of the desire of users to display richer information, there are an increasing number of applications that make the display three-dimensional (for example, Microsoft Windows Vista Aero and Ricoh quanp).

  However, in the conventional input device, the method of moving the cursor or the like in the depth direction with respect to the three-dimensional information on the display is not intuitive. For example, the mouse wheel is rotated to move in the depth direction, but the in-plane operation directly reflects the amount of mouse movement, which is unnatural.

  Further, for example, Patent Document 1 describes that the movement of a subject is detected from a plurality of images captured in time series by an imaging apparatus including a plurality of lenses such as a microlens array. It does not describe detecting the movement of the subject in the vertical direction.

  The present invention measures the movement of a subject such as a user's finger in a direction perpendicular to the apparatus with a simple configuration by using an imaging apparatus equipped with a lens array, and is used as an input device of an information device on a display The objective is to realize intuitive operation on the three-dimensional information without a sense of incongruity.

  In the present invention, for a lens array in which a plurality of lenses are arranged in a substantially planar shape and a subject that moves in a direction perpendicular to the lens array, single-eye images respectively formed by the plurality of lenses of the lens array Compound eye imaging means including an imaging device that acquires a compound eye image that is a set, and a compound eye image acquired by the compound eye imaging means are input in time series, and the image shift amount between the single eye images of the compound eye image is sequentially Based on the shift amount calculation means to be calculated, the image shift amount at the first time calculated by the shift amount calculation means, and the image shift amount at a second time different from the first time, the movement of the subject The basic configuration includes a moving amount calculating means for calculating the amount.

  In one embodiment, the compound-eye imaging means obtains a compound eye image that is a set of two single-eye images respectively formed by a lens array in which two lenses are arranged in a substantially planar shape and the two lenses of the lens array. The shift amount calculating means calculates an image shift amount between two single-eye images of the compound-eye image acquired by the compound-eye imaging means.

  In another embodiment, the compound-eye imaging means is a set of a lens array in which a large number of lenses are arranged in a substantially planar shape and a large number of single-eye images respectively formed by the large number of lenses in the lens array. The shift amount calculating means selects a plurality of sets of two single-eye images from the compound-eye images acquired by the compound-eye image pickup means, and for each set between the single-eye images. The image shift amount is calculated, and the average value thereof is finally set as the image shift amount.

  In still another embodiment, the image processing apparatus further includes a shift amount determination unit that determines whether or not the image shift amount calculated by the shift amount calculation unit is within a predetermined range, and the movement amount calculation unit includes the shift amount determination unit. The amount of movement of the subject is calculated according to the determination result.

  Specifically, the movement amount calculation means calculates and outputs the movement amount of the subject when the image shift amount is within a predetermined range. When the image shift amount is within the predetermined range or more, the movement amount calculation means moves backward or forward. The movement information is output continuously.

  The movement amount calculating means multiplies the difference or ratio between the image shift amounts between the first time and the second time by a predetermined coefficient so as to correspond to the movement amount.

  According to the present invention, a compound eye imaging apparatus using a lens array can measure the movement of a subject (for example, a user's finger) in a direction perpendicular to the apparatus with a simple configuration. In addition, by calculating the subject position in the direction perpendicular to the compound-eye imaging device and continuously performing it, it is possible to provide an input device that can intuitively operate the three-dimensional information on the display without a sense of incongruity. .

It is a functional block diagram of one embodiment of a movement measuring device of the present invention. It is a block diagram of the 1st Example of a compound eye imaging part. It is a block diagram of the 2nd Example of a compound eye imaging part. It is a figure explaining the relationship between a to-be-photographed object distance and the image shift amount between adjacent single-eye images. It is a figure explaining calculation of the amount of image shifts. It is a figure explaining the relationship between the distance of a to-be-photographed object and a threshold value. It is a figure which shows the whole process flow of FIG. It is a system configuration | structure figure of the example of application of the movement measuring device of this invention. It is a figure which shows an example of the three-dimensional application displayed on a display apparatus. It is a figure which shows another example of the three-dimensional application displayed on a display apparatus.

  FIG. 1 shows a functional block diagram of an embodiment of the movement measuring apparatus of the present invention. The movement measuring apparatus includes a compound eye imaging unit 10, a shift amount calculation unit 20, a shift amount determination unit 30, a movement amount calculation / output unit 40, and a shift amount storage unit 50. Here, each unit other than the compound-eye imaging unit 10 is generally realized using a microprocessor, a RAM, a ROM, and the like. The shift amount determination unit 30 may be omitted.

  First, the compound eye imaging unit 10 will be described. FIG. 2 shows a configuration diagram of the first embodiment as a basic configuration of the compound eye imaging unit 10. In FIG. 2, 11 is a lens array, 12 is a light shielding member, 13 is an image sensor, and 14 is a cover glass.

  The lens array 11 includes two lenses 11a and 11b arranged in a plane substantially orthogonal to the lens optical axis, and forms an optical image of the subject (user's finger or the like) 1 with each lens 11a and 11b. The light shielding member 12 prevents light passing through the lenses 11a and 11b of the lens array 11 from cross-talking on the image pickup surface of the image pickup device 13, and suppresses noise such as ghost and flare.

  The image sensor 13 receives optical images (single-eye images) formed by the lenses 11a and 11b of the lens array 11, converts them into electrical signals, and acquires a compound eye image composed of two single-eye images. Specifically, a CMOS image sensor, a CCD image sensor, or the like in which the photoelectric elements 13a are two-dimensionally arranged is used. The image sensor 13 includes a circuit that adjusts the gain of the electrical signal (analog signal) converted by the photoelectric element 13a or converts the analog signal into a digital signal, and outputs a compound eye image as digital data.

  The cover glass 14 is for protecting the outer surface of the compound-eye imaging unit 10, and images the subject (such as a user's finger) 1 through the cover glass 14.

  FIG. 3 shows a configuration diagram of the second embodiment of the compound-eye imaging unit 10. In FIG. 3, the same parts as those in FIG. In the compound eye imaging unit 10, the lens array 11 has a configuration in which a large number of lenses are arranged, and acquires a compound eye image composed of a large number of single-eye images of a subject image formed by each lens. The advantages of FIG. 3 will be described later.

  2 and 3, the lens of the lens array 11 is shown as a plano-convex lens that is convex toward the image sensor 13, but it may be a double-convex lens or a plano-convex lens that is convex toward the subject. It is. However, by making the image pickup element 13 side convex, the lens array 11 and the light shielding member 12 can be accurately aligned easily without using a highly accurate assembly device, and the assembly cost can be reduced. In addition, the device cost of the lens array can be reduced as compared with the case of a double-sided convex lens, and good imaging performance can be obtained over a wide angle of view compared to the case where the subject side is a convex surface.

  Next, the shift amount calculation unit 20, the shift amount determination unit 30, and the movement amount calculation / output unit 40 will be described.

  The compound eye imaging unit 10 continuously images the subject (user's finger or the like) 1 and outputs a compound eye image that is a set of a plurality of single-eye images in time series. Here, since each single-eye image is an image of a subject formed by lenses having different positions, there is an image shift (parallax) between the single-eye images. The parallax changes depending on the distance between the lens array 11 of the compound-eye imaging unit 10 and the subject 1. Therefore, for each time-series frame, by calculating the image shift amount between single-eye images (for example, between adjacent single-eye images) in the compound-eye image, the compound eyes of the subject (such as the user's finger) are calculated based on these image shift amounts. Movement in a direction perpendicular to the imaging unit 10 can be measured.

  FIG. 4 is a schematic diagram for explaining the relationship between the subject distance and the image shift amount between adjacent single-eye images. In FIG. 4, among the light emitted from the point P where the optical axis of the lens 11 a with the lens array 11 intersects the subject 1, the light passing through the lens 11 a is the optical axis of the lens 11 a and the imaging surface of the imaging device 13. An image is formed at a point Oa where the two intersect. On the other hand, the light that has passed through the adjacent lens 11 b forms an image at a point P ′ on the imaging surface of the imaging element 13. When Ob is a point where the optical axis of the lens 11b and the imaging surface of the image sensor 13 intersect, P'Ob is an image shift amount between monocular images formed by the lenses 11a and 11b. When the lens pitch d known at the time of designing the optical system and the distance f between the lens array 11 and the imaging surface are used, there is a relation of the following equation between the subject distance L and the image shift amount P′Ob between adjacent monocular images. is there.

  The shift amount calculation unit 20 receives a compound eye image output in time series from the compound eye imaging unit 10 and pays attention to two single-eye images (for example, adjacent single-eye images) in the compound-eye image for each frame. An image shift amount between the single-eye images is calculated.

  It should be noted that the region of the single image can be recognized by a simple binarization process. That is, the single-eye image in the compound eye image is an area surrounded by the shadow portion of the light shielding member 12, but the shadow portion is sufficiently darker than the other portions. Therefore, by binarizing a compound eye image using an appropriate threshold value, a shadow portion is distinguished from other portions, and an area surrounded by the shadow portion is easily extracted as a single eye image region. be able to.

For example, pattern matching can be used to calculate the image shift amount. A single-eye image (reference single-eye image) serving as a shift reference is arbitrarily selected from the compound-eye image, and a part of the single-eye image is used as a template. The image shift amount in the horizontal and vertical directions of the reference single-eye image is (0, 0). By selecting a single image (shift detected single image) such as an adjacent single image to be calculated for the shift amount, the similarity or difference between the reference single image and the shift detected single image is determined. An image shift amount is calculated. As is well known, a cross-correlation function can be used as the similarity, and SSD (the sum of squares of pixel value differences) or SAD (the sum of absolute values of pixel value differences) is used as the difference. can do. An example of the degree of difference calculated by pattern matching using SSD is shown in FIG.
In general, the degree of difference using SSD is expressed by equation (2).

ここで、ＲはＳＳＤで算出した相違度、Ｉは被シフト検出個眼画像の一部、Ｔはテンプレートとした基準個眼画像の一部である。ｉ，ｊはそれぞれテンプレートの横方向と縦方向の画素数、ｘ，ｙはシフトの探索範囲で変化する値である。Ｒが最小となる（ｘ，ｙ）が２つの個眼画像の間の画像シフト量となる。図５において、ｘ軸およびｙ軸は画像シフト、ｚ軸がＳＳＤ相違度である。前記（２）式で求まる画像シフト量は画素単位であるが、画素単位で算出した類似度や相違度を関数フィッティングする手法や、テンプレートあるいは被シフト検出個眼画像のどちらかを補間してパターンマッチングする手法などにより、サブピクセル単位でのシフト量算出が可能である。

Here, R is the dissimilarity calculated by SSD, I is a part of the shift-detected single-eye image, and T is a part of the reference single-eye image as a template. i and j are the numbers of pixels in the horizontal and vertical directions of the template, respectively, and x and y are values that change in the shift search range. (X, y) where R is the minimum is the image shift amount between the two single-eye images. In FIG. 5, the x-axis and y-axis are image shifts, and the z-axis is SSD dissimilarity. The image shift amount obtained by the above equation (2) is in units of pixels, but a pattern is obtained by interpolating either a template or a shift-detected single-eye image by a function fitting method of similarity or difference calculated in units of pixels. The shift amount can be calculated in units of subpixels by a matching method or the like.

  When the compound-eye image unit 10 is configured as shown in FIG. 3, a plurality of pairs of two single-eye images of interest are selected (for example, selected without being covered), and an image shift amount is calculated for each group. In addition, the average value of these values can be used as the image shift amount of the frame, thereby increasing the calculation accuracy of the image shift amount.

  The shift amount calculation unit 20 sends the calculated image shift amount (current frame shift amount) to the shift amount determination unit 30 and the shift amount storage unit 50.

  The shift amount storage unit 50 is a buffer memory that temporarily holds the image shift amount. The shift amount storage unit 50 is composed of, for example, a FIFO memory, holds at least the image shift amount (previous frame shift amount) sent from the shift amount calculation unit 20 at the previous time, and discards the previous ones in chronological order. To.

  The shift amount determination unit 30 determines whether or not the image shift amount (current frame shift amount) sent from the shift amount calculation unit 20 is within a predetermined range, and outputs the determination result. In the present embodiment, it is assumed that the image shift amount (current frame shift amount) is compared with three threshold values, and four determination results (flags) shown in Table 1 (cases 0 to 3) are output. Of course, this is just an example.

  FIG. 6 shows the relationship between the distance of the subject (for example, the user's finger) and the thresholds A, B, and C. Note that the thresholds A, B, and C are C <B <A because the subject distance and the image shift amount are inversely proportional to each other according to the above equation (1).

  The shift amount determination unit 30 sends the determination result to the movement amount calculation / output unit 40 and the shift amount storage unit 50. The moving amount calculation / output unit 40 also sends the image shift amount (current frame shift amount) at that time.

  Based on the determination result from the shift amount determination unit 30, the shift amount storage unit 50 reads the held previous frame shift amount and sends it to the movement amount calculation / output unit 40 only in case 1.

  The movement amount calculation / output unit 40 calculates and outputs the movement amount of the subject according to the determination result from the shift amount determination unit 30. Also, depending on the determination result, predetermined movement information is continuously output or nothing is output instead of the movement amount. In the present embodiment, movement information as shown in Table 2 is output according to the determination result of Table 1.

ここで、ケース１では、シフト量判定部３０を介してのシフト量算出部２０からの現フレームシフト量とシフト量記憶部２０からの前フレームシフト量に、それぞれ例えば、式（１）などを適用して被写体距離を求め、その差分等から被写体（ユーザの指等）の移動量を算出する。あるいは、直接、現フレームシフト量と前フレームシフト量の差分や比に適当な係数を乗じて移動量に対応させることも可能である。この場合、被写体距離の算出が不要となる。

Here, in case 1, the current frame shift amount from the shift amount calculation unit 20 and the previous frame shift amount from the shift amount storage unit 20 via the shift amount determination unit 30 are respectively expressed by, for example, Equation (1) and the like. The distance to the subject is obtained by application, and the amount of movement of the subject (such as the user's finger) is calculated from the difference. Alternatively, the difference or ratio between the current frame shift amount and the previous frame shift amount can be directly multiplied by an appropriate coefficient to correspond to the movement amount. In this case, it is not necessary to calculate the subject distance.

  FIG. 7 shows an overall processing flow of the movement measuring apparatus of FIG. Such a processing flow is configured by a computer program, and the function of each unit in FIG. 1 is realized by causing the computer to execute the program.

  The compound eye imaging unit 10 continuously photographs a subject (such as a user's finger), sequentially acquires compound eye images that are a set of a plurality of single-eye images, and outputs them. The shift amount calculation unit 20 inputs the compound eye image output from the compound eye imaging unit 10 in time series (step 110), and for each frame, two single-eye images (for example, adjacent single-eye images) in the compound-eye image. ) To calculate and output an image shift amount (parallax) between the single-eye images (step 120). In the case where the compound-eye imaging unit 10 is configured as shown in FIG. 3, a plurality of sets of two single-eye images of interest are selected (for example, selected without exhaustion) from a large number of single-eye images, The image shift amount may be calculated every time, and the average value thereof may be used as the image shift amount of the frame.

  The shift amount storage unit 50 stores the image shift amount output from the shift amount calculation unit 20 (step 130). The shift amount storage unit 50 holds at least the image shift amount of the previous frame (previous frame shift amount).

  The shift amount determination unit 30 determines whether the image shift amount output from the shift amount calculation unit 20 is within a predetermined range, and outputs the determination result (flag) (step 140). For example, the determination result shown in Table 1 is output. The shift amount determination unit 30 also outputs the image shift amount (current frame shift amount) at that time.

  The shift amount storage unit 50 reads the held image shift amount (previous frame shift amount) if necessary based on the determination result of the shift amount determination unit 30 (step 150). In the example of Table 1, the previous frame shift amount is read in case 1.

  The movement amount calculation / output unit 40 calculates and outputs the movement amount of the subject according to the determination result of the shift amount determination unit 30 (step 160). Depending on the determination result, predetermined movement information is continuously output instead of the movement amount. In the example of Table 1, movement information as shown in Table 2 is output. Here, in case 1, the movement amount is calculated from the current frame shift amount from the shift amount calculation unit 20 and the previous frame shift amount from the shift amount storage unit 50 via the shift amount determination unit 30, and output. To do.

  According to the present embodiment, it is possible to directly reflect the movement of the cursor or the like in the depth direction in a three-dimensional application.

Next, an application example of the movement measuring apparatus of the present invention will be described.
FIG. 8 shows an example of a system when the mobile measuring device of the present invention is used as an input device of an information device such as a personal computer. In FIG. 8, 1000 is an input device as a movement measuring device of the present invention, 1100 is a computer as information equipment such as a personal computer, and 1200 is a display device (display). For example, the input device 1000 is installed on the operation unit of the computer 1200. The display device 1200 displays three-dimensional information.

  The user moves a finger or the like in the vertical direction with respect to the input device 1000 while viewing the screen of the display device 1200. The input device 1000 measures the movement of the subject 1 such as the user's finger and outputs movement information as shown in Table 2, for example. The computer 1100 inputs movement information from the input device 1000, and operates the three-dimensional information on the display device 1200 according to the movement information.

  FIG. 9 shows an example of a three-dimensional application displayed on the display device 1200. The user moves a finger or the like in the vertical direction with respect to the input device 1000 while viewing the display screen. The input device 1000 measures the movement of the user's finger or the like and outputs movement information. The computer 1100 performs operations such as switching the display position of the screen in the depth direction on the display device 1200 according to the movement information from the input device 1000. In this case, the movement of the user's finger or the like in the direction perpendicular to the input device 1000 is directly reflected in the depth direction of the screen on the display device 1200.

  FIG. 10 schematically shows another example of the three-dimensional application displayed on the display device 1200. In this example, the computer 1100 moves the person object in the three-dimensional virtual space in the arrow direction according to the movement information from the input device 1000. In addition, as the person object goes farther, the screen in that direction is enlarged and displayed. Conversely, as the person object comes closer, the person object is reduced and displayed. Also in this case, the movement of the user's finger in the direction perpendicular to the input device 1000 is directly reflected in the depth direction of the three-dimensional virtual space on the display device 1200.

  In the system of FIG. 8, the input device 1000 includes only the compound eye imaging unit 10 shown in FIGS. 2 and 3, and the shift amount calculation unit 20, the shift amount determination unit 30, and the movement amount calculation / output shown in FIG. The processing function of the unit 40 may also serve as a processing unit (CPU) of the computer 1100, and the shift amount storage unit 50 may also serve as a memory (RAM or the like) in the computer 1100.

  As mentioned above, although one embodiment of the present invention was described, of course, the present invention is not limited to what has been described so far, and various modifications can be made within the scope of the following claims.

DESCRIPTION OF SYMBOLS 10 Compound eye imaging part 20 Shift amount calculation part 30 Shift amount determination part 40 Movement amount calculation and output part 50 Shift amount memory | storage part

JP 2008-34948 A

Claims (13)

A compound eye that is a set of single-eye images formed by a plurality of lenses of the lens array with respect to a lens array in which a plurality of lenses are arranged substantially in a plane and a subject that moves in a direction perpendicular to the lens array A compound eye imaging means comprising an imaging device for acquiring an image;
Shift amount calculation means for inputting compound eye images acquired by the compound eye imaging means in time series, and sequentially calculating an image shift amount between the single eye images of the compound eye image;
The movement amount calculation for calculating the movement amount of the subject based on the image shift amount at the first time calculated by the shift amount calculation means and the image shift amount at the second time different from the first time. Means,
A movement measuring device comprising: The compound eye imaging means is an image sensor that acquires a compound eye image that is a set of two single-eye images respectively formed by a lens array in which two lenses are arranged in a substantially planar shape and the two lenses of the lens array. And consist of
The shift amount calculating means calculates an image shift amount between two single-eye images of the compound eye image acquired by the compound eye imaging means;
The movement measuring device according to claim 1. The compound eye imaging means is an image sensor that acquires a compound eye image that is a set of a lens array in which a large number of lenses are arranged in a substantially planar shape and a large number of single-eye images respectively formed by the large number of lenses of the lens array And consist of
The shift amount calculating means selects a plurality of sets of two single-eye images from the compound-eye images acquired by the compound-eye imaging means, calculates an image shift amount between the single-eye images for each set, and averages them The value is finally set as the image shift amount.
The movement measuring device according to claim 1. A shift amount determination unit that determines whether the image shift amount calculated by the shift amount calculation unit is within a predetermined range;
The movement amount calculation means calculates the movement amount of the subject according to the determination result of the shift amount determination means.
The movement measuring device according to any one of claims 1 to 3, wherein The movement amount calculation means calculates and outputs the movement amount of the subject when the image shift amount is within a predetermined range, and when the image shift amount is less than or more than the range, movement information in the back direction or the front direction. Output continuously,
The movement measuring apparatus according to claim 4, wherein: The movement amount calculating means multiplies a difference or ratio between the image shift amounts between the first time and the second time by a predetermined coefficient to correspond to the movement amount.
The movement measuring device according to any one of claims 1 to 5, wherein A compound eye that is a set of single-eye images formed by a plurality of lenses of the lens array with respect to a lens array in which a plurality of lenses are arranged substantially in a plane and a subject that moves in a direction perpendicular to the lens array A movement measurement method for obtaining a movement amount of the subject by inputting the compound eye image in time series from a compound eye imaging device including an image pickup device for acquiring an image,
A shift amount calculating step for sequentially calculating an image shift amount between the single-eye images of the compound eye images input in time series;
A movement amount for calculating the movement amount of the subject based on the image shift amount at the first time and the image shift amount at a second time different from the first time, calculated in the shift amount calculation step. A calculation step;
The movement measuring method characterized by having. The compound eye image pickup device acquires a compound eye image that is a set of two single-eye images respectively formed by a lens array in which two lenses are arranged substantially in a plane and two lenses of the lens array. And consist of
The shift amount calculating step calculates an image shift amount between two single-eye images of the compound-eye image input from the compound-eye imaging device.
The movement measuring method according to claim 7. The compound-eye imaging device includes a lens array in which a large number of lenses are arranged in a substantially planar shape, and an imaging element that acquires a compound-eye image that is a set of a large number of single-eye images formed by the large number of lenses of the lens array And consist of
The shift amount calculating step selects a plurality of sets of two single-eye images from the compound-eye image input from the compound-eye imaging device, calculates an image shift amount between the single-eye images for each set, The average value is finally used as the image shift amount.
The movement measuring method according to claim 7. A shift amount determination step of determining whether or not the image shift amount calculated in the shift amount calculation step is within a predetermined range;
The movement amount calculating step calculates a movement amount of the subject according to a determination result of the shift amount determination step.
The movement measuring method according to claim 7, wherein: The movement amount calculating step calculates and outputs the movement amount of the subject when the image shift amount is within a predetermined range, and when the image shift amount is less than or more than the range, movement information in the back direction or the front direction. Output continuously,
The movement measurement method according to claim 10. The movement amount calculating step multiplies a difference or ratio between the image shift amounts between the first time and the second time by a predetermined coefficient to correspond to the movement amount.
The movement measuring method according to claim 7, wherein:   The program for making a computer perform each step of the movement measurement method of any one of Claims 7 thru | or 11.

Images