JP5464661B2 - Information terminal equipment - Google Patents

Description

  The present invention relates to an information terminal device that presents information, and more particularly to an information terminal device that can control display information on a display unit by changing a relative positional relationship between an imaging unit and an imaging target.

  The device that changes the position of the imaging target with respect to the imaging unit and controls the display information according to the relative position of the imaging unit and the imaging target can intuitively change the display information. This is useful for improving convenience.

  As a technique for realizing this, Japanese Patent Application Laid-Open No. H10-228667 describes a marker unit composed of a plurality of marker elements that are provided with a color that approximates the color of the object and that constitutes identification information by a combination of its outer shape and arrangement. There has been proposed a mixed reality system that performs positioning when displaying virtual information superimposed on an object in a real environment by arranging and analyzing the camera image of the object.

  Patent Document 2 proposes an image composition device that generates a composite image obtained by superimposing a measurement range of a position and orientation sensor and a camera on a real image in which a position and orientation sensor is arranged.

  Patent Document 3 discloses a function of scrolling or zooming a displayed portion according to the direction and amount of movement and tilt detected by a motion sensor, and stopping scrolling or zooming by operating a predetermined key for key input. A portable device having the above has been proposed.

  Non-Patent Document 1 proposes a method of detecting a fingertip from an image of a hand imaged by a camera and aligning display information with the detected hand location using an ellipse fitting technique. In this case, a portion having a large curvature is detected as the fingertip of the hand.

JP 2009-020614 A JP 2007-233971 A JP 2009-003799 A

T. Lee, et al., "Handy AR: Markerless Inspection of Augmented Reality Objects Using Fingertip Tracking," In Proc. IEEE International Symposium on Wearable Computers, pp. 83-90, Oct. 2007.

  In the mixed reality system of Patent Document 1, the position of virtual information superimposed and displayed in the real environment is controlled by the position of the marker unit in the camera image, but a plurality of artificial marker elements are arranged in advance on the object. There is a need. For this reason, there is a problem that places where the method of the mixed reality system can be used are limited.

  The image synthesizing apparatus of Patent Document 2 requires a position and orientation sensor. In addition, the portable device disclosed in Patent Document 3 requires a motion sensor. For this reason, there exists a subject that the apparatus which can utilize the method in these is limited. In addition, mounting sensors does not only increase the cost of the terminal, but also makes it difficult to reduce the size and power consumption of the device.

  In Non-Patent Document 1, since a part having a particularly large curvature is detected as a fingertip, it can be correctly displayed only in a state where the finger is straightly extended. The natural hand shape has a problem that the finger is slightly bent and the fingertip has a large movement, so that the positioning accuracy is not sufficient and the matching is difficult. There is also a problem that the fingertip cannot be detected correctly when the finger is closed.

  The object of the present invention is to solve the above-mentioned problems, and to change the spatial position and orientation of the imaging target with respect to the imaging unit without using a marker or a sensor, and reliably and accurately display information on the display unit. It is to provide an information terminal device that can be controlled easily.

In order to solve the above-described problem, the present invention is an information terminal device having an imaging unit that captures an imaging target using a hand as an imaging target, and uses a feature point extracted from an image input from the imaging unit as a reference An estimation unit that preliminarily sets the imaging target in the position and orientation and estimates the change in the position and orientation of the imaging target in association with a reference point that indicates a change in the overall position and orientation of the imaging target, and displays information A display unit, a storage unit that stores information to be displayed on the display unit, and information that is read from the storage unit and displayed on the display unit according to changes in the position and orientation of the imaging target estimated by the estimation unit The reference point is set inside or around the palm, and the estimation unit is configured to provide a boundary part between fingers from a skin region in an image input from the imaging unit, a boundary between the finger and the palm. portion Rui are basically characterized that it has a function of extracting feature points existing in the skin area after eliminating wrinkles portion.

Here, the reference points are, for example, end points of the heels such as the crotch and joint of the finger, the points outside the second joint of the thumb, and the outside of the second joint of the thumb with respect to the longitudinal line at the center of the palm. It can be set to a point corresponding to this point.

  According to the present invention, it is possible to control display information on the display unit only by changing the relative position and orientation of the imaging unit and the imaging target. Therefore, the user can control the display information by an intuitive operation of changing the relative position and posture of the imaging target with respect to the imaging unit.

  The display information can be controlled by analyzing the image input to the imaging unit and controlling the display information based on changes in the relative position and orientation of the imaging target with respect to the imaging unit. There is no need to incorporate special hardware of sensors into the information terminal.

  Furthermore, in the present invention, the feature point extracted from the image input from the imaging unit is associated with the reference point preset for the imaging target at the reference position and orientation, and the change in the position and orientation of the imaging target is performed. Since the reference point is set to a point indicating the change in the overall position and orientation of the imaging target, the change in the position and orientation of the imaging target can be estimated reliably and with high accuracy. Information can be reliably and accurately controlled.

It is a functional block diagram which shows one Embodiment of the information terminal device of this invention. It is a flowchart which shows the process sequence in an estimation part. It is a figure which shows the example of the reference point set beforehand. It is a figure which shows an example of the skin area | region extracted from the image, the circumscribed rectangle which includes this skin area | region, and a feature point. It is a figure which shows the example of the skin area | region extracted from the image, and the separated skin area | region. It is a figure which shows the example of a display by this invention. It is a figure which shows the other example of a display by this invention. It is a figure which shows the other example of a display by this invention.

The present invention will be described below with reference to the drawings. Below, the case where a mobile telephone is utilized as an information terminal device is demonstrated. However, the information terminal device of the present invention is not limited to a mobile phone, and may be any information terminal device provided with an imaging unit, such as a computer .

  FIG. 1 is a functional block diagram showing an embodiment of the information terminal device of the present invention. The information terminal device of this embodiment includes an imaging unit 11, an estimation unit 12, a control unit 13, a storage unit 14, and a display unit 15.

  The imaging unit 11 continuously captures an imaging target (hand) at a predetermined sampling period, and outputs the images to the estimation unit 12 and the control unit 13. As the imaging unit 11, a digital camera provided as a standard in a mobile phone can be used.

  In the estimation unit 12, a reference point is registered in advance with respect to an imaging target at a known position and orientation as a determination criterion for changes in the position and orientation of the imaging target. Here, the reference point is set to indicate a change in the overall position and orientation of the imaging target.

  The estimation unit 12 detects a feature point corresponding to the reference point from the image input from the imaging unit 11, and estimates the correspondence between the feature point and the reference point. Further, the estimation unit 12 estimates a conversion coefficient representing the relative position and orientation of the imaging target with respect to the imaging unit 11 from the coordinate positions of the corresponding feature points and reference points based on a preset conversion equation. The conversion equation used for estimating the conversion coefficient in the estimation unit 12 and the estimated conversion coefficient are output to the control unit 13. Details of the processing in the estimation unit 12 will be described later.

  The storage unit 14 stores a plurality of display information to be displayed on the display unit 15 in advance. The user can select arbitrary display information from the display information stored in the storage unit 14 and display it on the display unit 15 by an input operation on the control unit 13.

  When displaying information on the display unit 15, the control unit 13 processes the display information by applying the conversion formula and the conversion coefficient input from the estimation unit 12 to the display information. Thereby, the display information on the display unit 15 is controlled according to the estimation result in the estimation unit 12.

  FIG. 2 is a flowchart showing a processing procedure in the estimation unit 12. The estimation unit 12 repeatedly executes the region formation process S21, the feature detection process S22, and the posture estimation process S23 sequentially at predetermined timing intervals.

  First, in region formation processing S21, a skin region is extracted from an image input from the imaging unit 11. The skin region can be extracted based on the color information in the image input from the imaging unit 11. For example, when an image input from the imaging unit 11 is expressed in the RGB color space, it is converted into an expression in the HSV color space (brightness V, saturation S, hue H). The conversion from the RGB color space expression to the HSV color space expression can be performed by equations (1) to (3). The skin region can be extracted by extracting pixels in which the hue H falls within a preset range (that is, within the skin color range in the hue H of 0 ° to 360 ° on the HSV color space).

  The extraction of the skin region is not limited to the above method based on the position of the color information in the HSV color space, and other methods are possible. For example, the skin region can be determined and extracted based on the likelihood of the skin using a Gaussian mixture model (GMM). When using the GMM, learning is performed in advance so that the probability distribution of the skin and the non-skin is configured by the sum of a plurality of Gaussian distributions expressed by Expression (4).

Here, x and D respectively represent the feature amount of the pixel and the number of dimensions thereof, and N represents the number of Gaussian distributions. The pixel feature amount may be pixel information itself. Each Gaussian distribution has a weight coefficient w _i , and μ _i and Σ _i are an average value and a covariance matrix, respectively. A maximum likelihood estimation method such as an EM algorithm can be used to determine the parameters of the Gaussian distribution.

Let P (x | skin) be the probability that a feature quantity x occurs in a skin area, and P (x | ¬skin) the probability that a feature quantity x occurs in a non-skin area. The parameters of the respective probability calculation functions, that is, the number N of Gaussian distributions, the average value μ _i of each Gaussian distribution, the covariance matrix Σ _i , and the weighting coefficient w _i are recorded as learning results.

  The user's skin information may be reflected in the probability distribution. In this case, the probability distribution is corrected with pixel information extracted from the skin area of the user separately from the learning data. The skin probability distribution learned from the learning data is Pg (x | skin), the non-skin probability distribution is Pg (x | ¬skin), the skin probability distribution learned from the user's skin is Pu (x | skin), and the non-skin Assuming that the probability distribution is Pu (x | ¬skin), the skin probability distribution P (x | skin) reflecting the user's skin information and the non-skin probability distribution P (x | ¬skin) are expressed by the equations (5), ( Given in 6). It is also possible to obtain the skin probability distribution and the non-skin probability distribution by learning using only the user's skin information.

  When the GMM is used and the skin information of the user is reflected in the probability distribution, in the region forming process S21, when an image is input from the imaging unit 11, a preset threshold TH1 and an expression of a learning result ( Using 5) and (6), a pixel in which the ratio of skin probability distribution P (x | skin) and non-skin probability distribution P (x | ¬skin) satisfies equation (7) is defined as a skin region.

  Next, in the feature detection process S22, a point corresponding to a preset reference point is extracted from the image input from the imaging unit 11 as a feature point, and the coordinates of the feature point are detected. The reference point can reliably identify the overall position and posture change of the imaging target (hand) so that the posture estimation in the subsequent posture estimation process S23 is robust against finger bending and finger opening / closing. Set to point. For example, it is preferable to set a plurality of points separated from each other around the palm, such as the crotch of a finger, as reference points. For example, it is also preferable to set the reference point at a point on the opposite side of the point on the outer side of the second joint of the thumb and the longitudinal line in the center of the palm. These points and the crotch of the finger may be combined as a reference point.

  FIG. 3 is a diagram illustrating an example of preset reference points. Here, (1) is a reference point set to the crotch of the thumb, and (2) to (4) are reference points set to the crotch of the four fingers. (5) is the reference point set outside the second joint of the thumb, and (6) is set on the opposite side of the reference point (5) with respect to the vertical line at the center of the palm. This is the reference point. It is not necessary to set all of these reference points (1) to (6), and at least four reference points may be selected and set. In this case, as described above, it is preferable to select points separated from each other.

  The feature extraction process S22 will be described in more detail. In the feature extraction process S22, the skin area extracted in the area formation process S21 is shaped, and feature points in or around the skin area are extracted. Only the points corresponding to the reference points need to be extracted as feature points, but more points than the number of reference points are extracted depending on the conditions for extraction. In this case, a point having the same feature amount as the reference point may be selected as a feature point corresponding to the reference point from the extracted points.

  Before extracting feature points, it is preferable to compensate for partial defects in the skin region by a morphological filter. A partial defect in the skin area is caused by light exposure to the hand. Next, a circumscribed rectangle that includes the skin region and has the smallest area is formed. For calculation of the circumscribed rectangle, principal component analysis or the like can be used. This circumscribed rectangle is used for separation of fingers described later. The circumscribed rectangle can also be used to limit the range from which feature points are extracted.

  FIG. 4 is a diagram illustrating an example of a skin area extracted from an input image, a circumscribed rectangle that includes the skin area, and feature points corresponding to the reference points (1) to (6) in FIG.

  As shown in FIG. 4, since the hand that is the imaging target may be closed, in the feature extraction process S22, a process of separating fingers from each other is performed before the feature points are extracted. Separation between fingers can be realized by applying edge detection to detect an edge portion with a low skin color level, and further using the circumscribed rectangle to eliminate the edge portion between each finger to make a non-skin region. .

  It can be assumed that the major axis direction of the circumscribed rectangle including the skin region substantially coincides with the finger direction. Therefore, among the detected edge portions, the edges of the circumscribed rectangle that includes the skin region with respect to the major axis direction are within the predetermined threshold range, and the fingers are placed between the fingers. Can be separated.

  FIG. 5 is a diagram illustrating an example of a skin region (a) extracted from an input image and a skin region (b) separated by applying an edge detection technique. Between each finger, since the skin color level is low, it is set as a non-skin area.

  Finally, feature points are extracted at the boundaries of the separated skin regions. Extract the same number of feature points as the reference points using existing methods such as SIFT and SURF. It is also possible to extract more feature point candidates (end points) than the number of reference points, and select those satisfying a predetermined condition from among them as the feature points as follows.

  For the end points of the crotch part of the finger, the number and ratio of the skin areas are calculated for each small area including the extracted end points, and the ratio of the skin area is single and the ratio of the skin area is within a preset threshold range. , Selected as a feature point. Even if the end points of the non-skin region of the finger joint part are extracted as feature point candidates, since these two end points have two skin regions in the vertical direction of the joint, the end points are not selected as feature points. In addition, since the proportion of the skin region does not fall within a preset threshold range at the fingertips, the end points of the heel portion, etc., they are not selected as feature points.

  The feature point on the outer side of the second joint of the thumb and the feature point on the opposite side to the longitudinal line in the center in the lateral direction of the palm has the maximum line length parallel to the short axis of the circumscribed rectangle in the skin region. Are extracted as two points that intersect the skin region boundary.

  In the posture estimation process S23, the extracted feature point is associated with a preset reference point, and the relative position and posture of the imaging target with respect to the imaging unit 11 are estimated by comparing the coordinates of both.

  For this purpose, first, the extracted feature points are associated with preset reference points (1) to (6). The feature point opposite to the feature point outside the second joint of the thumb with respect to the vertical line at the center of the palm in the lateral direction can be determined as the feature point farthest from the center of gravity of the convex polygon formed by the plurality of feature points . This feature point is associated with a reference point (6) on the opposite side of the reference point outside the second joint of the thumb with respect to the vertical line in the center of the palm.

  The feature point outside the second joint of the thumb is the feature point determined as described above (the feature point on the opposite side of the feature point outside the second joint of the thumb relative to the vertical line in the center of the palm side) ) And a feature point located on or near the line segment extending in parallel with the short axis of the circumscribed rectangle. This feature point is associated with a reference point (5) outside the second joint of the thumb.

  The feature point of the thumb crotch can be determined as the feature point of the location closest to the outside of the second joint of the thumb, and can be associated with the reference point (1) outside the second joint of the thumb.

  The feature points of the four-finger crotch can be determined as the feature points of the most crowded points when the feature points are projected in the short axis direction of the circumscribed rectangle that includes the skin area. Corresponding to the reference points (2) to (4) of the crotch between the fingers in order of increasing distance.

As a result, the coordinates of the n feature points associated with each other are expressed as (x ′ _j , y ′ _j ) (1 ≦ j ≦ n), and the coordinates of the reference point are (x _j , y _j ) (1 ≦ j ≦ n). Then, a conversion coefficient a _k (1 ≦ k ≦ m, m ≦ 2n) is determined so that they match in a preset conversion equation.

In the conversion equation using the projective transformation of equations (8) and (9), when m = 2n, the conversion coefficient a _k can be obtained as a 2n simultaneous equation. In the case of m <2n, the conversion coefficient _ak can be _obtained by the least square method.

The conversion coefficient _ak obtained in the posture estimation process S23 is sent to the control unit 13 together with the conversion formula. The control unit 13 applies the conversion formula and the conversion coefficient input from the estimation unit 12 to the display information and processes the display information. Thereby, the display on the display unit 15 is controlled based on the estimation result in the estimation unit 12. For display control in the display unit 15, CG (Computer Graphics) technology can be used.

  For example, when information such as an image list, a map, an application such as a web browser, an address book, or a menu item is displayed on the display unit 15, the user captures an image on the imaging surface of the imaging unit 11 (hand). By moving in the vertical direction or horizontal direction, the display information on the display unit 15 can be moved in conjunction with the position of the imaging target. In addition, when the user moves the imaging target in the depth direction with respect to the imaging surface of the imaging unit 11, the display information on the display unit 15 can be enlarged or reduced in conjunction with the apparent size of the imaging target. it can. Furthermore, when the user tilts the posture of the imaging target, the table information on the display unit 15 can be rotated in conjunction with the tilt of the imaging target.

  6, 7 and 8 are diagrams showing display examples according to the present invention. FIG. 6 is a display example of the image list, and the image list is displayed as if it was put on the hand. Here, if the hand is tilted with respect to the image pickup unit, the image list display can be similarly tilted while being placed on the hand, and if the hand is brought close to or away from the image pickup unit, it is placed on the hand. The image list display can be enlarged or reduced as it is. FIG. 7 is a display example of an address book expressed by a 3D CG avatar. In this case, the address book display can be controlled on the hand.

  Although the embodiment has been described above, the present invention is not limited to the above embodiment. For example, the display information can be controlled by setting a reference point on the back of the hand. FIG. 8 shows a display example in this case. In this case, it is possible to display as if a ring is actually attached.

  In addition, a reference point may be set at the boundary between the finger and palm or the end point of the eyelid. In this case, among the edge portions detected in the feature extraction process S22, an edge portion in which the angle of the edge direction with respect to the minor axis direction of the circumscribed rectangle that includes the skin region falls within a preset threshold range is determined. By using a heel part such as a boundary or a finger joint and making this part a non-skin part, the finger and palm or each finger joint are separated. Thereafter, feature points are extracted at the boundaries of the separated skin regions and associated with the respective reference points.

  In the above embodiment, a circumscribed rectangle that includes the skin region is formed and used to extract feature points and limit the range for extracting feature points. However, this is not limited to the circumscribed rectangle, and is a circumscribed polygon. Good. In this case, the feature point located in the maximum deep part inside the maximum non-skin area that touches the side of the circumscribed polygon (the part where the end point of the edge between the fingers is deepest) can be determined as the crotch of the thumb, and from the relationship with the crotch of the thumb Reference points corresponding to other feature points can be determined.

11 ... Imaging unit, 12 ... Estimation unit, 13 ... Control unit, 14 ... Storage unit, 15 ... Display unit

Claims (17)

An information terminal device having an imaging unit that images a hand as an imaging target ,
The feature points extracted from the image input from the imaging unit are set in advance as imaging targets at the reference position and orientation, and imaged in association with the reference points that indicate changes in the overall position and orientation of the imaging target. An estimation unit for estimating a change in the position and orientation of the target;
A display for displaying information;
A storage unit for storing information to be displayed on the display unit;
A control unit that controls information read from the storage unit and displayed on the display unit according to changes in the position and orientation of the imaging target estimated by the estimation unit ;
The reference point is set inside or around the palm, and the estimation unit excludes a finger-to-finger boundary, a finger-palm boundary, or a heel from a skin area in an image input from the imaging unit. An information terminal device having a function of extracting feature points existing in a skin region from The information terminal device according to claim 1 , wherein the reference point includes a crotch or a joint of a finger and other end points of the heel. The reference point includes one or both of a point outside the second joint of the thumb and a point set on the opposite side of the point outside the second joint of the thumb relative to the longitudinal line in the middle of the palm. The information terminal device according to claim 2 . The estimation unit, according to claim 1, characterized in that it has a function to limit the scope of utilizing includes circumscribed polygon skin region, extracts feature points in the image input from the imaging unit Information terminal device. The estimation unit generates a circumscribed rectangle that includes a skin region in the image input from the imaging unit, and removes the image edge portion of the circumscribed rectangle in the long axis direction from the skin region, thereby removing a boundary portion between fingers. The information terminal device according to claim 1 , having a function of excluding from a skin region. The estimation unit generates a circumscribed rectangle that includes a skin region in an image input from the imaging unit, and excludes an image edge portion of the circumscribed rectangle in the short axis direction from the skin region, thereby demarcating the boundary between the finger and the palm Alternatively, the information terminal device according to claim 1 , which has a function of excluding the heel portion from the skin region. The estimation unit has a function of extracting a skin region from an image input from the imaging unit, and extracting the same number or more points as preset reference points from the skin region as feature point candidates. The information terminal device according to claim 1 . The estimation unit has a function of selecting, as a feature point, a feature point candidate that is suitable for association with a reference point from feature point candidates extracted from an image input from the imaging unit. Item 8. The information terminal device according to Item 7 . The estimation unit, when selecting a feature point candidate that matches a reference point as a feature point, uses the number of skin regions of a small region including the feature point candidate point and a ratio occupied by the skin region. 8. The information terminal device according to 8 . The estimation unit selects, as a feature point, a feature point candidate in which a small skin region including a feature point candidate point is within a single threshold and a ratio of the skin region is within a preset threshold range. The information terminal device according to claim 8 . The estimation unit extracts, as a feature point, a point where a line segment that is parallel to a short axis of a circumscribed rectangle including the skin region in the image input from the imaging unit and has the maximum length in the skin region intersects the skin region boundary. The information terminal device according to claim 3 , wherein the information terminal device has a function of: The estimation unit sets the feature point farthest from the center of gravity of the convex polygon formed by the feature point extracted from the image input from the imaging unit to the second vertical finger line with respect to the vertical line in the middle of the palm. The information terminal device according to claim 1 , wherein the information terminal device has a function of associating with a reference point set on the opposite side of the reference point set outside the joint. The estimation unit includes a skin region through a feature point corresponding to a reference point set on the opposite side of the reference point set on the outer side of the second joint of the thumb with respect to the vertical line in the center of the horizontal direction of the palm. And a feature point located on or near the line segment extending in parallel with the short axis of the circumscribed rectangle is associated with a reference point set outside the second joint of the thumb. 12. The information terminal device according to 12 . When the estimation unit projects the feature points extracted from the image input from the imaging unit in the short axis direction of the circumscribed rectangle that includes the skin region, the feature points of the most dense locations are displayed on the crotch of four fingers. The information terminal device according to claim 1 , having a function of associating with a set reference point. The estimation unit associates the feature points extracted from the image input from the imaging unit with reference points set on the four-finger crotches in order of increasing distance from the thumb crotch feature point, respectively. The information terminal device according to claim 14 . The estimation unit has a function of obtaining a conversion coefficient so that a feature point and a reference point that are associated with each other are matched by a preset conversion formula in order to estimate a change in the position and posture of a finger as an imaging target. The information terminal device according to claim 1 , comprising:   The information terminal according to claim 1, wherein the control unit has a function of controlling display information on the display unit in accordance with a change in position and orientation of the imaging target estimated by the estimation unit. apparatus.

Images