JP2014106603A - Information terminal device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information terminal device for surely and highly accurately controlling display information.SOLUTION: An imaging part 1 obtains an image of a prescribed imaging target, an estimation part 2 extracts a first feature quantity and a second feature quantity of the imaging target from an image and estimates a position, an attitude and a shape, and a control part 3 controls display at a display part 5 according to the estimation result. At this time, a feature extraction part 21 extracts a border line of the imaging target, and a feature correspondence part 22 specifies correspondence between the extracted border line and the imaging target, and specifies the first feature quantity and the second feature quantity. An attitude estimation part 23 estimates the position and the attitude from the correspondence between the reference value of the first feature quantity and the specified value, and a shape estimation part 24 estimates the second feature quantity by adding correction of the estimation result.

Description

  The present invention relates to an information terminal device that presents information on a display unit, and more particularly to an information device terminal that can control display information on a display unit by changing a relative positional relationship between an imaging unit and an imaging target and by changing the imaging target. .

  An apparatus that presents information according to a relative positional relationship with an imaging target can intuitively change the information to be presented, and can improve user convenience. As techniques for realizing such an apparatus, the following are disclosed.

  In Patent Document 1, a marker unit that is provided with a color approximate to the color of an object and that includes a plurality of marker elements that constitute identification information by a combination of its outer shape and arrangement is arranged on the object, and a camera image of the object A mixed reality system that performs alignment when virtual information is superimposed and displayed on an object in a real environment is disclosed.

  Patent Document 2 discloses an image composition device that generates a composite image in which a position and orientation sensor and a measurement range of a camera are superimposed on a real image in which a position and orientation sensor is arranged.

  Patent Document 3 has a function of scrolling or zooming information to be displayed according to the movement and tilting direction and amount detected by the motion sensor, and stopping scrolling or zooming by operating a predetermined key for key input. A portable device is disclosed.

  In Non-Patent Document 1, there is a method in which a part having a large curvature is detected as a fingertip from an image of a hand imaged by a camera, and information is displayed by aligning with the detected hand position using an ellipse fitting technique. It is disclosed.

JP 2009-020614 A JP 2007-233971 A JP 2009-003799 A

Non-Patent Document 1: T. Lee, et al., "Handy AR: Markerless Inspection of Augmented Reality Objects Using Fingertip Tracking," In Proc. IEEE International Symposium on Wearable Computers, pp.83-90, Oct. 2007. 2

  In the mixed reality system of Patent Document 1, the position of virtual information to be superimposed and displayed in the real environment is controlled by the position of the marker unit in the camera image. However, it is necessary to place an artificial marker element on the object in advance. . For this reason, there is a problem that a place where the method in the mixed reality system can be used is limited.

  The image synthesizing apparatus of Patent Document 2 requires a position and orientation sensor. In addition, the portable device disclosed in Patent Document 3 requires a motion sensor. For this reason, there exists a subject that the apparatus which can utilize the method in these is limited. In addition, mounting sensors does not only increase the cost of the terminal, but also makes it difficult to reduce the size and power consumption of the device.

  In the method of Non-Patent Document 1, a portion having a particularly large curvature is detected as a fingertip. Therefore, in order to correctly align and display information, a short-distance and high-resolution image that allows the finger to appear round is required. In addition, since the fingertips are detected using ellipse fitting with a large processing load, there is a problem that it takes too much processing time to detect the fingertips of all fingers.

  The objective of this invention is providing the information terminal device which can control the information displayed on a display part reliably and with high precision, without solving the subject of the said prior art and using a special marker, a sensor, etc.

  In order to achieve the above object, the present invention is an information terminal device having an imaging unit, wherein the imaging unit captures a predetermined imaging target to obtain an image, and the imaging target is determined from the image. Extracting the first feature amount and the second feature amount, estimating the relative positional relationship of the imaging target with respect to the imaging unit based on the first feature amount, and based on the positional relationship and the second feature amount An estimation unit that estimates the shape of the imaging target, a display unit that displays information, a storage unit that stores information to be displayed on the display unit, and information that is read from the storage unit and displayed on the display unit, And a control unit that controls based on the estimated positional relationship and shape.

  According to the present invention, since the display information on the display unit can be controlled simply by changing the position and orientation of the imaging target or changing the shape of the imaging target with respect to the imaging unit, the information terminal device is used. An intuitive operating environment can be provided to the user. Further, since the control is based on image analysis, no special sensor or the like is required. Furthermore, since the control is performed by changing the position, posture, and shape, various controls are possible.

It is a functional block diagram of the information terminal device concerning one embodiment. It is a figure for demonstrating an example of the hand shape as an example of an imaging target, the feature-value of the said hand shape, and the extraction method of the said feature-value. It is a figure for demonstrating an example of the hand shape as an example of an imaging target, the feature-value of the said hand shape, and the extraction method of the said feature-value. It is a figure which shows an example of the control by this invention.

  The present invention will be described below with reference to the drawings. Hereinafter, a case where a mobile terminal is used as the information terminal device and the imaging target is a hand will be described. However, the information terminal device of the present invention is not limited to a portable terminal, and may be any information terminal device provided with an imaging unit, such as a desktop type, a laptop type, or another computer. But you can. In addition, if the image processing has similar characteristics, the present invention can be applied to an imaging target other than the hand by appropriately replacing color information to be used.

  FIG. 1 is a functional block diagram of an information terminal device according to an embodiment of the present invention. The information terminal device 10 includes an imaging unit 1, an estimation unit 2, a control unit 3, a storage unit 4, and a display unit 5. Further, as will be described, the estimation unit 2 includes a feature extraction unit 21, a feature correspondence unit 22, a posture estimation unit 23, and a shape estimation unit 24.

  The imaging unit 1 continuously captures an imaging target at a predetermined sampling period and outputs the images to the estimation unit 2. As the imaging unit 1, a digital camera provided as a standard in a portable terminal can be used.

  In the estimation unit 2, a reference feature amount is registered in advance for an imaging target at a known position and orientation as a criterion for determining the position and orientation of the imaging target. The estimation unit 2 detects a feature amount corresponding to the reference feature amount from the image input from the imaging unit 1 by the feature extraction unit 21, and estimates the correspondence between the two by the feature correspondence unit 22. Next, the estimation unit 2 calculates a coefficient representing the relative position and orientation of the imaging target with respect to the imaging unit based on a preset conversion formula from the coordinate position of the corresponding feature amount and the reference feature amount in the orientation estimation unit 23. Is estimated.

  Further, the estimation unit 2 applies the posture estimation result to the reference feature amount that is not used in the posture estimation process in the shape estimation unit 24, and compares the corresponding feature amount with the amount of change in the shape of the imaging target. Is estimated as shape information. The conversion formula, the estimated coefficient, and the shape information used for estimating the coefficient in the estimation unit 2 are output to the control unit 3.

  Among the above reference feature amounts, those used for position and orientation estimation are referred to as first feature amounts, and those used for shape estimation are referred to as second feature amounts. Details of the processing in the estimation unit 2 will be described later.

  The storage unit 4 stores a plurality of display information to be displayed on the display unit 5 in advance. A user using the information terminal device 10 can select arbitrary display information from the display information stored in the storage unit 4 and display it on the display unit 5 by an input operation on the control unit 3.

  When displaying information on the display unit 5, the control unit 3 controls the display information so as to process the display information by applying the conversion formula and the conversion coefficient input from the estimation unit 2 to the display information. Further, the control unit 3 controls display information according to the shape information input from the estimation unit 2.

  Hereinafter, details of the estimation unit 2 will be described. In the estimation unit 2, the processing of the feature extraction unit 21, the processing of the feature correspondence unit 22, the processing of the posture estimation unit 23, and the processing of the shape estimation unit 24 are sequentially and repeatedly executed for each image input at a predetermined timing interval. . 2 and 3 are diagrams for explaining a hand shape as an example of an imaging target, a feature amount thereof, and an extraction method thereof, and the example will be referred to as appropriate in the description of each process.

In FIG. 2, as shown in (1), the hand shape is configured by grasping the third to fifth fingers using the palm side for imaging and opening the first and second fingers. . In the case of the shape, it is desirable to set the reference feature amount as follows. In this case, keep the gripped part as it is and do not move it, change the position and posture of the palm itself, and change in the state where the first and / or second fingers are almost open. By doing so, control is performed.
First feature amount ... the end point of the boundary line of the third finger to the fifth finger (each line segment group L23, L34, L45 and L50 of (3)), or the coordinates of the first joint and the second joint of each finger (( 3) Points P31, P32, P41, P42, P51 and P52)
2nd feature amount ... fingertip coordinates of the first and / or second finger (points P10 and P20 in (3))

In FIG. 3, as shown in (1), after grasping the hand, the hand shape uses one side of the back of the hand, on which the first and second fingers can be seen, for imaging. In the case of the shape, it is desirable to set the reference feature amount as follows.
The first feature amount is the end point (points P201, P202, P203, P211, P212 and P221 of (4)) of the line segment group (line segment group L200 of (3)) formed on the side surface of the second finger grasped
2nd feature amount ... 1st finger tip coordinates (point P30 in (3))

In addition, each of the segment groups L200 defining the first feature amount in the example of FIG. 3 specifically, as shown in (4), on the side surface of the second finger that is grasped and the joint is folded, is as follows: The line segments L201 to L220 are formed as described above.
Line segment L201: “wrinkle” formed on the side surface of the second finger by the folded first joint (or a boundary line formed by contacting the belly of the finger across the first joint)
Line segment L202: “wrinkle” formed on the side surface of the second finger by the folded second joint (or a boundary line formed by contacting the belly of the finger across the second joint)
Line segment L203: "wrinkle" formed on the side surface of the second finger by the folded third joint (or a boundary line formed by contact of the belly and base portion of the finger across the third joint)
Line segment L210: a boundary line formed by the contact between the belly of the finger on the tip side from the first joint and the belly of the finger between the second joint and the third joint Line segment L220: on the tip side of the first joint Boundary formed by the contact between the belly of the finger and the belly of the third joint base

(Processing of the feature extraction unit 21)
First, the feature extraction unit 21 extracts the feature amount of the imaging target corresponding to the reference feature amount from the image input from the imaging unit. As an example of boundary line extraction, first, edge detection such as Canny is performed, and noise removal and edge widening are performed. Next, the edge component coordinates are converted into a polar coordinate system by Hough transform. Subsequently, a large number of polar coordinate points having the same coordinates are detected as straight lines in the orthogonal coordinate system, and both ends are extracted as end point coordinates of the boundary line. The extracted example is shown in (2) as a line segment extracted from the image (1) in FIGS.

Finally, in order to be able to make a stable determination by subsequent feature correspondence processing, a correct feature amount is selected as a boundary line from the extracted line segments. In the case of selecting the hand shape shown in FIG. 2 for selection, it can be imposed on the condition that all or part of the following [1] to [5] are satisfied. In the case of FIG. 3, [3] can be excluded and all or a part of [1], [2], [4], and [5] can be satisfied as a condition.
[1] The color of the line is smaller than a predetermined threshold
[2] The length of the line segment is longer than a predetermined threshold
[3] There are parallel line segments in the vicinity (for parallel, the threshold is determined by the angle of the line segment)
[4] Color change in the vicinity of the line segment is smaller than a preset threshold
[5] Density of multiple line segments is less than a predetermined threshold

  Note that the above [4] is based on the fact that the boundary line is formed in the palm of the hand, so that the palm is generally skin-colored. When imposing [4], after detecting a palm region based on a predetermined color feature, a line segment may be selected for the palm region and the region boundary. In addition, when selecting the line segment corresponding to [4], information on whether the line segment is formed inside the palm area or at the boundary between the palm area and other areas is acquired. You may keep it. The “near area” of [4] is preferably limited to the inside of the palm, especially when selecting a line segment formed at the boundary.

  According to the conditions, as conceptually shown in (3) of FIG. 2, the line segment groups L10 and L11 as the boundary line of the first finger, the line segment groups L20 and L21 as the boundary line of the second finger, and the first Line segment groups L23 to L50 as the boundary lines of the third to fifth fingers described in the one feature amount are obtained. In particular, in [3] above, by making the angle formed by each line segment within the line segment group fall within a predetermined range, a line segment group corresponding to each boundary line can be obtained.

  In addition, as an example of extracting the coordinates of joints, fingertips, etc. in the first and second feature quantities, after extracting the boundary line, the center line between the boundary lines is closer to other edge components than a preset threshold value. The extracted coordinates are extracted as joint coordinates or fingertip coordinates. Alternatively, since blood flow is different from other parts in the joint part, when the change in luminance or color difference is seen along the center line, coordinates that take extreme values are extracted as joint coordinates.

  In (3) of FIG. 2, for example, points P31 and P32 are extracted as joint coordinates as locations that are close to the edge component or locations that take extreme values on the center line M30 of the boundary lines L23 and L34. Similarly, points P41 and P42 are extracted from the center line M40 of the boundary lines L34 and L45, and points P51 and P52 are extracted from the center line M50 of the boundary lines L45 and L50 as joint coordinates. Similarly, a point P10 is extracted from the center lines (not shown) of the boundary lines L10 and L11, and a point P20 is extracted from the center lines (not shown) of the boundary lines L20 and L21 as fingertip coordinates.

  On the other hand, according to the conditions, the above-described line segment group L200 formed by the second finger and the line segment L300 connected to the tip coordinate P30 of the first finger as conceptually shown in (3) of FIG. Obtained from the inner area. These line segment groups L200 and L300 will be referred to as “internal lines”. The line segment L300 is formed as a boundary line between the first finger and the second finger in a grasped state.

(Processing of feature correspondence unit 22)
Next, the feature correspondence unit 22 determines the correspondence between the feature amount extracted by the feature extraction unit 21 and a preset reference feature amount (reference value for each of the first feature amount and the second feature amount). decide.

  When the imaging target is a hand shape as shown in FIG. 2, the corresponding relationship is determined by determining which finger is the boundary line extracted as the feature amount. Since the hand shape is biased with respect to arrangement and length, it is possible to use the arrangement of feature quantities or the length of the feature quantity in a region (a palm region) including the feature amount.

  As an example of determining the correspondence relationship by arrangement, the center of gravity of the region is obtained, and the top two boundary lines in the order of short distance from the center of gravity are determined as the feature amounts that the third finger configures. In the case of (3) in FIG. 2, since the top two boundary lines closest to the center of gravity G1 of the palm are L23 and L34, it is determined that the boundary lines L23 and L34 are boundary lines generated by the third finger. it can.

  Or, as three boundary line pairs that face each other in close proximity, the pair L23 and L34, the pair L34 and L45, and the pair L45 and L50 are surrounded by a region surrounded by connecting the end points of the boundary line with a straight line. The boundary line constituting the third finger may be determined as the pair L23 and L34 having the center of gravity closest to the palm center of gravity.

  Furthermore, the feature amounts configured by the fourth finger and the feature amounts configured by the fifth finger are assigned in the order in which the line segments perpendicular to the boundary lines L23 and L34 intersect. The order may be such that the center line of each boundary line is close to the center lines of the boundary lines L23 and L34. Thus, it can be determined that the boundary lines L34 and L45 are boundary lines generated by the fourth finger, and the boundary lines L45 and L50 are boundary lines generated by the fifth finger.

  Alternatively, the boundary line having the minimum length, that is, the boundary line L50 is determined as the boundary line generated by the fifth finger, and the same assignment as above is performed in the reverse direction in the order in which the line segments perpendicular to the feature amount intersect. May be. That is, another boundary line generated by the fifth finger is L45, the boundary lines generated by the fourth finger are L45 and L34, and the boundary lines generated by the third finger are L34 and L23.

  Note that, when performing the assignment of the third finger to the fifth finger, it is necessary to specify in advance the boundary line generated by the first finger and the second finger and exclude it from the search target. At this time, the center of gravity of the palm may be used to exclude the four boundary lines on the side having a larger distance from the center of gravity. Further, as described above, the center of gravity of the region surrounded by the border line pair that is closely opposed to each other is also obtained for the pair L10 and L11 and the pair L20 and L21, and the distance from the center of gravity of the palm among the five pairs of center is lower. You may exclude it as two.

  Alternatively, it is possible to exclude two boundary lines that are substantially parallel and face each other in the vicinity, both of which are also present on the boundary of the palm area. Good. In addition, the palm region for obtaining the center of gravity or the like may be extracted from the image based on the predetermined color feature relating to the skin color as described above.

  Alternatively, by the technique disclosed in Japanese Patent Application Laid-Open No. 2012-8936 by the present inventors, the circumscribed polygon of the palm area is obtained, and the vertices of two sides forming the minimum angle among the angles formed in the non-skin color area are determined. It may be determined that the tip of the first finger and the second finger.

  Further, according to the determination of the correspondence relationship, the correspondence is also determined for the second feature amount obtained by the feature extraction unit 21. In the example of FIG. 2, that is, it is determined that the point P10 is the tip of the first finger and the point P20 is the tip of the second finger.

  On the other hand, when the imaging target is a hand shape as shown in FIG. 3, it is necessary to first extract the line segment group L200 from the line segment L300 out of the internal lines when specifying the first feature amount in the corresponding relationship. As an example of extracting L200, first, the internal line is extracted, and then the center of gravity of the end point of the line segment group (the end point existing inside the palm area excluding the point P30 existing at the boundary) is obtained. According to the conditions, as the end points, the end point that is not P30 of the line segment L300 connected to the tip coordinate P30 of the first finger (point P99 as the end point on the opposite side of the point P30 shown in (3) of FIG. 3), and L200 Since the four end points with the three end points (points P201, P202 and P203 shown in (4) of FIG. 3) are obtained, the end point closest to the center of gravity of the four end points is shown in (4) of FIG. Extracted as an end point P203 of L203.

  Further, three line segments connecting from the end point P203 of L203 to the other three end points are obtained, and the end point having the most different inclination is set as the end point P201 of L201, and the end point close to L201 is extracted as the end point P202 of L202. Since the line segments connected to the respective end points are obtained as L203, L201, and L202, respectively, the remaining line segments are assigned to L210 and L220.

  The “end point” for extraction is an end point formed when the entire internal lines (L200 and L300) are viewed as one body. The end points used as the first feature amount are the end points of each of the line segments L201, L202, L203, L210, and L220 constituting the line segment group L200. A point (points P211 and P212 indicated by white circles in FIG. 3 (4)) and an end point (point P221) on the side of the line segment L220 that is not the end point of the line segment L210 are added.

  For the point P30, which is the second feature amount, the circumscribed polygon of the hand area is obtained by the technique disclosed in the publication of Japanese Patent Laid-Open No. 2012-8936 by the present inventors, and the smallest of the angles formed in the skin color area The vertexes of the two sides forming the angle are selected, and the vertex far from the center of gravity is determined to be P30 with the tip of the first finger. Alternatively, P30 may be determined as a point that is an end point of the line segment L300 and exists at the boundary portion of the palm region.

(Processing of posture estimation unit 23)
The posture estimation unit 23 compares the first feature amount of the reference feature amounts registered in advance on the same plane with the detected first feature amount, thereby comparing the imaging unit 1 and the imaging target relative to each other. The position and orientation are estimated as conversion coefficients. The transformation coefficient estimated by the posture estimation unit 23 is sent to the shape estimation unit 24 and the control unit 3.

  In the example of FIG. 2, the first feature amount obtained from the palm image of the shape when the imaging unit 1 is at a predetermined position and posture is registered in advance as a predetermined reference value, and the posture estimation target Compared with the first feature amount detected from the image of, a change in the position and posture of the palm in the image is estimated as a change from the predetermined position and posture that gives the reference value. The same applies to the example of FIG.

The first feature amount detected and associated from the image, that is, the boundary line end point coordinates or joint coordinate coordinates of the third to fifth fingers in the example of FIG. 2, and the boundary line segment in the example of FIG. The coordinates of each end point are (x ' _j , y' _j ) (1 ≤ j ≤ n) (n = 6 in the examples of Figs. 2 and 3, but n including the general case of using different types of feature quantities. The coordinate as the reference feature is (x _j , y _j ) (1 ≦ j ≦ n), and the conversion coefficient a _k (1 ≦ k ≦ m, m ≦ 2n). When the transformation equation uses the projective transformation of the following equations (Equation 1) and (Equation 2), it can be solved as a 2n simultaneous equation when m = 2n. If m <2n, it can also be solved by the least squares method.

(Processing of the shape estimation unit 24)
The shape estimation unit 24 predicts a value at a predetermined position and posture from a result estimated by the posture estimation process for another feature amount that is not used for posture estimation, that is, a second feature amount, and the predicted value The shape of the imaging target is estimated by comparing the obtained value with the value of the reference feature value. The shape information estimated by the shape estimation unit 24 is sent to the control unit 3.

  In the example of FIG. 2, by applying the projection transformation of the posture estimation result to the tip coordinates of the first finger and the second finger detected and correlated from the image, the first feature value is the predetermined reference value. It is calculated as the value when looking at the position and posture of the palm when taking. That is, the projection matrix estimated in the posture estimation process is applied to the tip coordinates of the first finger and the second finger, and the first finger and The distance from the feature point coordinates of the second finger is calculated and used as shape information. Similarly, in the example of FIG. 3, the position and orientation of the palm when the first feature value takes the reference value by applying the projective transformation obtained from the first feature value to the tip coordinates of the first finger obtained from the image. Find the value when viewed in.

  For example, the origin of the reference coordinate system is defined as the center of gravity of the palm region or the center of gravity of the third to fifth fingers (the center of gravity of the points constituting the first feature amount) in the example of FIG. By comparing, it is possible to know whether the tips of the first finger and the second finger are close to or far from the center of gravity. In particular, with the hand shape of FIG. 2, it is possible to detect with a predetermined accuracy whether the first finger and the second finger are extended outward or bent inward. In the case of the hand shape of FIG. 3, similarly, the center of gravity of the hand region or the center of gravity of the constituent point of the first feature amount is set as the origin, and the part beyond the first joint of the first finger extends outward, Whether it is folded inward can be detected.

  Alternatively, in the example of FIG. 2, the distance D12 between the tip points of the first finger and the second finger as shown in (3) of FIG. 2 may be used as shape information based on the value at the reference feature point coordinates. In this case, whether the distance between the first finger and the second finger is large or small, that is, whether the first finger and the second finger are separated from each other or facing each other to pinch something with a predetermined accuracy is detected. can do.

  Here, the reference value of the second feature value is registered in advance as a reference value in a predetermined shape of the palm, as in the case of the first feature value. In the example of FIG. 2, as the predetermined shape, a value in a state where the first finger and the second finger have a predetermined positional relationship with the palm of each hand is registered as a reference value. By moving the first finger and the second finger from the position, the shape changes from the predetermined shape.

  In the example of FIG. 3, the state where the first joint of the first finger is in a predetermined open state with respect to the grasped second to fifth fingers is defined as the predetermined shape, and the value at that time Is registered as a reference value.

  Note that if the position and orientation of the palm itself changes, the shape change cannot be estimated just by specifying the second feature value from the image, but in the present invention, the value of the second feature value is specified from the image. Furthermore, since the palm position and posture are obtained as values when they are in a predetermined reference arrangement, the shape change can be estimated.

  FIG. 4 is an example of display information control according to the present invention. Here, the mouse cursor is controlled on the screen G10. The position and orientation are estimated based on the first feature amount, and using only the value of the two-dimensional coordinates obtained by projecting the position onto the predetermined xy plane, the movement control of the mouse cursor is performed as indicated by the movement A1, and the position M10 Has been moved to position M11. You may perform control using all the estimated positions and postures.

  On the other hand, using the second feature amount, it is also possible to perform the mouse cursor state control A2, for example, by adding a threshold judgment to the distance from the origin, and here, when it becomes smaller than the threshold, transition to the press state M12. ing. Various controls corresponding to the movement of the finger tip may be performed based on the time change of the distance.

  Here, in particular, the control according to the first feature value and the control according to the second feature value can be performed independently, so that the present invention enables highly convenient control for the user. For example, when using the hand shape of FIG. 2 and moving the palm itself and executing the mouse cursor movement control A1, as long as the first and second fingers are fixed, the image Even if the above appearance changes, the transition to the pressed state of A2 does not occur by mistake.

  Next, supplementary matters in the present invention will be described. The present invention is not limited to the hand shape as described with reference to FIGS. 2 and 3, and a part of the palm is fixed and not moved by a predetermined arrangement, the other part is movable, and the fixed part has the first feature. By setting the amount and setting the second feature amount in the movable part, it can be applied to other hand shapes. The position and posture of the entire palm can be obtained from the first feature amount, and the shape change of the palm caused by the movable part can be obtained from the second feature amount, which can be used independently of each other to control the display information.

  Also, when specifying the first feature amount and the second feature amount in the case of other hand shapes, as in the examples of FIGS. 2 and 3, the relative positional relationship in the palm region (e.g., the center of gravity) A predetermined method that is specified on the basis of a predetermined point such as a deviation in arrangement when using a predetermined point as a reference, a length relationship, or the like may be determined in advance. In the case of the example in FIGS. 2 and 3 as well, identification may be performed by a method other than the method described based on the characteristics such as the positional relationship. Other than the palm may be set as the imaging target, and the above may be performed.

  DESCRIPTION OF SYMBOLS 10 ... Information terminal device, 1 ... Imaging part, 2 ... Estimation part, 3 ... Control part, 4 ... Memory | storage part, 5 ... Display part, 21 ... Feature extraction part, 22 ... Feature correspondence part, 23 ... Attitude estimation part, 24 ... Shape estimation section

Claims (7)

An information terminal device having an imaging unit, which captures a predetermined imaging target and obtains an image thereof,
Extracting a predetermined first feature quantity and second feature quantity of the imaging target from the image, estimating a relative positional relationship of the imaging target with respect to the imaging unit based on the first feature quantity, and the position An estimation unit that estimates the shape of the imaging target based on the relationship and the second feature amount;
A display for displaying information;
A storage unit for storing information to be displayed on the display unit;
An information terminal device comprising: a control unit that controls information read from the storage unit and displayed on the display unit based on the estimated positional relationship and shape.   The imaging target is a palm, and the predetermined first feature amount is set in a first portion provided by being fixed on the palm, and the predetermined second feature amount is provided so as to be movable on the palm. 2. The information terminal device according to claim 1, wherein the information terminal device is set in the second portion. The imaging object is a palm that opens the first finger and the second finger and grasps the third to fifth fingers,
The predetermined first feature amount is an end point of a boundary line of the third finger to the fifth finger or the coordinates of the first joint and the second joint of each finger,
The predetermined second feature amount is fingertip coordinates of the first finger and / or the second finger,
The estimation unit is
A feature extraction unit for extracting a finger boundary from the image;
Identifying the correspondence of which finger the extracted boundary line is attributed to, and a feature corresponding unit that identifies values in the image of the first feature value and the second feature value;
A posture for estimating the position and posture of the palm with respect to the imaging unit based on the identified first feature amount and the first feature amount when the palm is in a predetermined position and posture with respect to the imaging unit An estimation unit;
The second feature amount at a predetermined position and posture of the palm is obtained from the estimated position and posture, and based on the obtained second feature amount and the second feature amount in the predetermined shape of the palm The information terminal device according to claim 1, further comprising: a shape estimation unit that estimates a palm shape.    The feature extraction unit detects a line segment from the image, and from the detected line segment, the color of the line segment, the length of the line segment, the presence of a generally parallel line segment in the vicinity of the line segment, The information terminal device according to claim 3, wherein the boundary line is extracted based on at least one of a monochromaticity of a color distribution near the line segment and a density of the line segment.   The feature corresponding unit includes an arrangement relationship between the extracted boundary lines, a length of the extracted boundary line, a position of a center of gravity of the palm region extracted from the image based on a predetermined color feature, and the 5. The information terminal device according to claim 3, wherein the association is specified based on one of an arrangement relationship with the extracted boundary line. 6. The predetermined first feature amount is the coordinates of the first joint and the second joint of the third to fifth fingers,
The feature-corresponding unit is configured such that the center line between the boundary lines caused by the fingers is close to the edge component of the image or on the center line between the boundary lines caused by the fingers. 6. The information terminal device according to claim 3, wherein a value in the image of the first feature amount is specified as a coordinate of a place where a variation in color difference takes an extreme value. The imaging object is to image the palm of each finger from the side of the first and second fingers,
The predetermined first feature amount is an end point coordinate in each line segment of the boundary line formed by the ventral side portion of the second finger on the side surface of the grasped second finger,
The predetermined second feature amount is fingertip coordinates of the first finger,
A feature extraction unit for extracting a boundary line resulting from finger gripping from the image;
Identifying the correspondence of which part the extracted boundary line is attributed to, and a feature correspondence unit that identifies values in the image of the first feature value and the second feature value;
A posture for estimating the position and posture of the palm with respect to the imaging unit based on the identified first feature amount and the first feature amount when the palm is in a predetermined position and posture with respect to the imaging unit An estimation unit;
The second feature amount at a predetermined position and posture of the palm is obtained from the estimated position and posture, and based on the obtained second feature amount and the second feature amount in the predetermined shape of the palm The information terminal device according to claim 1, further comprising: a shape estimation unit that estimates a palm shape.