JP5569973B2 - Information terminal device, method and program - Google Patents

Description

The present invention relates to an information terminal device , method, and program including an imaging unit and a display unit, and more particularly to an information terminal device , method, and program for controlling a display unit based on the position and orientation of a subject with respect to the imaging unit.

  An apparatus that presents information according to a relative positional relationship with a subject to be imaged can intuitively change the information to be presented, and can improve user convenience. The following methods are disclosed as methods for realizing the above.

  Patent Document 1 proposes that the movement of the position of the imaging target is detected on the detection plane by using a laser sensor.

  Patent Document 2 proposes a method of scrolling or zooming a displayed portion according to the direction and amount of movement and tilt detected by a motion sensor.

  In Patent Document 3, the largest moved block of the input image is detected, and the largest moved block is determined as the position of the imaging target. Then, the feature of the block is analyzed, and the center coordinates of the moved imaging target are tracked.

JP 2010-244480 A JP 2009-003799 A JP 2010-170300 A

  The techniques disclosed in Patent Document 1 and Patent Document 2 require a laser sensor and a motion sensor, respectively, so that there is a problem that devices that can be used are limited. In addition, the installation of special sensors not only increases the cost of the terminal, but also makes it difficult to reduce the size and power consumption of the device.

  In Patent Document 3, since presence / absence of motion is used for detection of an imaging target, there is a problem that only a planar front / rear / right / left input can be handled. In addition, since the center coordinates are used, there is a problem that it is easily influenced by an apparent change of the imaging target.

  The present invention solves the above-described problems of the prior art, and can control information displayed on the display unit of the information terminal device by using a spatial motion of the subject with respect to the information terminal device without using a special sensor or the like. And it aims at providing a highly accurate information terminal device.

  In order to achieve the above object, the present invention provides an area forming unit that extracts an area of a subject from a photographed image based on color characteristics in an information terminal device including a photographing unit that continuously photographs the subject and a display unit. A first polygon forming unit that forms a first circumscribed polygon that includes the extracted subject area; and an inner background area that excludes the subject area from the inside of the first circumscribed polygon. An internal background extraction unit for extracting; a second polygon forming unit for forming a second circumscribed polygon including the internal background region; the first circumscribed polygon; the internal background region; and the second circumscribed region. Based on a polygon, a posture estimation unit that estimates at least one of a position and a posture of the subject with respect to the photographing unit, and information displayed on the display unit based on at least one of the estimated position and posture of Characterized in that it comprises a control unit for controlling the parts.

  According to the present invention, since the display unit is controlled based on at least one of the position and orientation of the subject, an intuitive operation interface is provided to the user who uses the information terminal device of the present invention. Further, according to the present invention, at least one of the position and orientation of the subject can be estimated at high speed and with high accuracy by simple image processing on the image of the subject continuously photographed by the imaging unit. In addition, an operation interface that is controlled with high accuracy and is easy to use for a user who uses the information terminal device of the present invention is provided. Furthermore, according to the present invention, only simple image processing is used, and no special sensor or the like is required.

It is a functional block diagram of the information terminal device of the present invention. It is a figure explaining the outline | summary of operation | movement of this invention. It is a figure explaining the process of a feature detection part and an attitude | position estimation part. It is a figure explaining estimation of the shape of a subject (opening / closing of a finger) by a posture estimation unit. It is a more detailed functional block diagram of a control part. It is a figure which shows the example of a movement control of a part of display information by a movement control part. FIG. 4 is a diagram illustrating rotation in each direction from a reference position as an example of a posture change of a subject (hand). FIG. 8 is a diagram showing an example in which a part of display information is rotation-controlled by a rotation control unit based on the estimation result of each rotation in FIG. FIG. 8 is a diagram illustrating an example of the shape of an internal background region and the position of a reference point when such a rotation actually occurs for the posture estimation unit to estimate each rotation in FIG. 10 is a chart showing a correspondence relationship between FIGS. 7 to 9; It is a figure which shows the example of the expansion / contraction control of a part of display information by an expansion control part. It is a figure explaining an auxiliary | assistant display part. It is a figure which shows the example of the general to-be-photographed object which can be utilized in this invention and is not limited to a hand.

  Hereinafter, the present invention will be described in detail with reference to the drawings. FIG. 1 shows a functional block diagram of the information terminal device of the present invention. The information terminal device 10 includes an imaging unit 1, an estimation unit 2, a control unit 3, and a display unit 4. The estimation unit 2 includes a region formation unit 21, a feature detection unit 22, and a posture estimation unit 23. The feature detecting unit 22 includes a first polygon forming unit 221, an internal background extracting unit 222, and a second polygon forming unit 223.

  As the information terminal device 10, a mobile terminal can be used, and a digital camera provided as a standard in the mobile terminal can be used as the photographing unit 1, but the configuration to which the present invention is applied is not limited to this. Any information terminal having an imaging function may be used. For example, a personal computer having an imaging function using an external camera or the like may be used.

  FIG. 2 shows an outline of the operation of the present invention, and the outline of the operations of the photographing unit 1, the estimation unit 2, the control unit 3, and the display unit 4 will be described with reference to FIG. An information terminal device 10 as shown in (a) (here, it is a portable terminal and has an imaging unit 1 on the back of the display unit 4), the imaging unit 1 causes the subject (b) to be captured at a predetermined sampling period. Shooting continuously. As shown in FIG. 2, the subject (b) is a hand in a preferred embodiment of the present invention, grasps the third to fifth fingers (middle finger, ring finger and little finger), and the grasped portion is separated by a gap. The first finger (thumb) and the second finger (forefinger) are arranged so as to surround them.

  The grasped third to fifth fingers are not fixed and opened as they are, but the first and second fingers may be moved as long as they surround the grasped portion. By moving it, the belly of the first finger and the belly of the second finger may be in contact with each other, or may be in a separated state as shown in FIG.

  The imaging unit 1 captures the hand (b) as a subject from the first and second fingers. The third to fifth fingers grasped are located behind the surface (b1) substantially formed by the first finger and the second finger when viewed from the imaging unit 1.

  The estimation unit 2 estimates the relative position and orientation of the hand (b) with respect to the imaging unit 1 using the captured image of the hand (b) as shown in (d) sent from the imaging unit 1. FIG. 2 shows an example of a spatial arrangement serving as a reference for estimating the position and orientation. That is, the imaging surface of the imaging unit 1 (which is drawn in an expanded manner) is shown as (a1), and is in a state parallel to the plane (b1) that is generally formed by the first and second fingers of the hand (b). Show. Further, it is assumed that the hand (b) is located in front of the imaging unit 1.

  The reference arrangement is an example arrangement used for convenience of explanation, and other arrangements may be used as a reference.

  The estimation unit 2 estimates changes in the position and orientation of the subject (b) using the known spatial arrangement and the captured image (d) of the subject (b) in the known subject (b) as a reference. To do. The estimated position is a spatial position determined by movement in the x-axis and y-axis directions (c2) in the surface (b1) of the subject (b). The estimated posture is a posture corresponding to the tilt of the surface (b1) that appears as the tilt of the hand (b), for example, by turning or tilting the wrist, and is determined by rotation (c3), (c4), etc. Is. In addition, the estimation unit 2 also estimates the distance (c1) between the surface (a1) and the surface (b1) in the depth z-axis direction, the shape of the hand, that is, the opening and closing of the first and second fingers.

  In particular, the estimation unit 2 detects the region (d4) of the hand (b) in the captured image (d) and is surrounded by the regions (d2) and (d1) of the first finger and the second finger, and (d4) A U-shaped or V-shaped region (d5) that does not belong is detected. Further, a reference point (d30) assuming the second joint of the third finger is detected from (d5). When the position and orientation of the subject change, the appearance thereof, that is, the position and shape of (d4) change, and the detected results (d4), (d5), and (d30) also change. The estimation unit 2 estimates the changed position and orientation of the subject (b) from these detection results based on geometric considerations.

  The control unit 3 controls a part of information displayed on the display unit 4 based on the position and orientation estimated by the estimation unit 2. For example, if the display unit 4 has a pointer, the estimated result of the position change in the xy direction in (c2) is directly or multiplied by a predetermined value, and the pointer on the display unit 4 is moved to function as a substitute for a mouse. be able to. Further, for example, when a stereoscopic image is displayed on the display unit 4, the stereoscopic image is displayed as if it were a hand (b) in conjunction with a change in posture estimated as the rotation direction such as (c3) and (c4). It can be rotated as if you are riding on. Also, other display control can be performed based on the estimated position and orientation and others.

  Note that in the whole of the present invention, the estimation unit 2 estimates the position (x, y) and posture (angle θ, etc.), and the control unit 3 changes the position (Δx, Δy), which is a change in the estimated amount. Control is performed using a change in angle or posture (change in angle Δθ or the like). In addition, using the distance z estimated by the estimation unit 2, the control unit 3 performs control using the change Δz. As for the shape estimated by the estimation unit 2 described later, the shape estimated at each time point is used as it is by the control unit 3, but as a result, the change in the estimated shape is also used by the control unit 3.

  According to the present invention as schematically shown in FIG. 2, the following effects (1) to (3) are obtained. (1) A part of display information on the display unit 4 can be controlled only by changing the relative position and posture of the imaging unit 1 and the subject. Therefore, the user can control a part of the display information by an intuitive operation of changing the relative position and posture of the subject (particularly the user's own hand) with respect to the imaging unit 1.

  (2) Since the estimation unit 2 estimates the position and orientation by simple image processing on the image of the subject, it can be realized by software, for example, and special hardware such as special sensors is provided in the information terminal device 10. There is no need to include it.

  (3) Since the position, posture, and the like are estimated at high speed and with high accuracy by the processing described in the captured image (d) in FIG. 2, a part of the display information can be controlled at high speed and with high accuracy.

  Next, details of the processing of each unit of the estimation unit 2, that is, the region forming unit 21, the feature detection unit 22, and the posture estimation unit 23 will be sequentially described. First, the region forming unit 21 extracts a skin color region that is a region of the subject's hand from the image input from the imaging unit 1. The skin color region can be extracted based on the color information in the image input from the imaging unit 1. For example, when the image input from the imaging unit 1 is expressed in the RGB color space, the skin color area can extract pixels that fall within a preset range of RGB as the skin color area. For example, you may extract using the setting range like (Formula 1) introduced by the following nonpatent literature 1.

  (Non-Patent Document 1) Vladimir Vezhnevets, Vassili Sazonov, Alla Andreeva, "A Survey on Pixelbased Skin Color Detection Techniques," Pattern Recognition, vol.40, no.3, pp.1106-1122, March 2007.

  Alternatively, the RGB color space is converted into a representation of the HSV color space (brightness V, saturation S, hue H). The conversion from the RGB color space representation to the HSV color space representation is expressed by (Equation 2).

  Pixels in which the hue H in the above (Equation 2) falls within a preset range may be set as the skin color region.

  Alternatively, other color spaces such as YCbCr may be used. The conversion from the RGB color space representation to the YCbCr color space representation is expressed by (Equation 3).

  Pixels in which the color differences Cb and Cr in (Equation 3) are within a preset range may be used as the skin color region.

  Or it is good also considering the area | region where the weighted average of the skin color area | region result in several color spaces exceeds the preset threshold value as a skin color area | region. The weight applied to each method is set in advance using a known skin color region so that the skin color can be extracted most accurately. Depending on the processing performance of the information terminal device 10, the type and number of methods to be combined may be set. If there is a margin in processing performance, a skin color detection method based on a probability model such as a Gaussian mixture model may be combined.

  Next, the processing of the feature detection unit 22 will be described in the order of the first polygon forming unit 221, the internal background extraction unit 222, and the second polygon forming unit 223, and further the processing of the posture estimation unit 23 will be described. FIG. 3 shows an example diagram for explaining these processes. As described below, the feature detection unit 22 and the posture estimation unit 23 are set in advance by the processes exemplified in (A) to (E) of FIG. 3 and finally shown as the root position of the arrow (E). The coordinates corresponding to the reference point (position coordinates in the image) and other information are detected. As described above, the reference point set in advance is the second joint of the third finger if the hand is assumed as the subject, and the position coordinates of the joint are detected by the following processing. Then, the posture estimation unit 23 estimates the posture using the reference point and the like.

  First, the first polygon forming unit 221 selects the largest skin color region among the skin color regions extracted by the region forming unit 21, and compensates for partial defects in the skin color region using a morphology filter. As a result, a skin color region as shown in (A) is obtained. Subsequently, the first polygon forming unit 221 forms a circumscribed polygon as shown in (B), which includes the entire skin color area shown in (A).

  Subsequently, the internal background extraction unit 222 extracts the non-skin color region having the maximum area existing inside the formed circumscribed polygon (B) as the above-described internal background region, as indicated by the hatched portion (C). In addition, by extracting the maximum non-skin color region, for example, the skin color region (A) and circumscribed polygon (B) occurring on the back side of the hand, the wrist side, etc., other than the region surrounded by the first finger and the second finger A small gap area between and is excluded from the internal background area.

  Subsequently, the second polygon forming unit 223 forms a circumscribed polygon including the entire extracted internal background region (C) as indicated by a dotted line (D). In the example of FIG. 3, since the first finger and the second finger are separated from each other, the circumscribed polygon (D) shares a part of the side with the circumscribed polygon (B). If the first finger and the second finger are in contact with each other and the hand is in a state of pinching the fingers, the circumscribed polygon (D) does not share the side with the circumscribed polygon (B).

  The circumscribed polygons (B) and (D) are convex by applying various well-known techniques to the skin color area (A) and the internal background area (C), which are the areas from which the polygons are formed. Formed as a polygon.

  Subsequently, the posture estimation unit 23 detects the reference point (F) corresponding to the corner point formed by the third joint, which is indicated as the base point of the arrow (E), and calculates its coordinates. Perform the following process. That is, the minimum distance among the distances from each point on the boundary of the inner area (C) to the point on the side of the formed circumscribed polygon (D) without passing through the inner area (C) Find what will be. And, as shown as the root of the arrow (E), the point on the boundary of the inner region (C) that gives the maximum distance among the minimum distances determined from each point on the boundary of the inner region (C), The reference point (F).

  Further, the posture estimation unit 23 uses at least one of the obtained reference point (F), the internal region (C), or the relative positional relationship between (F) and (C) to determine the position of the subject and Estimate posture. Specifically, the coordinates of the reference point (F), that is, the (x, y) position coordinates on the image are estimated as the position of the subject. Further, the posture of the subject is estimated using the positional relationship between the reference point (F) and the center of gravity of the internal region (C). The estimation of the posture will be described in detail together with the description of the control unit 3 described later.

  Further, the posture estimation unit 23 opens the shape of the hand that is the subject, that is, the first finger and the second finger, based on whether or not the circumscribed polygons (B) and (D) have a shared side. Whether it is closed or touching. The shape estimation will be described with reference to the example of FIG. That is, when the first finger and the second finger are opened apart as in (10), the circumscribed polygons (B) and (D) have overlapping portions as in (11), and the side ( Share some). Conversely, when the first finger and the second finger are in contact and closed as in (20), the circumscribed polygons (B) and (D) have no overlapping part as in (21), and the side ( Do not share). As described above, the shape of the hand, that is, the opening / closing of the fingers can be estimated by the presence / absence of the shared side of the circumscribed polygons (B) and (D).

  Then, the control unit 3 moves or rotates part of the information displayed on the display unit 4 in conjunction with the estimated position and posture of the subject. As shown in FIG. 5, the control unit 3 includes a movement control unit 31, a rotation control unit 32, a switching control unit 33, and a magnification control unit 34, and each control function is carried out independently or in conjunction with each other. Provide an intuitive interface. These will be described below.

  The movement control unit 31 moves a part of the display information based on the estimated change in the position of the subject. An example of the movement control is shown in FIG. That is, the reference point (F) determined as a change in the estimated position (x, y) (Fx) movement (Δx, Δy) in pixels on the image (1) as it is or multiplied by a constant a, the display unit (2) The position of a part of the display information, for example, the pointer, is moved by (Δx, Δy) or (aΔx, aΔy). Further, the adjustment of the movement amount of a part of the display information by the constant a times may use different constants in the x direction and the y direction.

  When the subject is a hand held in the shape described with reference to FIG. 2 or FIG. 4, the estimated position is the position of the corner of the third joint of the second finger grasped. The position of the corner is a position that moves in a rigidly interlocked manner with the entire hand holding the third to fifth fingers. Accordingly, the movement control unit 31 provides an intuitive and easy-to-operate interface for the user, similar to the movement of the hand as it is in the known mouse, which directly becomes the amount of movement of the pointer.

  The rotation control unit 32 rotates a part of the display information based on the estimated change in the posture of the subject. An example of the rotation control and posture estimation by the posture estimation unit 23 required therewith will be described with reference to FIGS. FIG. 7 shows (1) an example of each rotation of the subject (hand) as an example of the actual posture change of the subject to be estimated. FIG. 8 shows (1) an example in which the rotation control unit 32 controls (2) part of display information on the display unit 4 based on the estimation result of each rotation of the subject.

  9 is obtained by the feature detection unit 22 and the posture estimation unit 23 as described in FIG. 3 in each case of (1) rotation of the subject (3) internal background region and reference point, and the reference point and the internal An example of a reference point / centroid vector v having a reference point representing a positional relationship with the centroid of the background region as a start point and an end point as the centroid is shown. Since the shape of the internal background region and the positional relationship with the reference point change according to the posture (rotation) of the subject, posture estimation by the posture estimation unit 23 becomes possible. The posture estimation unit 23 estimates the posture corresponding to each rotation of the subject (1) using the reference point / centroid vector in (3) above. Hereinafter, the correspondence between the respective drawings in FIGS. 7 to 9 will be described.

  FIG. 7 shows (1) an example of each rotation of the subject (hand) as an example of the posture change of the subject. In FIG. 7, the hand (b) that is the subject exists on the plane (b1) in the state of the reference position, posture, and shape as described in FIG. 2, and an image thereof is obtained as (d). It shows the state. In this example, the posture estimated to rotate in the up / down direction on the axis of the arm shown as the axis (c3) and the left / right direction of the plane (b1) shown as the axis (c4), Although a part of the display information is controlled, other rotation axes may be set together with other reference positions.

  In FIG. 8, corresponding to (1) in FIG. 7, the subject's hand is rotated up / down around the axis (c3) corresponding to the wrist direction axis from the reference position of (1), and (c4) An example is shown in which the information displayed on the display unit 4 is controlled when rotating right / left about the axis. As an example of part of the display information to be controlled, a rectangular parallelepiped having an elliptical pattern on one side is drawn. Then, an example is shown in which the rectangular parallelepiped is rotated and displayed in accordance with each rotation of the hand under the control of the control unit 3 from the state of facing the front indicated by (2 reference position). The xyz axis is shown as a convenience for easy understanding of the state of rotation.

  In other words, in FIG. 8, (c3) up rotation / down rotation and (c4) right rotation / left rotation corresponding to the cases of (2 up rotation) / (2 down rotation) and (2 right rotation), respectively. Rotated and displayed as / (2 left rotation). The rotation control is an example of control in which the rectangular parallelepiped sticks to the plane (b1) in FIG. 7 of the hand that is the subject and rotates in the same direction in conjunction with the rotation of the hand.

  FIG. 9 shows the shape of the internal background region, the reference point, and the reference point detected by the feature detection unit 22 and the posture estimation unit 23 in the rotation in each case of (c3) and (c4) described above with reference to FIG. An example of a reference point / center-of-gravity vector v starting from, and ending at the center of gravity of the internal background region is shown. Similarly to the above, examples after rotation with respect to (3 reference positions) are shown as (3 upper rotation) / (3 lower rotation) and (3 right rotation) / (3 left rotation).

Further, as shown in FIG. 9, the reference point / centroid vector v in each posture is as follows.
(Reference position) v = (− x _ref , 0)
(Up rotation) v = (− x _up , + y _up )
(Down rotation) v = (− x _down , −y _down )
(Rotate right) v = (+ x _right , 0)
(Rotate left) v = (− x _left , 0)

Here, since (3 right rotation) / (3 left rotation) and (3 reference position) have the following relationship, the value corresponding to each posture of the reference point / centroid vector v is within a predetermined threshold. And the posture estimation unit 23 can distinguish and estimate these postures.
x _left > x _ref > 0, x _right > 0

In addition, since (3 upward rotation) / (3 downward rotation) and (3 reference position) have the following relationship, the posture estimation unit 23 converts these postures to the above (3 right rotation) / It is possible to make a distinction estimation including the distinction with (3 left rotation).
x _up > 0, y _up > 0, x _down > 0, y _down > 0, x _up ≒ x _ref , x _down ≒ x _ref

  In addition, in FIG. 9, the shape of the internal background region in each posture, the reference point, the reference point / centroid vector, and the change state between these postures can be more easily grasped. A rectangular region having the same position and the same size on the obtained image is displayed by a dotted line. As is clear from the image comparison in each posture with the dotted line rectangle as a guide, the present invention has the following effects. That is, a plane is formed by pinching the first finger and the second finger as a subject, and a hand with the third to fifth fingers held behind the plane as viewed from the imaging unit 1 is set. Therefore, the appearance changes significantly according to each rotation, and the remarkable reference point / centroid vector v change in each posture as described above is obtained, so that the posture estimation can be performed with high accuracy. There is.

  This effect is particularly based on the fact that there is a step between the first finger and the second finger and the third to fifth fingers grasped. For example, in (3 left rotation), the part surrounded by the first finger and the second finger is further away from the imaging unit 1 than (3 reference position), so the internal background area is smaller in both the x direction and the y direction. The convex part of the three fingers is also reduced. Conversely, in (3 right rotation), the enclosed portion is closer than (3 reference position), and the internal background area is reduced in the x direction and enlarged in the y direction. The convex part of the third finger is enlarged and emphasized. In the case of (3 right rotation), if it rotates to some extent, the second joint of the fourth or fifth finger is obtained as the reference point instead of the second joint of the third finger. Will be promoted.

  In the same consideration, the shape of the internal background region changes significantly in the case of (3 upward rotation) / (3 downward rotation) as compared with (3 reference position). In the case of (3 upward rotation), when the first finger approaches the imaging unit 1 and the second finger moves away, the appearance of the third to fifth fingers grasped at the same time changes in conjunction with ( The (3 reference position) horizontal U-shaped or V-shaped inner background area that is generally vertically symmetric changes to a shape that is biased upward in (3 upward rotation). Conversely, in the case of (3 downward rotation), when the first finger moves away from the imaging unit 1 and the second finger approaches, the appearance of the third to fifth fingers grasped simultaneously also changes. Thus, in (3 downward rotation), the shape changes to a downwardly biased shape.

  The effect that the appearance and shape change remarkably as described above is due to the presence of the step and is unique to the present invention. If the (3 reference position) U-shaped or V-shaped internal background region is obtained by a planar subject, not by a subject with a three-dimensional step like the hand in the present invention, the subject rotates. Even so, the internal background area only changes in an affine transformation or projective transformation, and a remarkable shape change as shown in FIG. The change of the center of gravity vector cannot be obtained.

FIG. 10 shows a chart summarizing the correspondence between FIGS. In FIG. 10, the rotation control unit 32 performs the display rotation (2) described in FIG. 8 based on the estimated posture (1) described in FIG. If the rotation is upward, the display information is rotated by aθ _up by multiplying the rotation amount θ _{up of the} subject by a predetermined number a as in the case of the movement control unit described above. The constant a is used in the downward rotation θ _down , the right rotation θ _right, and the left rotation θ _left , but another constant may be used.

  As described above, the estimated posture (1) is the angle of the actual subject posture, and the rotation (2) of the display information is the amount of rotation based on the estimated angle change, but it is displayed because the notation is complicated In the column of information rotation (2), the notation of Δ representing the amount of change is omitted.

  Since the subject is a hand in the present invention, when the rotation amount of the hand is around 90 °, the internal background region (C) and the reference point (F) as described in FIG. It becomes difficult. However, by setting a value where a> 1 as a, for example, it is possible to rotate part of the display information by 90 ° or more only by a single hand rotation operation from the reference position.

  In FIG. 10, the posture estimation unit 23 estimates the posture (1) described in FIG. 7 based on the reference point / centroid vector (3) described in FIG. As for the allocation to each of the upper, lower, left and right directions of rotation, only one direction may be allocated with a predetermined threshold value, or a plurality of directions may be allocated by weighting by fitting or the like. For example, simultaneous rotation of upward rotation and right rotation may be estimated. The amount of rotation in each direction may be estimated in proportion to the length of the reference point / centroid vector. In addition, the estimation of the amount of rotation in each direction is such that if the change in the appearance of the internal background region due to the rotation is uneven, the change in the amount of rotation is estimated smoothly in proportion to the actual amount of rotation of the hand. A correspondence relationship to be corrected may be used.

  It should be noted that the correspondence values as shown in FIG. 10 have different specific values for estimating the posture depending on individual hands (characterized by finger length, thickness, etc.) as subjects. Such values are set in advance in the estimation unit 2 for each subject and each reference position. Further, instead of using such individual setting values for individual hands, representative setting values may be used assuming a reference position corresponding to a representative hand shape.

  As an application of the posture estimation and rotation control of FIGS. 7 to 9 and FIG. 10 as described above, the inner background region itself is enlarged by widening the open width of the first finger and the second finger. Thus, the amount of rotation for display control can be adjusted on the user side by utilizing the fact that the amount of movement of the center of gravity of the internal background region accompanying the rotation of the hand increases. That is, even for the same hand rotation amount, the rotation amount controlled for display is small when the first finger and the second finger are close, and the rotation amount controlled for display is increased when the first finger and the second finger are far away. Can do.

  Note that, as described above, the posture estimation and the rotation control in FIGS. 7 to 9 and 10 are merely examples, and other rotation axes and reference positions can be used.

  The switching control unit 33 performs switching of the control method in the control unit 3 based on whether the estimated shape described above with reference to FIG. 4 is “open” or “closed”. Various uses of the switching will be described later. As an example, when a part of information displayed on the display unit 4 is a pointer, the state corresponding to the press state and the release state of a mouse button in a known mouse pointer is used. , The estimated shape can be associated with “closed” and “open”. In this case, by moving the first finger and the second finger in the order of “(open) →” “close” → “open”, it is possible to realize processing corresponding to the click operation on the mouse pointer. Further, if the movement control unit 31 is in charge of the movement of the pointer as described in FIG. 6, the same processing as that of the mouse pointer can be realized.

  The magnification control unit 34 enlarges or reduces part of the information displayed on the display unit 4 based on a change in the estimated distance between the imaging unit 1 and the subject to be described next. As a preferred embodiment, the smaller the estimated distance, the larger the displayed distance, and the smaller the estimated distance, the smaller the displayed distance. Can be displayed. That is, as a preferred embodiment, as shown in FIG. 11, the magnification control unit 34 has a rectangular parallelepiped as the hand that is the subject of (1) and the rectangular parallelepiped that is an example of part of the display information of (2) are closer to each other. Can be controlled such that the cuboid is enlarged and the rectangular parallelepiped is reduced as the hand is moved away.

  Here, the estimated distance used in the magnification control unit 34 corresponds to the distance (c1) shown as the z-axis in FIG. 2, and is estimated by the posture estimation unit 23. The distance is estimated by the area of the internal background region. Using a predetermined proportional constant or correspondence relationship, the distance is smaller as the area is larger, and the distance is larger as the area is smaller.

  The distance estimation may be performed only when the estimated shape is “closed”, that is, when the first finger and the second finger are closed. In this case, since the first finger and the second finger are substantially fixed, the distance can be more reliably estimated from the area of the internal background region, and the enlargement / reduction control can be performed by moving the entire hand. Conversely, if distance estimation is performed even when the estimated shape is not “closed”, in addition to movement of the entire hand, enlargement / reduction control linked to the degree of opening of the first and second fingers is also performed. can do.

  As described above, only one of the control of each unit (movement control unit 31 to magnification control unit 34) of the control unit 3 may be used individually, or a plurality of combinations may be used. A preferable example among the embodiments using the combination will be described below.

  For example, when a pointer is handled as part of display information to be controlled and the pointer is moved by the movement control unit 31, it is considered rare to use the rotation control unit 32 at the same time. Do not use part 32. Consider moving the pointer in such a state to display, for example, a website for mail order on the display unit 4 and aligning the pointer with the product image in the web page. Here, since the product is rotated and viewed, the switching control unit 33 can be used to switch from the pointer movement control to the product image rotation control.

  For example, if the pointer is on the product image and the estimated shape is “closed” (or conversely “open”), the control target is switched from the pointer to the product image, and the rotation control unit is not used without using the movement control unit 31. 32 can be set so that only the product is rotated without worrying about the pointer position. Further, in addition to the rotation by the rotation control unit 32 on the product image to be controlled, the magnification control by the magnification control unit 34 may be performed. The operation to pick up can be realized. When selecting another product, the shape may be set to “open” (or conversely “closed”) and the pointer may be moved to the other product, and the same may be performed.

  Further, in the case where, for example, a = 1 is set in FIG. 10 to perform rotation control of an amount equal to the rotation amount of the hand, the product is selected in the direction in which the rotation control is switched after selection of the product. Rotate to return the shape to “open” (or conversely “closed”), move to the movement control state, return the hand rotation to the reference position, and again “close” (or conversely “open”) ), The rotation control of the product can be continued and rotated 45 ° + 45 ° = 90 ° in the same direction.

  Further, when part of the display information to be controlled is, for example, a scroll bar or a scroll menu, only the rotation control unit 32 may be used instead of the movement control unit 31. Also in this case, the switching control unit 33 can be used in the same manner as described above when, for example, another scroll menu is desired to be switched for switching the control target and the control method.

  FIG. 12 shows an example of an auxiliary display unit that can be used supplementarily in the display unit 4. In FIG. 12, the auxiliary display unit 40 is provided by using a partial area of the display unit 4, but may be provided by using another display. As shown in FIG. 12, the auxiliary display unit 40 includes an image including a subject such as a hand photographed by the imaging unit 1 (by a cycle similar to the sampling cycle of the imaging unit 1 or a rough cycle). It can be displayed in real time. Alternatively, as described with reference to FIG. 3, the image may be displayed as a skin color region, circumscribed polygon, reference point, or the like obtained by performing processing in the estimation unit 2 on the image of the subject's hand.

  By using the auxiliary display unit 40, as shown in FIG. 12, the user moves and rotates the subject by moving his / her hand, and the movement and rotation of a part of the display information exemplified as a rectangular parallelepiped in conjunction with the movement. When the control is performed, there is an effect that it is possible to prevent the subject from being taken out of the captured image without the user's knowledge and the situation where the control is not performed even if the hand is moved. For example, it is possible to check using the auxiliary display unit 40 so that the hand does not disappear from the captured video image by moving the hand too much. Further, for example, the auxiliary display unit 40 can be used to check at any time so that the internal background region cannot be obtained due to excessive rotation of the hand and normal control cannot be performed.

  As is clear from the significance of the use of the auxiliary display unit 40 as described above, the display on the auxiliary display unit 40 is different from the display information of a part of the display unit 4 to be controlled by the control unit 3. That is, in principle, the auxiliary display unit 40 performs display independently of the display control by the control unit 3. However, as an exception, the switching control unit 33 may be used for switching whether to use the auxiliary display unit 40 or not.

  Note that the switching control unit 33 described above may be used for switching whether or not to perform display control itself according to the present invention. For example, display control is started by bringing the hand to a reference position from a state in which the hand is not shown in the photographed image and performing, for example, the above-described click operation or a double-click operation in that state. Also good. Similarly, display control may be terminated. By doing in this way, the influence of the operation | movement which a user does not intend as display control like the operation | movement until a hand is brought to a reference position can be excluded.

  FIG. 13 shows a general example that is a subject that can be used in the present invention and is not limited to a hand. The subject includes a pair of semi-annular portions A1 and A2 corresponding to the first finger and the second finger in the subject “hand”, and A1 and A5 corresponding to the third to fifth fingers grasped in the subject “hand”. A convex portion B is provided which is surrounded by A2 and located behind it. A1 and A2 may be U-shaped or V-shaped, and “semi-annular” includes such a shape.

  A1 and A2 can be opened and closed by, for example, a hinge C (rotary axis C) located at the base thereof, and an annular portion is formed by closing. Even when such a general subject is used, the region forming unit 21 uses the color feature of the subject that is not necessarily a skin color (preferably a uniform color as a whole), and the above (Equation 1 ) To (Equation 3) can be used to extract the general subject area by a method similar to that described above.

DESCRIPTION OF SYMBOLS 10 ... Information terminal device, 1 ... Imaging | photography part, 2 ... Estimation part, 21 ... Area | region formation part, 22 ... Feature detection part, 221 ... First polygon formation part, 222 ... Internal background extraction part, 223 ... Second polygon Forming unit, 23 ... Attitude estimation unit, 3 ... Control unit, 4 ... Display unit

Claims (12)

In an information terminal device including a photographing unit and a display unit that continuously photograph a subject,
An area forming unit that extracts an area of the subject based on color characteristics from a captured image;
A first polygon forming unit forming a first circumscribed polygon including the extracted subject area;
An internal background extraction unit that extracts an internal background region that excludes the region of the subject from the inside of the first circumscribed polygon;
A second polygon forming portion forming a second circumscribed polygon including the inner background region;
A posture estimation unit that estimates a position and a posture of the subject with respect to the photographing unit based on the first circumscribed polygon, the inner background region, and the second circumscribed polygon;
A control unit that controls a part of information displayed on the display unit based on the estimated position and orientation ;
The photographed image of the subject is an image of a hand that forms a step and a U-shaped or V-shaped region by gripping a finger and surrounding the gripped portion with another finger, and forming the region Extract the skin-colored area,
The posture estimation unit estimates the position based on a reference point extracted from the internal background region based on the shape of the region and corresponding to a corner portion formed by grasping,
The posture estimation unit estimates the posture based on a positional relationship between the reference point and the internal background region depending on the presence of the step;
The posture estimation unit further includes a finger that surrounds the shape of the subject based on whether there is an overlapping portion between the side of the first circumscribed polygon and the side of the second circumscribed polygon. Infers whether it is open away or closed close,
The control unit moves a part of the information displayed on the display unit based on the estimated change in position, and is displayed on the display unit based on the estimated change in posture. a rotation control unit for rotating a portion of that information, based on the estimated shape, characterized Rukoto and a switching control unit for switching the part of the state of the information displayed on the display unit Information terminal device. In an information terminal device including a photographing unit and a display unit that continuously photograph a subject,
An area forming unit that extracts an area of the subject based on color characteristics from a captured image;
A first polygon forming unit forming a first circumscribed polygon including the extracted subject area;
An internal background extraction unit that extracts an internal background region that excludes the region of the subject from the inside of the first circumscribed polygon;
A second polygon forming portion forming a second circumscribed polygon including the inner background region;
A posture estimation unit that estimates a position and a posture of the subject with respect to the photographing unit based on the first circumscribed polygon, the inner background region, and the second circumscribed polygon;
A control unit that controls a part of information displayed on the display unit based on the estimated position and orientation ;
The photographed image of the subject grasps the third and fifth fingers to the back side, and the hand in a state where the first and second fingers surround the grasped portion from the front side, the first and second fingers The image is taken from the side, and the region forming unit extracts a skin color region,
The reference point that is extracted from the internal background region based on the shape of the region by the posture estimation unit and that corresponds to the corner portion of the second joint by the grasped third to fifth fingers Estimating the position based on
The posture estimation unit estimates the posture based on a positional relationship between the reference point and the internal background region depending on the existence of the back side and the near side;
The posture estimation unit further determines the shape of the subject based on whether or not there is an overlapping portion between the side of the first circumscribed polygon and the side of the second circumscribed polygon. Infers whether the second finger is open apart or closed in contact;
The control unit moves a part of the information displayed on the display unit based on the estimated change in position, and is displayed on the display unit based on the estimated change in posture. a rotation control unit for rotating a portion of that information, based on the estimated shape, characterized Rukoto and a switching control unit for switching the part of the state of the information displayed on the display unit Information terminal device. 3. The information terminal device according to claim 1, wherein the first circumscribed polygon and the second circumscribed polygon are convex polygons. The internal background extraction unit, according to claim 1, wherein extracting the area of the maximum area of the first circumscribed polygon area that excludes a region of the subject from the inside as the inner background area The information terminal device according to any one of the above. The posture estimation unit calculates the shortest distance from each point on the boundary of the internal background region to the side of the second circumscribed polygon without passing through the internal background region, and sets the shortest distance to the longest distance claims 1, characterized in that such a point on the boundary of the internal background region is extracted as the reference point and the information terminal apparatus according to any one of 4. The posture estimation unit, the information terminal device according to any one of 5 claims 1 and estimates the posture based on the positional relationship between the center of gravity of the inner background area and the reference point. The posture estimation unit compares a vector value starting from the reference point and ending at the center of gravity of the internal background area with a preset value of the vector when the subject is rotated in each direction from the reference position. 7. The information terminal device according to claim 6, wherein the posture is estimated by doing so. Any said control unit is further of claims 1 to 7, characterized in that it comprises a magnification control unit for enlarging or reducing a portion of the information displayed on the display unit based on a change of the area of the inner background area An information terminal device according to any one of the above. The display unit further the captured image is continuously captured by the imaging unit, of claims 1 to 8 to the control by the control unit, characterized in that it comprises an auxiliary display section for displaying independently The information terminal device according to any one of the above. In a method executed by an information terminal device including an imaging unit and a display unit for continuously imaging a subject,
An area forming step of extracting the area of the subject based on color characteristics from the captured image;
A first polygon forming step of forming a first circumscribed polygon that encompasses the extracted subject area;
An internal background extraction step for extracting an internal background area excluding the area of the subject from the inside of the first circumscribed polygon;
A second polygon forming step for forming a second circumscribed polygon including the inner background region;
A posture estimation step of estimating a position and a posture of the subject with respect to the photographing unit based on the first circumscribed polygon, the inner background region, and the second circumscribed polygon;
A control step of controlling a part of information displayed on the display unit based on the estimated position and orientation;
The photographed image of the subject is an image of a hand that forms a step and a U-shaped or V-shaped region by gripping a finger and surrounding the gripped portion with another finger, and forming the region The stage extracts the skin color area,
The posture estimation step estimates the position based on a reference point extracted from the internal background region based on the shape of the region and corresponding to a corner portion formed by grasping;
The posture estimation step estimates the posture based on a positional relationship between the reference point and the internal background region depending on the presence of the step;
The posture estimation step further includes determining whether the surrounding finger has a shape of the subject based on whether there is an overlapping portion between the side of the first circumscribed polygon and the side of the second circumscribed polygon. Infers whether it is open away or closed close,
The control step is displayed on the display unit based on the movement control step of moving a part of information displayed on the display unit based on the estimated change in position and the estimated change in posture. A rotation control step for rotating a part of the information to be rotated, and a switching control step for switching a state of a part of the information displayed on the display unit based on the estimated shape. . In a method executed by an information terminal device including an imaging unit and a display unit for continuously imaging a subject,
An area forming step of extracting the area of the subject based on color characteristics from the captured image;
A first polygon forming step of forming a first circumscribed polygon that encompasses the extracted subject area;
An internal background extraction step for extracting an internal background area excluding the area of the subject from the inside of the first circumscribed polygon;
A second polygon forming step for forming a second circumscribed polygon including the inner background region;
A posture estimation step of estimating a position and a posture of the subject with respect to the photographing unit based on the first circumscribed polygon, the inner background region, and the second circumscribed polygon;
A control step of controlling a part of information displayed on the display unit based on the estimated position and orientation;
The photographed image of the subject grasps the third and fifth fingers to the back side, and the hand in a state where the first and second fingers surround the grasped portion from the front side, the first and second fingers The image is taken from the side, and the region forming stage extracts a skin color region,
The reference point extracted from the internal background region based on the shape of the region, and the reference point corresponding to the corner portion of the second joint by the grasped third to fifth fingers. Estimating the position based on
The posture estimation step estimates the posture based on a positional relationship between the reference point and the internal background region depending on the existence of the back side and the near side,
The posture estimation step further includes determining the shape of the subject based on whether there is an overlapping portion between the side of the first circumscribed polygon and the side of the second circumscribed polygon. Infers whether the second finger is open apart or closed in contact;
The control step is displayed on the display unit based on the movement control step of moving a part of information displayed on the display unit based on the estimated change in position and the estimated change in posture. A rotation control step for rotating a part of the information to be rotated, and a switching control step for switching a state of a part of the information displayed on the display unit based on the estimated shape. . 10. A program for causing a computer to function as the information terminal device according to claim 1.

Images