JP6168597B2 - Information terminal equipment - Google Patents

Description

  The present invention relates to an information terminal device that presents information three-dimensionally, and more particularly to an information terminal device that can control display information on a display unit by changing a relative positional relationship between the device and a user.

  An apparatus that presents information three-dimensionally can provide a user with a sense of reality. The following methods are disclosed as a method for realizing the device.

  In Patent Document 1, a right-eye image and a left-eye image in which a deviation that is inversely proportional to the distance to the subject is prepared, and images are presented to the corresponding eyes by a parallax barrier method, a lenticular lens method, or the like. We have proposed an autostereoscopic display that enables stereoscopic expression.

  Japanese Patent Application Laid-Open No. 2004-228688 proposes an autostereoscopic display that generates an image for each viewpoint by estimating the position of the viewpoint and increases the number of viewpoints that can be stereoscopically viewed. In particular, even when a large depth or protrusion is expressed with respect to the surface of the display device, it is possible to suppress degradation of the resolution and to realize an effect of displaying a stereoscopic image with a high resolution.

JP2012-249060A JP2013-13055A

  In patent document 1, the viewpoint from which a user views a display is set discretely, and there exists a problem that the position which can be felt in three dimensions is limited.

  In Patent Document 2, although the problem of Patent Document 1 is partially solved, a special technique is used to make the parallax between adjacent viewpoint images different by devising a shooting method when shooting a plurality of viewpoint images. There is a problem that it is necessary to generate video data for autostereoscopic viewing.

  Patent Documents 1 and 2 both require a special optical member such as a lenticular lens, and therefore cannot be realized by software alone.

  In view of the above-described problems of the prior art, the present invention displays the information according to the relative positional relationship of the user with respect to the information terminal device by a simple method that does not require dedicated autostereoscopic video data or special members. An object of the present invention is to provide an information terminal device capable of presenting information to be displayed three-dimensionally.

  In order to achieve the above object, the present invention provides an information terminal device having an imaging unit and a display unit, which stores display information as a predetermined three-dimensional object based on the display plane of the display unit. And an estimation unit that detects a user's viewpoint coordinates from a captured image captured by the imaging unit, and estimates a positional relationship between the viewpoint coordinates and a predetermined solid based on a display plane in the display information; The display information read out from the storage unit and displayed on the display unit is processed according to the estimated positional relationship and then displayed on the display unit, whereby the processed display information is detected. And a control unit that allows the user to view the image as a predetermined solid with the display plane as a reference when viewed from the viewpoint coordinates.

  According to the present invention, a positional relationship between a predetermined three-dimensional object based on the display plane of the display unit and the user's viewpoint coordinates detected from the captured image is obtained, and the positional relationship is determined when actually displayed on the display unit. Since display is performed by processing accordingly, stereoscopic display is possible by a simple method that does not require dedicated autostereoscopic video data or special members.

It is a functional block diagram of the information terminal device concerning one embodiment. It is a figure which shows the example of a comparison with the original user and a non-user of the appearance on the display part in this invention. It is a functional block diagram of an estimation part. It is a figure which shows that a viewpoint coordinate detection part and a viewpoint coordinate tracking part are switched and applied on a time series. It is a figure for demonstrating that the solid as originally intended is visible by calculation of the conversion coefficient by a conversion coefficient calculation part, and control of a control part. It is a figure for demonstrating the shadow setting process by a control part. It is a figure for demonstrating calculation of a sun azimuth | direction.

  FIG. 1 is a functional block diagram of an information terminal device according to an embodiment. The information terminal device 1 includes an imaging unit 2, an estimation unit 3, a control unit 4, a storage unit 5, and a display unit 6.

  For example, the information terminal device 1 can be configured as a mobile terminal such as a mobile phone or a smartphone. However, the information terminal device 1 of the present invention is not limited to a mobile terminal, and may be any information terminal device provided with an imaging unit 2 having an imaging function, for example, configured as a computer. May be.

  In addition, although a projector is assumed as the display unit 6, it is sufficient that the display portion is configured as a flat surface, and the display unit 6 is not limited to the projector and may be a display. When a projector is used, an external output or the like may be used as appropriate according to the configuration of the information terminal device 1.

  In addition, 3D-CG (3D computer graphics) model composed of multiple polygons is assumed as display information, but it is not limited to 3D-CG model. Information expressed on a plane such as a document may be used. (However, when displayed on the display unit 6, since it is displayed as information on a plane arranged in space, the information expressed on the plane is substantially a 3D-CG model. Similarly, it can be regarded as a solid.)

  The outline of each part in FIG. 1 is as follows.

  The imaging unit 2 captures a user in real time and obtains a series of captured images in time series. The estimation unit 3 detects the user's viewpoint coordinates in real time from the captured image as a first function, and the positional relationship between the predetermined solid stored in the storage unit 5 and the viewpoint coordinates as a second function. Ask. The storage unit 5 stores information to be displayed on the display unit 6 as a predetermined solid with the display plane of the display unit 6 as a reference. The control unit 4 controls the information read from the storage unit 5 in real time according to the viewpoint coordinates and the positional relationship estimated by the estimation unit 3, and displays the information on the display unit 6.

  The control of the control unit 4 allows the user of the information terminal device 1 to appear as if a solid exists on the display unit 6 that displays on a plane when viewed from its own viewpoint. . Here, when the user changes the viewing position and looks at the display unit 6, the real-time processing by the imaging unit 2, the estimation unit 3, and the control unit 4 is the same as when the solid actually exists in the three-dimensional space. The way it looks will change.

  For example, when the display unit 6 is configured as a wall-mounted display, it is possible to display as if an actual wall clock is present when the user wants to see it. When the user sees the wall from the front, the watch can be seen from the front, and when the user changes the position and looks at an angle with respect to the wall, the watch appears to be tilted so that the watch can be seen from the angle. The clock as a solid will appear to exist. Similarly, when the information represented on the above-mentioned photograph or other plane is shown, the plane will also appear to exist in the three-dimensional space (for example, in the form of a signboard).

  Although the user perceives a solid as described above, information regarding what solid is perceived by what position, size, orientation, color, and the like is display information stored in the storage unit 5. . That is, the display information is a three-dimensional model, and is preset by an administrator or the like and stored in the storage unit 5.

  In addition, under the control according to the present invention, when the display unit 6 is viewed from a “non-user” whose viewpoint coordinates are not estimated, generally, the display unit 6 displays the image to the original user. It is recognized as a distorted “pattern” that is different from the original appearance of the three-dimensional model assumed to be generated. The example is shown in FIG.

  In FIG. 2, the original user's viewpoint P1 that has been estimated for its own viewpoint shown in (1) on the display unit 6 and the “non-user” viewpoint P2 that has not been estimated for its viewpoint shown in (2). And are shown. From the viewpoint P1 of the original user, the display unit 6 looks like R1, paper is arranged on the desk, and a plurality of domino-shaped solids with a predetermined pattern are arranged side by side. . However, from the “non-user” viewpoint P2, the display unit 6 displaying the same at the same time appears to be distorted as R2.

  The following additional processing may be performed. That is, the estimation unit 3 estimates the user's gesture from the captured image in addition to the estimation of the user's viewpoint coordinates, and the control unit 4 performs a predetermined motion corresponding to the gesture on the display solid on the display unit 6. You may control so that it may be performed. For example, in the case of the above-described wall clock, the clock may be controlled to swing according to the movement of the user's hand.

  Hereinafter, details of each part of FIG. 1 will be described.

  The imaging unit 2 continuously captures the user's face at a predetermined sampling period, and outputs the captured image to the estimation unit 3. As the imaging unit 2, a digital camera provided as a standard in a mobile terminal can be used. Alternatively, a stereo camera, a depth sensor, an infrared camera, or the like may be used. The various cameras and sensors may be used in combination.

  The estimation unit 3 detects, as a first function, the coordinates of the user's eyes from the image input from the imaging unit 2 as viewpoint coordinates. The estimation unit 3 also has a positional relationship between the viewpoint coordinates and display information as a predetermined solid (stereoscopic model) stored in the storage unit 5 and based on the display coordinates of the display unit 6 as a second function. Ask.

  The positional relationship is perceived by the user as a three-dimensional model as intended by the administrator who has set the model so that the display information is presented three-dimensionally when the display unit 6 is viewed by the user. As described above, by applying a conversion formula (projective conversion) set in advance, it is obtained in the form of a conversion coefficient when the display information is projected onto the coordinates corresponding to the display plane of the display unit 6.

  The conversion formula (projective conversion) used for estimating the conversion coefficient in the estimation unit 3 and the estimated conversion coefficient are output to the control unit 4. Details of the estimation unit 3 will be described later.

  In the present invention, gesture input by a finger, an indicator stick or the like can be realized as an additional process. In this case, the imaging unit 2 captures not only the face but also the finger, the pointing stick, and the like, and outputs the captured image to the estimation unit 3. The estimation unit 3 obtains operation information by recognizing the motion in the gesture by recognizing the finger in addition to the conversion coefficient obtained from the viewpoint coordinates, and outputs the operation information to the control unit 4.

  The control unit 4 applies the projective transformation based on the conversion coefficient input from the estimation unit 3 to the display information read out from the storage unit 5, processes the display information, and sets the display information to be displayed on the display unit 6. Thus, when displaying on the display unit 6, the appearance changes in the same manner as the real solid according to the viewing angle and position, giving the user a perception as if the real solid exists. be able to.

  The control unit 4 may further perform one or both of the following first process and second process when processing the display information in order to enhance the perceived stereoscopic reality. The first process can be performed after the projective transformation is applied, and the second process can be performed before the projective transformation is applied.

  As a first process, a stereoscopic effect may be further enhanced by applying an image effect that strongly blurs the display information after projective transformation in proportion to the distance of the depth information to simulate defocusing. That is, the display information configured as a three-dimensional object may be more blurred as a part that exists far away from the user's viewpoint coordinates (modeled in advance as a thing).

  Since the display information is configured as a solid, the relative positional relationship in the display information and depth information such as a three-dimensional object can be obtained based on the spatial positional relationship with the viewpoint coordinates estimated by the estimation unit 3. Such information can be used to perform the blurring process.

  As the second process, the control unit 4 may add a shadow of the display information so that the user can easily perceive the three-dimensional display position of the display information. In this case, display information including newly generated shadows is used as display information, and projection control is applied including the shadows to be controlled by the control unit 4. In the example of R2 in FIG. 2 described above, a shadow on a domino solid is also configured as display information.

  The first process and the second process will be described again when the estimation unit 3 and the control unit 4 are described in detail.

  The storage unit 5 stores a plurality of pieces of display information to be displayed on the display unit 6 in a predetermined three-dimensional format with reference to the display plane of the display unit 6 in advance. The user can select from the display information stored in the storage unit 5 by an input operation on the control unit 4 (an input operation on an input interface such as a touch panel (not shown) configured as a part of the control unit 4). Display information can be selected and displayed on the display unit 6. Thus, for example, the user can select which of a plurality of 3D-CG models to display.

  When displaying information on the display unit 6, as described above, the control unit 4 applies the projective transformation based on the conversion coefficient input from the estimating unit 3 to the display information, or, further, when applying the projective transformation. The display information is processed by performing processing that enhances the sense of reality. Thereby, the display information on the display unit 6 is controlled in accordance with the estimation result of the estimation unit 3, and the user is given a perception that an actual solid exists.

  Note that the display information as a three-dimensional object stored in the storage unit 5 may be set so as to change according to the time when it is actually displayed on the display unit 6. In the case of the above-described wall clock, the pendulum may swing at a predetermined period and the hands may indicate time.

  Here, details of the estimation unit 3 will be described. FIG. 3 is a functional block diagram of the estimation unit 3. The estimation unit 3 detects a viewpoint coordinate using a captured image and obtains a conversion coefficient, a viewpoint coordinate detection unit 31, a viewpoint coordinate tracking unit 32, a switch 33, and a conversion coefficient calculation unit 34 as a first configuration, and additional processing, An operation information calculation unit 35 as a second configuration for obtaining operation information of a gesture with a finger or the like using a captured image. Hereinafter, the first configuration will be described.

  The viewpoint coordinate detection unit 31 detects the coordinates of the user's eyes as viewpoint coordinates (spatial coordinates) from the captured image. Here, first, it is necessary to detect an eye in the image coordinates. For the eye detection, for example, detection techniques based on various known feature quantities such as the technique disclosed in Non-Patent Document 2 below can be used.

  [Non-Patent Document 2] “P. Viola and M. Jones,“ Robust real time object detection, ”In IEEE ICCV Workshop on Statistical and Computational Theories of Vision, July 2001.”

  The coordinates in the captured image of the eye are obtained by eye detection based on the feature amount. The actual eye is determined by applying a relationship in a predetermined optical system constituting the imaging unit 2 to the coordinates (x, y) in the image, and a certain direction d (x seen from the imaging unit 2 , y) somewhere on the straight line. Therefore, in addition to the direction determined from the coordinates in the image, the viewpoint coordinate detection unit 31 obtains the distance in the depth direction from the imaging unit 2 to obtain the viewpoint as the coordinates in the real space with the imaging unit 2 as a reference. Find the coordinates.

  The distance in the depth direction may be estimated as a distance inversely proportional to the length between both eyes obtained by eye detection. The inversely proportional constant can be set in advance using configuration information in the predetermined optical system constituting the imaging unit 2, an actually measured value of the length between both eyes, or the like.

  As for the distance in the depth direction, when a depth sensor corresponding to each pixel position of the imaging unit 2 can be used, the value of the depth sensor may be adopted as the distance in the depth direction to measure the viewpoint coordinates. If a stereo camera can be used as the imaging unit 2, the viewpoint coordinates may be calculated by triangulation.

  Note that the viewpoint coordinates may be calculated as a predetermined position of both eyes, or may be calculated as a predetermined position defined from both eyes, such as the midpoint of both eyes.

  In another embodiment, the viewpoint coordinate detection unit 31 may detect a predetermined position of the face as viewpoint coordinates. In this case, a face area is detected by applying a known face detection technique based on feature amount extraction or the like. Further, as a predetermined point such as the center of the face area, a point corresponding to the coordinates in the captured image in the eye detection is obtained. Furthermore, the viewpoint coordinates can be calculated assuming that the distance inversely proportional to the size of the face area (the length of the predetermined portion) corresponds to the distance in the depth direction in the eye detection.

  As described above, the viewpoint coordinate detection unit 31 detects the viewpoint coordinates with high accuracy by extracting the feature amount from the captured image. Next, description will be given from the viewpoint on the time axis.

  In one embodiment, the viewpoint coordinate tracking unit 32 and the switch 33 in FIG. 3 are omitted, and the viewpoint coordinate detection unit 31 detects viewpoint coordinates for a series of all captured images input in time series, and The detection result is output to the conversion coefficient calculation unit 34.

  In another embodiment, the viewpoint coordinate detection unit 31 is intermittently applied to a part of the time series in order to reduce a load caused by continuing application of the viewpoint coordinate detection unit 31, and the application is not performed. In the meantime, the viewpoint coordinate tracking unit 32 is applied instead, and the load is reduced by performing tracking using the result of viewpoint coordinates estimated in the past.

  Note that the switch 33 performs switching as to which of the viewpoint coordinate detection unit 31 and the viewpoint coordinate tracking unit 32 is applied in the other embodiment when obtaining the viewpoint coordinates to be output to the conversion coefficient calculation unit 34. It is shown in order to clarify that processing is performed.

  FIG. 4 is a diagram for schematically showing the switching. Here, each of the captured images on the time series is referred to as a “frame”, and a frame (captured image) with a frame number t is expressed as a frame F (t). As shown in (1), at a certain time t = n, the viewpoint coordinate detection unit 31 is applied to the frame F (n), and the viewpoint coordinates are detected as D (n). Similarly, as shown in (3), at a certain time t = n + N thereafter, the viewpoint coordinate detection unit 31 is applied to the frame F (n + N), and the viewpoint coordinates are D (n + N). Is detected.

  In FIG. 4, the viewpoint coordinates D (n) and the like are present at predetermined positions inside the rectangle surrounded by the dotted line. The rectangle is an example of a small area as a template used in the tracking process described later. In FIG. 4, for simplification of illustration, it is assumed that the viewpoint coordinates exist at a predetermined position inside the small area, and the small area and the viewpoint coordinates are treated as a unit, and the rectangular small area indicated by the dotted line is “ Reference numerals such as “viewpoint coordinates D (n)” are attached.

  On the other hand, at each time t = n + k (k = 1, 2,..., N-1) between (1) and (3), as shown in (2), the frame F (n The viewpoint coordinate tracking unit 32 is applied to (+ k), and the viewpoint coordinates D (n + k) are detected (within the rectangle illustrated by the dotted line in accordance with the above-described simplified scheme). At this time, tracking processing may be performed only in the vicinity of the past detection location D_n, the detection location D (n) of the viewpoint coordinate detection unit 31 in the latest past shown in (1) is included in the past detection location D_n. n) may be used, and the detection result D (n + k-1) of the viewpoint coordinate tracking unit 32 at the time t = n + k-1 (not shown) immediately before the time t = n + k May be used.

  Details of the tracking processing by the viewpoint coordinate tracking unit 32 are as follows. That is, the viewpoint coordinates in the current captured image are detected by tracking using template matching or the like of the local region including the viewpoint coordinates detected in the past captured image. Specifically, using a small region including viewpoint coordinates detected in a past captured image as a template, a region where the sum of squared differences is minimized in the current captured image is searched. Corresponding point of the area that takes the minimum value (for example, if there is a viewpoint coordinate at a predetermined point such as the center of gravity in the template of the past detection, the predetermined point such as the center of gravity in the searched small area) is the viewpoint in the current captured image The coordinates are output to the conversion coefficient calculation unit 34.

  In the example of FIG. 4, at each time t = n + k in (2), as a template, viewpoint coordinates detected in the latest past t = n to which the line-of-sight estimation unit 31 shown in (1) is applied. A small area including D (n) (a small area having a predetermined shape such as a rectangle and a predetermined size) can be used. As described above, the search range may also be limited to a predetermined range near the past detection location D_n.

  In addition, the determination as to which of the viewpoint coordinate tracking unit 32 and the viewpoint coordinate detection unit 31 is applied to each captured image in time series can be made as follows. Since there is no past detection result for the first captured image read in time series, the viewpoint coordinate detection unit 31 is applied.

  In one embodiment, the predetermined period N may be determined, and the viewpoint coordinate detection unit 31 may be applied only once every N times. In one embodiment, the result of the time t at which the result of the viewpoint coordinate tracking unit 32 is determined to be inappropriate is abandoned, the viewpoint coordinate detection unit 31 is applied for the time t, and the subsequent time t + 1 or later The viewpoint coordinate tracking unit 32 may be tried to be applied to the captured image while determining whether or not it is inappropriate (and abandoning when inappropriate). At this time, an upper limit may be set for the number of consecutive times the viewpoint coordinate tracking unit 32 is applied, and the viewpoint coordinate detection unit 31 may be forcibly applied when the upper limit is reached.

  Here, the determination that the result of the viewpoint coordinate tracking unit 32 is inappropriate can be made as follows. That is, the minimum sum of squares of differences obtained in template matching exceeds a predetermined first threshold, and / or the viewpoint coordinates obtained by tracking at the time t and the viewpoint coordinates obtained at the immediately preceding time t-1. If the distance between the two exceeds a predetermined second threshold, it can be determined that the distance is inappropriate.

  Hereinafter, supplementary matters in the tracking processing of the viewpoint coordinate tracking unit 32 will be described.

  “Viewpoint coordinates” when tracking by template matching by the viewpoint coordinate tracking unit 32 does not include information on “distance in the depth direction”, and in the description of the viewpoint coordinate detection unit 31, an eye on a captured image by feature amount extraction is used. It means the coordinates for detection (or face detection). The small area in the template may be defined as one small area including both eyes, or one small area may be defined for each eye.

  On the other hand, regarding the “distance in the depth direction” when the viewpoint coordinate tracking unit 32 is applied, the coordinates detected by the eye detection (or face detection) using the small region are used. What is necessary is just to calculate by the method similar to the case. The viewpoint coordinate tracking unit 32 outputs the “viewpoint coordinates” in the real space obtained by adding the calculated distance information in the depth direction to the conversion coefficient calculating unit 34 as a final output.

  In addition, regarding the determination of the second threshold value in the determination that the result of the viewpoint coordinate tracking unit 32 is inappropriate, the determination is made with respect to the “viewpoint coordinates” in the real space including the distance in the depth direction. May be.

  The conversion coefficient calculation unit 34 is set in advance by the administrator with the intention of stereoscopically displaying the viewpoint coordinates detected as described above (appearing with reference to the display plane of the display unit 6 at the user viewpoint). The positional relationship with the display information stored in the storage unit 5 is calculated in the form of a conversion coefficient in projective conversion. The calculated conversion coefficient is output to the control unit 4 as an output in the first configuration of the estimation unit 3 (configuration other than the operation information calculation unit 35 forming the second configuration).

  The control unit 4 applies a conversion expression (projection conversion conversion expression) preset by the output conversion coefficient. With this application, the display information that has been preset and stored in the storage unit 5 for apparent stereoscopic display is converted into coordinates that are actually displayed on the display unit 6. As a result, the user who sees the display unit 6 can see a solid as intended by the administrator when storing in the storage unit 5 in advance.

  FIG. 5 is a diagram for explaining that a solid as originally intended can be seen by the calculation of the conversion coefficient in the conversion coefficient calculation unit 34 and the control of the control unit 4.

  In FIG. 5, the viewpoint coordinate E, the virtual three-dimensional coordinate X that the user perceives as a solid, the display plane P of the display unit 6, and the actual display plane P of the virtual three-dimensional coordinate X on the display plane P. Display coordinates Y are shown. Here, the solid in the virtual three-dimensional coordinate X intended to be perceived by the user is set in advance with reference to the display plane P, and is stored in the storage unit 5 as display information. Note that (1) and (2) in FIG. 5 will be described later as supplementary matters.

  As illustrated by the solid line arrow, the conversion coefficient calculation unit 34 performs projection conversion when projecting (projecting) the virtual three-dimensional coordinate X of the display information as viewed from the viewpoint coordinate E to the coordinate Y on the display plane P. The matrix M can be calculated by the following equations (1) to (9). For example, the points q1 to q4 in the virtual three-dimensional coordinate X are projected onto the points p1 to p4 in the coordinate Y on the display plane P, as shown on the solid line arrow by the projection.

  On the other hand, when the light beam reaches the user's eye in the direction opposite to the projection direction, the user can see a rectangular signboard that floats vertically on the display plane P at the points q1 to q4 in the virtual three-dimensional coordinate X. Are actually perceived (illusion) as if they existed, but the substance is the distorted shapes p1 to p4 displayed on the display plane P itself in the display unit 6 controlled by the control unit 4. It has become. In FIG. 5, rectangles q1 to q4 are shown to simply illustrate the relationship of projection, but a desired solid such as a 3D-CG model can be displayed.

  The calculation of the projective transformation matrix M will be described. First, since the coordinate Y is on the plane P, the following equation (1) is established.

  In Equation (1), the coordinates X and Y represent the homogeneous coordinates of the respective three-dimensional coordinates (x, y, z), and the plane P represents the coefficient vector of the plane equation ax + by + cz + d = 0. Represent.

  Since the plane coordinate Y is on a straight line connecting the three-dimensional coordinate X and the viewpoint coordinate E, the following equations (2) and (3) hold.

  Note that the viewpoint coordinates E also represent the homogeneous coordinates of the three-dimensional coordinates (x, y, z). Since 0 in the above equations (2) and (3) is a scalar, the following equations (4) and (5) can be obtained.

  Substituting the latter gives the following equations (6), (7), (8).

  Therefore, the projective transformation matrix M is obtained by the following equation (9). Where I is the identity matrix.

  As shown in the following equation (10), the control unit 4 applies the projection transformation matrix M to the virtual three-dimensional coordinates X of the display information (that is, the three-dimensional model stored in the storage unit 5). Then, the coordinate Y on the plane P is calculated.

  If the stereo model is composed of multiple solids and there is an occlusion relationship such as the series of dominoes described in R2 in FIG. Projective transformation may be applied.

  Supplementary matters related to (1) and (2) in FIG. 5 are as follows.

  In the present invention, as shown in (1) of FIG. 5, the viewpoint coordinate E is first analyzed as a spatial coordinate based on the imaging unit 2 by analyzing the captured image, and the estimation unit 3 (the viewpoint coordinate detection unit 31 and the viewpoint coordinate). It is determined by the coordinate tracking unit 32).

  On the other hand, as shown in (2), the positional relationship on the spatial coordinates between the imaging unit 2 and the display unit 6 (the display plane P thereof) is also given in advance as a known one in the present invention. For example, the imaging unit 2 and the display unit 6 are provided fixed in the casing of the information terminal device 1, and information on their positional relationship is acquired.

  As described above, on the premise of (1) and (2) in FIG. 5, in the information terminal device 1, the viewpoint coordinate E, the virtual three-dimensional coordinate X, the display plane P, and the display coordinate Y are shared 3. It can be handled by a dimensional coordinate system, and the above equations (1) to (10) can be calculated.

  That is, although the description is omitted in the above-described case, the estimation unit 3 (the viewpoint coordinate detection unit 31 and the viewpoint coordinate tracking unit 32) obtains the viewpoint coordinates based on the imaging unit 2 from the captured image by the above-described processes. Thereafter, the viewpoint coordinate E is obtained by applying the relationship (2) in FIG. 5 and converting it into the common coordinate system.

  Although the common coordinate system can be arbitrarily determined, the display plane P may be set as a reference. In this case, the display information using various 3D-CG models etc. on the virtual three-dimensional coordinate X presented as a solid to the user is displayed at what position and in what size with respect to the display plane P. The manual setting can be made in advance in consideration of whether or not it looks like. At the time of the manual setting, display information can be prepared using well-known 3D-CAD (3D computer-aided design). The manually set stereoscopic information is stored and used in the storage unit 5 as described above.

  In addition, the positional relationship between the imaging unit 2 and the display unit 6 shown in (2) of FIG. 5 may be obtained in the following manner while allowing a change, instead of giving a fixed value in advance. Good. That is, an AR (augmented reality marker) such as a known square marker is provided at a predetermined position on the display plane P, the marker is imaged by the imaging unit 2, and the estimation unit 3 estimates the position and orientation of the marker. Accordingly, the positional relationship between the imaging unit 2 and the display unit 6 shown in (2) of FIG. 5 may be acquired in real time. Even if the marker is provided not in the display plane P but in a known positional relationship with respect to the display plane P, the same processing can be performed by additionally using the known positional relationship.

  Here, with reference to FIG. 5, the first process (blurring process) and the second process (shadow generation process) for enhancing the reality by the control unit 4 will be described.

  As is clear from FIG. 5 described above, the first processing is performed by calculating the distance d (E, q) between the viewpoint coordinate E and each point q of the solid in the three-dimensional coordinate X in the known blur processing. Use as a blurring degree parameter to blur the actual point p on the display plane P corresponding to the point q of the virtual solid (the position projected on the plane P when q is viewed from E) Is possible.

  The second processing can also be performed by newly generating a “shadow” in the three-dimensional coordinate X by a well-known light source setting processing in the 3D-CAD (three-dimensional computer-aided design) field and other processing. In one example, the generated shadow may be configured to be projected on the display plane P.

  FIG. 6 is a diagram for explaining an example of the second processing. (1) shows a state in which no shadow is set, and (2) and (3) show examples in which a shadow is set for (1). Show. In (1), display information D1 that appears as a solid when viewed from the viewpoint coordinates of the user is shown on the display plane P by processing by the control unit 4. The solid D1 is a rectangular parallelepiped arranged on the display plane P. Although FIG. 6 illustrates an example in which a shadow is generated on the display plane P, it may be generated on another plane.

  In one example, as shown in (2), a region where the display information D1 is in contact with the display plane P1 is enlarged by a predetermined ratio on the plane P1 may be used as the shadow S1. In one example, as shown in (3), a point light source may be arranged at a predetermined position L2 in the coordinate system of the display plane P to generate the shadow S2. Alternatively, as shown in (3), the shadow S2 may be generated by allowing parallel rays to enter from a predetermined direction D2 in the coordinate system of the display plane P.

  Here, the user's viewpoint coordinate E may be used for the predetermined position L2, or a position having a predetermined positional relationship with respect to the viewpoint coordinate E, for example, a predetermined position in the vertical direction of the display plane P from the viewpoint coordinate E. You may use the position raised only by height. For the predetermined direction D2, a solar azimuth (solar height h and solar azimuth angle a) calculated by a method described below with reference to FIG. 7 may be used.

  The solar altitude h and the solar azimuth angle a at the point of latitude L and longitude φ have the following equations (11), (12), and (13) when the spherical trigonometry is applied.

  Here, H and d are hour angle and solar red longitude, respectively. If Japan standard time JST and standard meridian (135 degrees) are used, hour angle H is given by the following equation (14).

  Here, Eq is the difference (equal time difference) between the time of average solar time and the time of true solar time. The time difference Eq is obtained by the following equation (15).

  Here, w = 2π / 365 (the leap year is w = 2π / 366), and J is the total number of days -1 from the first day of the year. Therefore, the solar height h and the solar direction a are given by the following equations (16) and (17).

  Where α is the solar declination and is obtained by the following equation (18).

  Note that the solar direction a is represented by 0 degree in the south, positive in the southwest direction, and negative in the southeast direction.

  As described above, the solar azimuth (solar altitude h and solar azimuth angle a) is calculated. Note that latitude L, longitude φ, Japanese standard time JST, and the total number of days from New Year's Day -1 = J, which are necessary as inputs, are obtained by collecting information from the network, and the angle between the horizontal plane and the display plane P is What is necessary is just to acquire separately by an inclination sensor etc. Or when the installation location of the display plane of the display unit 6 is fixed and known in advance, the time and the total number of days may be given manually.

  Here, returning to FIG. 3, the operation information calculation unit 35 as the second configuration will be described. The operation information calculation unit 35 detects a user's gesture by a predetermined target such as a hand, a finger, an indicator stick, or the like from the captured image by various known methods, and calculates operation information based on the gesture. For example, the operation information is calculated as a velocity vector related to hand movement. In the calculation, the viewpoint coordinates detected in the first configuration may be used as a reference. For example, the operation information as the angular velocity may be calculated by detecting a hand rotated around the viewpoint coordinates.

  The control unit 4 receives the calculated operation information and changes display information as a three-dimensional model before applying projective transformation according to the operation information. For example, the position on the display plane P may be moved based on the detected velocity vector, and the arrangement on the display plane P may be rotated based on the detected angular velocity. If configured with 3D-CG, its parameters (size, etc.) may be changed according to the operation information.

  By applying the conversion coefficient in the first configuration to the changed display information, the solid displayed on the display unit 6 is moved or rotated according to its own gesture from the user's standpoint. Intuitive operation is possible.

  DESCRIPTION OF SYMBOLS 1 ... Information terminal device, 2 ... Imaging part, 3 ... Estimation part, 4 ... Control part, 5 ... Memory | storage part, 6 ... Display part

Claims (8)

An information terminal device having an imaging unit and a display unit,
A storage unit for storing display information as a predetermined solid with reference to a display plane of the display unit;
Detecting the user's viewpoint coordinates from the captured image captured by the imaging section, and estimating the positional relationship between the viewpoint coordinates and a predetermined solid with reference to the display plane in the display information;
The display information read out from the storage unit and displayed on the display unit is processed according to the estimated positional relationship and then displayed on the display unit, whereby the processed display information is detected. A control unit that allows a user to view the image as a predetermined solid with the display plane as a reference when viewed from the viewpoint coordinates ,
The estimation unit further detects a user's gesture from the captured image captured by the imaging unit, calculates operation information based on the gesture,
Wherein the control unit further wherein when controlling so that the display appears plane as predetermined solid relative to the information terminal device characterized that you control varied according to the operation information.   The estimation unit detects the user's eye area from the captured image as viewpoint coordinates, and projects a predetermined solid with the display plane as a reference from the viewpoint coordinates toward the display plane. The information terminal device according to claim 1, wherein the positional relationship is estimated.   The information terminal apparatus according to claim 2, wherein the estimation unit obtains the projection relationship as a projective transformation matrix.   The information terminal apparatus according to claim 1, wherein the estimation unit detects viewpoint coordinates based on a feature amount extracted from the captured image. The imaging unit continuously performs imaging on a time series,
The estimation unit detects viewpoint coordinates by extracting the feature amount for a part of the captured images on the time series, and determines the feature amount in the latest past for the other captured images on the time series. 5. The information terminal device according to claim 4, wherein the viewpoint coordinates are detected by template matching using a small area including the viewpoint coordinates extracted and detected.   The control unit, when performing the processing, displays corresponding to each position based on the distance from the detected viewpoint coordinates to each position in a predetermined solid with reference to the display plane in the display information. 6. The information terminal device according to claim 1, wherein blurring processing is performed on the information.   The information terminal device according to claim 1, wherein the control unit adds a shadow to a predetermined three-dimensional object based on a display plane in the display information in advance when the processing is performed.   The control unit calculates a sun azimuth based on the inclination of the display plane of the imaging unit with respect to a horizontal plane, longitude / latitude and date / time information, and generates the added shadow based on the sun azimuth. The information terminal device according to claim 7.