JP2015022589A - Display device, display method, display program, and information storage medium storing the display program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform video-see-through display having less deviation from a real environment even in a non-planar environment.SOLUTION: The display device includes: display means; an inner side camera; an outer side camera; positional information acquisition means for acquiring positional information of a feature point based on image information including a non-planer environment in a three-dimensional space acquired by the outer side camera; viewpoint position information acquisition means for acquiring viewpoint position information showing a viewpoint position of a user; attitude information acquisition means for acquiring attitude information showing an attitude of the outer side camera; conversion matrix calculation means for calculating a conversion matrix showing a homography conversion approximated to transmission and projection conversion to the screen of the feature point based on the positional information of the feature point, the user viewpoint position information, and the attitude information; and conversion means performing homography conversion of the image information acquired by the outer side camera using the conversion matrix. The display means displays the homograph-converted image information on the screen.

Description

  The present invention relates to a display device, a display method, a display program, and an information storage medium storing the display program.

  In recent years, augmented reality technology has been used. Specifically, for example, in the following Non-Patent Document 1, in a tablet terminal having an inner camera and an outer camera and having a display screen, the user from the inner camera is given a three-dimensional position of a planar environment in a three-dimensional space. There is disclosed a technique for acquiring a three-dimensional position of a viewpoint and displaying an image (video see-through display) so that a screen portion of a terminal is transparent by homography conversion.

Geometric correction method for augmented reality on tablet-type information terminals, Takuya Kisegawa, Master's thesis at Nara Institute of Science and Technology (NAIST-IS-MT0851027), February 2, 2012

  However, the above-described conventional technique has a problem that the environment to which it is applied is limited. Specifically, the homography transformation used in the above prior art can only be applied to a planar environment, and its application to a non-planar environment has been a problem.

  The present invention is not limited to the above shooting environment in view of the above problems, and can display an image in which the screen portion of the terminal can be seen through with less deviation from the actual environment even in a non-planar environment. It is an object to provide a display device, a display method, a display program, and an information storage medium storing the display program.

  The display device according to the present invention includes a display unit including a screen, an inner camera arranged in the same direction as a display direction of the display unit, an outer camera arranged in a direction opposite to the display direction, Based on the image information including the non-planar environment in the three-dimensional space acquired by the outer camera, the position information acquiring means for acquiring the position information of the feature point in the three-dimensional space, and the image information acquired by the inner camera Viewpoint position information acquisition means for acquiring viewpoint position information representing the viewpoint position of the user based on, attitude information acquisition means for acquiring attitude information representing the attitude of the outer camera, position information of the feature points, user viewpoint position Based on the information and the posture information, a transformation matrix representing a homography transformation that approximates a transmission projection transformation of the feature points onto the screen is calculated. Conversion matrix calculation means; and conversion means for performing homography conversion on the image information acquired by the outer camera using the conversion matrix, and the display means displays the image information subjected to the homography conversion on the screen. It is characterized by displaying.

  The display method of the present invention comprises display means including a screen, an inner camera arranged in the same direction as the display direction of the display means, and an outer camera arranged in a direction opposite to the display direction. In the display device, the position information acquisition unit acquires the position information of the feature points in the three-dimensional space based on the image information including the non-planar environment in the three-dimensional space acquired by the outer camera, and the viewpoint position information The acquisition means acquires viewpoint position information representing the viewpoint position of the user based on the image information acquired by the inner camera, the attitude information acquisition means acquires attitude information representing the attitude of the outer camera, and a transformation matrix Based on the position information of the feature point, the user viewpoint position information, and the posture information, the calculation means transmits the feature point to the screen. A transformation matrix representing a homography transformation that approximates a projection transformation is calculated, the transformation means performs homography transformation on the image information acquired by the outer camera using the transformation matrix, and the display means The converted image information is displayed on the screen.

  The display program of the present invention includes display means including a screen, an inner camera arranged in the same direction as the display direction of the display means, and an outer camera arranged in a direction opposite to the display direction. In the computer system, based on image information including a non-planar environment in the three-dimensional space acquired by the outer camera, position information acquisition means for acquiring position information of feature points in the three-dimensional space, acquired by the inner camera Viewpoint position information acquisition means for acquiring viewpoint position information indicating the viewpoint position of the user based on the obtained image information, attitude information acquisition means for acquiring attitude information indicating the attitude of the outer camera, position information of the feature point, the user Based on the viewpoint position information and the posture information, the homogenous that approximates the transmission projection conversion of the feature points onto the screen A transformation matrix calculation means for calculating a transformation matrix representing a phy transformation; and a computer system that functions as a transformation means for performing homography transformation on the image information acquired by the outer camera using the transformation matrix, and the display means The image information subjected to the homography conversion is displayed on the screen.

It is a figure which shows an example of the external shape of the display apparatus in 1st Embodiment. It is a figure which shows the outline | summary of the hardware constitutions of the display apparatus in 1st Embodiment. It is a figure which shows an example of a functional structure of the control part shown in FIG. It is a figure for demonstrating the perspective projection conversion by a user's viewpoint position and outer camera using a pinhole camera model. It is a figure for demonstrating an example of a virtual object. It is a figure for demonstrating another example of a virtual object. It is a figure which shows an example of the outline | summary of the flow of the display apparatus in 1st Embodiment. It is a figure which shows an example of a functional structure of the control part in 2nd Embodiment.

  Embodiments of the present invention will be described below with reference to the drawings. In addition, about drawing, the same code | symbol is attached | subjected to the same or equivalent element, and the overlapping description is abbreviate | omitted.

[First embodiment]
FIG. 1 is a diagram illustrating an example of an outer shape of a display device according to this embodiment. As shown in FIG. 1, the display device 100 is a portable terminal such as a tablet, for example, and includes a screen 101, an inner camera 102, and an outer camera 103.

  Specifically, for example, as illustrated in FIG. 1A, the display device 100 has a plate shape, and includes a flat screen 101 on a first surface of the display device 100. For example, the screen 101 may function as a touch panel as well as function as a screen 101 that displays an image.

  The display device 100 has an inner camera 102 at a predetermined position around the screen 101. In addition, as shown in FIG. 1B, an outer camera 103 is provided at a predetermined position on the second surface opposite to the first surface of the display device 100. Here, as the inner camera 102 and the outer camera 103, for example, CCD cameras are used.

  Note that the shape of the display device 100 illustrated in FIG. 1 and the positions and sizes of the screen 101, the inner camera 102, and the outer camera 103 are merely examples, and the present embodiment is not limited thereto. Absent.

  FIG. 2 is a diagram for describing an outline of the hardware configuration of the display device according to the present embodiment. As shown in FIG. 2, the display device 100 according to the present embodiment includes, for example, a control unit 201, a storage unit 202, an operation unit 203, a display unit 204, an inner camera 102, and an outer camera 103. Each unit is connected to each other by an internal bus 109.

  The control unit 201 is, for example, an MPU or CPU, and operates according to a program stored in the storage unit 202.

  The storage unit 202 is an information recording medium configured by an information recording medium such as a ROM or a RAM, and holds a program executed by the control unit 201. The storage unit 202 also operates as a work memory for the control unit 201. The program may be provided by being downloaded via a network or the like, or may be provided by various information storage media that can be read by a computer such as a card-type memory.

  The operation unit 203 includes, for example, an interface such as a plurality of buttons or a touch panel, and outputs the content of the instruction operation to the control unit 201 in accordance with a user instruction operation. A display unit 204 corresponds to the screen 101 and is, for example, a liquid crystal display, an organic EL display, or the like, and displays information according to an instruction from the control unit 201. In addition, the said structure is an example, Comprising: This Embodiment is not limited to this. For example, the display device 100 may further have a telephone function and a communication function.

  Next, an example of a functional configuration of the control unit 201 will be described. FIG. 3 is a diagram illustrating an example of a functional configuration of the control unit illustrated in FIG. 2.

  As illustrated in FIG. 3, the control unit 201 functionally includes, for example, an inner camera image information acquisition unit 301, an outer camera image information acquisition unit 302, a viewpoint position information acquisition unit 303, a position information acquisition unit 304, and posture information acquisition. Unit 305, conversion matrix calculation unit 306, conversion unit 307, virtual object information acquisition unit 308, determination unit 309, and superimposition unit 310.

  The inner camera image information acquisition unit 301 acquires image information captured by the inner camera 102. The outer camera image information acquisition unit 302 acquires image information captured by the outer camera 103.

  The viewpoint position information acquisition unit 303 acquires viewpoint position information of the user who uses the display device 100 based on image information captured by the inner camera 102. Specifically, the viewpoint position information acquisition unit 303 acquires the viewpoint position based on the position of the user's eyes included in the image information captured by the inner camera 102.

  The position information acquisition unit 304 acquires position information of feature points in the actual environment based on the image information acquired by the outer camera 103. In addition, the posture information acquisition unit 305 acquires posture information representing the posture of the outer camera 103. Here, the posture information and the position information of the feature point are acquired by simultaneously estimating in real time using the well-known Visual SLAM, for example.

A transformation matrix calculation unit 306 calculates a homography transformation matrix based on the feature point position information, posture information, and viewpoint position information. Specifically, for example, the sum of the distances d between the position where each feature point in the real environment is perspective-projected on the screen 101 and the position where the feature point is homographically transformed on the screen 101 is minimized. A homography transformation matrix is calculated. More specifically, for example, it is calculated using the following formula (1). As the distance, for example, other distances may be used in addition to the square distance.

In the above equation (1), F represents a set of three-dimensional positions of feature points, and H represents a homography transformation matrix. P _UPR represents a transformation matrix indicating a perspective projection transformation to a position on the screen 101 in the pinhole camera model with the user's viewpoint position as the projection center, and P _DPR represents the lens center of the outer camera as the projection center. 3 represents a transformation matrix indicating a perspective projection indicating transformation of a feature point in a pinhole camera model into a position on an image of an outer camera.

Here, the pinhole camera model is a model that approximates the mechanism of image formation by a camera by perspective projection conversion without aberration by a lens. In FIG. 4, Viewpointv corresponds to the viewpoint position information, Display D corresponds to the screen 101, and x corresponds to the feature point position information. Further, u _D characteristic point x corresponds to the perspective projection transformation positions on the screen 101, u _C1 corresponds to the position on the image of the outer camera 103 feature points. Furthermore, the camera C ₁ corresponds to the outer camera. Since details of the pinhole model and the like are well known, detailed description is omitted. Further, as can be seen from FIG. 4, the above v, x, u _D , u _C1, etc. represent vectors.

  The conversion unit 307 uses the homography conversion matrix calculated as described above to perform homography conversion on the image information acquired by the outer camera 103 based on the calculated homography conversion matrix. The virtual object information acquisition unit 308 acquires virtual object information including position information of feature points of one or more virtual objects stored in advance in the storage unit 202, for example.

  Here, the virtual object is, for example, an advertisement display or a guidance display, and in this embodiment, as described later, when it is determined that an object corresponding to the virtual object exists in the real environment. This corresponds to an object displayed superimposed on the object displayed on the screen 101. More specifically, for example, as shown in FIG. 5, an object including the letters of University A displayed so as to show the building of University A displayed on the screen 101 as a video see-through display, or as shown in FIG. 6. This corresponds to the object identified and displayed in a predetermined manner on a predetermined book displayed on the screen as a video see-through display. Note that the image information of each virtual object is stored in the storage unit 202 in association with the feature point of the corresponding virtual object, for example.

  The determination unit 309 compares the virtual object information with the viewpoint position value information acquired by the viewpoint position information acquisition unit 303 and the positional relationship between the feature points acquired by the position information acquisition unit 304, and is acquired by the outer camera 103. It is determined whether or not there is an object in which a preset virtual object is superimposed in the image information.

  The superimposing unit 310 superimposes the virtual object on the image information that has been subjected to the homography conversion in accordance with the determination result of the determining unit 309. When the determination unit 309 determines that there is no object on which the virtual object is to be superimposed, the display unit 204 displays the image information subjected to the homography conversion. In other words, the virtual environment is simply not displayed, but the real environment is simply displayed as video see-through.

On the other hand, when the determination unit 309 determines that there is an object on which the virtual object is to be superimposed, the image information on which the virtual object is superimposed on the image information obtained by homography conversion is displayed. For example, in this case, the superimposing unit 310 converts the image information obtained by the homography conversion and the virtual object (the object in the real environment corresponding to the virtual object) using the HP _DPR (acquired by the outer camera 103 using the P _DPR ). The display unit 204 displays the synthesized image information.

Note that image information obtained by perspective projection conversion (converted by _PUPR ) of the virtual object whose viewpoint position is the projection center may be combined with the image information obtained by the homography conversion. In the above description, the case of actual superimposition has been described, but the superimposition in the present embodiment includes the case of displaying other formats such as indicating an object corresponding to a virtual object.

  Next, an example of an outline of the flow of the display device 100 will be described with reference to FIG. The following flow is an example, and the flow in the present embodiment is not limited to the following. Further, in the following, for the sake of simplicity of explanation, a case where a virtual object is displayed in a superimposed manner will be described as an example.

  First, the outer camera 103 and the inner camera 102 each acquire image information (S101). The position information acquisition unit 304 acquires the position information of feature points based on the image information acquired by the outer camera 103 (S102). The posture information acquisition unit 305 acquires posture information representing the posture of the display device 100 (S103).

  The viewpoint position information acquisition unit 304 acquires the viewpoint position information of the user based on the image information from the inner camera 102 (S104). The transformation matrix calculation unit 306 calculates a transformation matrix based on the feature point position information, posture information, and viewpoint position information (S105). The conversion unit 307 performs homography conversion on the image information acquired by the outer camera 103 based on the conversion matrix (S106).

  The superimposing unit 310 superimposes the corresponding virtual object on the image information subjected to the homography conversion in accordance with the determination result of the determination unit 309 described above (S107). The display unit 204 displays the image information subjected to the homography conversion on the screen 101 (S108).

  Note that the flow shown in FIG. 7 is merely an example, and the present embodiment is not limited to this. For example, the position information acquisition of the feature point in S102 and the attitude information acquisition in S103 may be performed simultaneously as described above. Further, the acquisition of the viewpoint position information in S104 may be performed simultaneously with S102 and S103. Furthermore, in the above description, the display device 100 that realizes so-called augmented reality by superimposing virtual objects has been described. However, the video see-through display device 100 that does not superimpose virtual objects is realized by omitting S107. You may comprise so that a user can select.

  According to this embodiment, even in a non-planar environment, an image can be displayed (video see-through display) so that the screen portion of the terminal is transparent with less deviation from the actual environment. Further, for example, the above-described image can be displayed in real time only by a device that can be mounted on a portable terminal such as a tablet. Furthermore, when superimposing virtual objects, it is possible to allow the user to recognize a real object corresponding to the virtual object in a shorter search time.

  The present invention is not limited to the above-described embodiment, and is substantially the same configuration as the configuration shown in the above-described embodiment, a configuration that exhibits the same operational effects, or a configuration that can achieve the same purpose. May be replaced.

For example, the calculation of the homography transformation matrix by the transformation matrix calculation unit 306 may be configured to further calculate the homography transformation matrix based on the feature points of the virtual object. Specifically, for example, as shown in the following formula (2), the feature point of the virtual object is included in the feature point, and the homography transformation matrix (H ′) that minimizes E (H ′) is calculated.

In this case, the conversion unit 307 performs homography conversion on the image information from the outer camera 103 at the above H ′, and after performing perspective projection conversion on the virtual object by the above P _DPR , performs the homography conversion at the above H ′, and performs the superposition unit 310 Combines these conversion results, and the display unit 204 displays the combined image information.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the present embodiment, mainly in the calculation of the homography transformation matrix, each of the positions obtained by perspective projection transformation of the feature points on the screen 101 and the positions obtained by homography transformation of the feature points on the screen 101 are used. The point of calculating a homography transformation matrix that minimizes the sum after performing a predetermined weighting on the distance is different from the first embodiment. In the following, description of the same points as in the first embodiment will be omitted.

  FIG. 8 is a diagram illustrating an example of a functional configuration of the control unit in the present embodiment. As shown in FIG. 8, in the present embodiment, the control unit 201 further includes a region of interest detection unit 311 and a weighting unit 312 functionally.

  The attention area detection unit 311 extracts the attention area that the user gazes on based on the image information from the inner camera 102. Specifically, for example, a predetermined range is extracted as a region of interest from a gazing point based on the user's line of sight. The weighting unit 312 determines the weighting value so that the weighting value for each feature point included in the region of interest is larger than the other feature points.

Then, the transformation matrix calculation unit 306 corresponds to each feature point to each distance between the position where each feature point is perspective-projected on the screen 101 and the position where each feature point is homographically transformed on the screen 101. After weighting with the weighting value, a homography transformation matrix that minimizes the sum is calculated. Specifically, for example, the conversion matrix calculation unit 306 calculates a conversion matrix based on the following equation (3).

In the above equation (3), w _x represents a value to be weighted. Further, the weighting unit 312 calculates the weighting value using, for example, the following formula (4).

Here, a corresponds to the position of the gazing point, and r _th indicates a predetermined threshold value. As can be seen from the above equation (4), the weighting unit 312 is, for example, w ₀ (w ₀₎ when the distance between the gazing point a and the feature point projected on the screen 101 is equal to or less than a predetermined threshold value r _th. Is a value larger than 1, for example, 2 to the nth power (where n is a natural number) is a weighting value, and when it is larger than a predetermined threshold value r _th , 1 is a weighting value. Since other points are the same as those in the first embodiment, description thereof will be omitted.

  According to the present embodiment, as in the first embodiment, an image can be displayed so that the screen portion of the terminal is transparent in the non-planar environment with less deviation from the actual environment. Further, for example, the above-described image can be displayed in real time only by a device that can be mounted on a portable terminal such as a tablet. Furthermore, when superimposing virtual objects, it is possible to allow the user to recognize a real object corresponding to the virtual object in a shorter search time.

  Furthermore, according to the present embodiment, it is possible to further suppress the occurrence of a position shift at an important part where the user can easily recognize the position shift.

  The weighting is an example, and the present embodiment is not limited to this. For example, in the above description, the case where the feature points included in the region of interest are weighted more than other feature points has been described. However, a highly significant pattern is stored in advance, and is included in the highly significant pattern. A larger weighting value may be given to the distance corresponding to the feature point. Thereby, for example, it is possible to suppress the occurrence of a position shift at an important part where the user can easily recognize the position shift.

Further, as the feature point reaches the periphery of the viewing angle of the outer camera 103, the weight to the distance corresponding to the feature point may be reduced. In this case, specifically, for example, the weighting unit 312 obtains a weighting value using the following equation (5).

  In the above equation (5), c represents the position of the image center, and γ represents an increase rate count of a predetermined weight. Thereby, when moving the display apparatus 100, the jitter of the display image which arises when the set of the feature points acquired by the outer camera 103 changes in the moving direction can be reduced. The above weights may be used in combination or may be configured so that the user can select them.

  The present invention is not limited to the above-described embodiment, and is substantially the same configuration as the configuration shown in the above-described embodiment, a configuration that exhibits the same operational effects, or a configuration that can achieve the same purpose. May be replaced.

  100 display device, 101 screen, 102 inner camera, 103 outer camera, 201 control unit, 202 storage unit, 203 operation unit, 204 display unit, 301 inner camera image information acquisition unit, 302 outer camera image information acquisition unit, 303 viewpoint position Information acquisition unit, 304 position information acquisition unit, 305 posture information acquisition unit, 306 transformation matrix calculation unit, 307 conversion unit, 308 virtual object information acquisition unit, 309 determination unit, 310 superimposition unit, 311 attention area detection unit, 312 weighting unit .

Claims (12)

Display means including a screen;
An inner camera arranged in the same direction as the display direction of the display means;
An outer camera arranged in a direction opposite to the display direction;
Position information acquisition means for acquiring position information of feature points in the three-dimensional space based on image information including a non-planar environment in the three-dimensional space acquired by the outer camera;
Viewpoint position information acquisition means for acquiring viewpoint position information representing the viewpoint position of the user based on the image information acquired by the inner camera;
Attitude information acquisition means for acquiring attitude information representing the attitude of the outer camera;
A transformation matrix calculation unit that calculates a transformation matrix representing a homography transformation that approximates a transmission projection transformation of the feature point onto the screen based on the position information of the feature point, the user viewpoint position information, and the posture information. When,
Image information acquired by the outer camera, and conversion means for performing a homography conversion using the conversion matrix,
The display means displays the image information subjected to the homography conversion on the screen.
A display device characterized by that.   The homography transformation matrix calculation means minimizes a distance between a position obtained by perspective projection transformation of the feature points on the screen and a location obtained by homography transformation of the feature points on the screen using the transformation matrix. The display device according to claim 1, wherein a conversion matrix is calculated. The feature points are a plurality of feature points,
The homography transformation matrix calculation means calculates each distance between a position obtained by perspective projection transformation of the feature points on the screen and a position obtained by homography transformation of the feature points on the screen using the transformation matrix. The display device according to claim 1, wherein a transformation matrix that minimizes a sum of values weighted with a predetermined weight for each feature point is calculated. The display device further includes:
Based on image information acquired by the inner camera, attention area detection means for extracting a user's attention object;
Weighting means for obtaining a weighting value such that a weighting value for each feature point included in the region of interest among the plurality of feature points is larger than other feature points;
The weighting is performed based on the weighting value.
The display device according to claim 3.   The display device according to claim 1, further comprising a superimposing unit that superimposes a virtual object on the image information subjected to the homography conversion.   6. The display device according to claim 5, wherein the superimposing unit superimposes the virtual object on the screen through transmission projection conversion, and superimposes the virtual object on the image information subjected to the homography conversion.   The superimposing unit superimposes the image information obtained by performing the perspective projection conversion on the image surface of the outer camera and then performing the homography conversion on the image information subjected to the homography conversion. 5. The display device according to 5. The display device further includes virtual object information acquisition means for acquiring virtual object information representing position information of feature points on the virtual object,
The display device according to claim 5, wherein the transformation matrix calculation unit further calculates the homography transformation matrix based on position information of the virtual object.   The superimposing unit superimposes the image information obtained by performing the perspective projection conversion on the image surface of the outer camera and then performing the homography conversion on the image information subjected to the homography conversion. 9. The display device according to 8. Display means including a screen;
An inner camera arranged in the same direction as the display direction of the display means;
In a display device including an outer camera disposed in a direction opposite to the display direction,
Based on the image information including the non-planar environment in the three-dimensional space acquired by the outer camera by the position information acquisition means, the position information of the feature points in the three-dimensional space is acquired,
The viewpoint position information acquisition means acquires viewpoint position information representing the viewpoint position of the user based on the image information acquired by the inner camera,
Attitude information acquisition means acquires attitude information representing the attitude of the outer camera,
Based on the position information of the feature points, the user viewpoint position information, and the posture information, a transformation matrix that represents a homography transformation that approximates the transmission projection transformation of the feature points onto the screen is obtained by a transformation matrix calculation unit. Calculate
The image information acquired by the outer camera is converted by the conversion means using the conversion matrix, and the homography is converted.
The display means displays the image information subjected to the homography conversion on the screen.
A display method characterized by that. Display means including a screen;
An inner camera arranged in the same direction as the display direction of the display means;
A computer system including an outer camera disposed in a direction opposite to the display direction;
Position information acquisition means for acquiring position information of feature points in the three-dimensional space based on image information including a non-planar environment in the three-dimensional space acquired by the outer camera;
Viewpoint position information acquisition means for acquiring viewpoint position information representing the viewpoint position of the user based on the image information acquired by the inner camera;
Attitude information acquisition means for acquiring attitude information representing the attitude of the outer camera;
A transformation matrix calculation unit that calculates a transformation matrix representing a homography transformation that approximates a transmission projection transformation of the feature point onto the screen based on the position information of the feature point, the user viewpoint position information, and the posture information. And conversion means for performing homography conversion on the image information acquired by the outer camera using the conversion matrix,
As a computer system
The display means displays the image information subjected to the homography conversion on the screen.
A display program characterized by that.   A computer-readable information storage medium storing the display program according to claim 11.

Images