JP2015184391A - Image display method, device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve such a problem that an AR (Augmented Reality) device visible in a user's eyes cannot provide an image display with a sense of unity for an actual landscape conventionally.SOLUTION: An AR device 10 is grounded on a paper surface 20 and moved. An area to be cut out from image data 30 with information is moved by a same amount as the movement and is displayed on an image display section 13. The image display section 13 creates an illusion as if a user directly views the paper surface 20 through a transparent glass so as to make the user feel a sense of unity. Further, the user feels as if information 32 is written on the transparent glass portion. The start and finish of the AR movement is automated by detecting a change in acceleration in the X-axis direction.

Description

The present technology relates to a method, an apparatus, and a program for photographing an object to be photographed and displaying information superimposed on the photographed image.

In recent years, applications using augmented reality (AR) have been realized. For example, when a target object such as a paper surface is photographed using a smartphone or a tablet PC, the application is displayed by superimposing information on the photographed image. More specifically, it is an application called “Tokyo AR” provided by the Tokyo head office of Chunichi Shimbun. Also, FIG. 1 of Non-Patent Document 1 describes an example of a similar application. Further, Patent Document 1 discloses a similar application.

Specific procedures for using these applications will be described with reference to the drawings.

100 in FIG. 1A is a conventional AR device (specifically, a smartphone or a tablet PC). The AR device 100 includes an image display unit 110 and an imaging unit 120. The imaging unit 120 is installed on the opposite side of the image display unit 110 and is not visible in the figure. The inside of the range indicated by the four dotted lines 130, 131, 132 and 133 is imaged by the imaging unit 120. Incidentally, the inside of the range indicated by the four dotted lines is called the viewing angle. Reference numeral 200 denotes a paper surface. The page is, for example, a specific page of a magazine. Or it is one side of a newspaper. FIG. 1A shows an example in which a photograph of a car is posted on the paper surface 200. When the user uses the imaging unit 120 to photograph the paper surface 200, the AR device 100 analyzes the image, performs desired processing, and displays the image and information on the image display unit 110. That is, the image 300 ((A) in FIG. 1) photographed by the imaging unit 120 is displayed on the image display unit 110, and information 310 ((U) in FIG. 1) is also superimposed and displayed. In the example of FIG. 1, the information 310 is text information “You can test ride at the exhibition hall”. In the image display unit 110 of FIG. 1A, such an image 300 and information 310 are displayed.

Reference numeral 400 in FIG. 1A denotes the user's eyes. FIG. 1A shows a scene where the user is looking at the image display unit 110.

A view seen from the user's eye 400 is shown in FIG. As shown in FIG. 2, the AR device 100 is visible to the user. An image 300 and information 310 can be seen on the image display unit 110 in the AR device 100. In addition, a real world landscape can be seen outside the AR device 100. That is, the paper surface 200 can be seen. In FIG. 2, the paper surface 200 and the image 300 in the image display unit 110 do not have a sense of unity. This is because the size of the target automobile is different.

Let us consider a case where only a part of the photographed image 300 (dotted line rectangular region indicated by 320 in FIG. 1A) is displayed on the image display unit 110. In this case, the information 310 also causes the image display unit 110 to display only the portion 330 in FIG. It is assumed that the positional relationship of the partial image 320 with respect to the image 300 and the positional relationship of the partial information 330 with respect to the information 310 are the same. In this case, the view seen from the user's eyes 400 is shown in FIG. As shown in FIG. 3, the AR device 100 is visible to the user. The partial image 320 and the partial information 330 are visible on the image display unit 110 in the AR device 100. In addition, a real world landscape can be seen outside the AR device 100. That is, the paper surface 200 can be seen. Unlike FIG. 2, the paper surface 200 and the image 320 in the image display unit 110 have a sense of unity. This is because they are the same size. It is as if the image display unit 110 is transparent glass, and the portion of the image display unit 110 can cause an illusion that the user is looking directly at the paper surface 200 through the transparent glass. Feel the experience. And it feels as if the partial information 330 is written on the transparent glass part.

Instead of displaying the entire photographed image 300 on the image display unit 110, by displaying an appropriate part of the image 300 (partial image 320), it is possible to display an image with a sense of unity with respect to the actual landscape. can do. However, since the area of the partial image suitable for display depends on the positional relationship among the user's eyes 400, the image display unit 110, and the paper surface 200, an appropriate partial image is obtained as long as the positional relationship is unknown. I could not. That is, it was impossible to present a scene as shown in FIG.

Suppose that the scene shown in FIG. 3 can be presented. Consider the case where the user moves the AR device 100 after the display. In this case, in order to display an image with a sense of unity even after movement, the area of the partial image cut out from the image 300 must be changed. However, this area could not be changed. This is because the appropriate partial image area to be displayed after movement depends on the positional relationship among the user's eyes 400, the image display unit 110, and the paper surface 200 after the movement, and the positional relationship is unknown. is there.

In summary, the prior art has the following drawbacks.
(Disadvantage 1) The AR device 100 visible from the user's eyes 400 cannot display an image with a sense of unity with respect to an actual landscape.
(Disadvantage 2) Even if it is possible to display an image with a sense of unity, if the user subsequently moves the AR device 100, the image cannot be displayed with a sense of unity after the movement. .
(Disadvantage 3) The user is tired because the AR device 100 must be held in the air.
(Defect 4) Since the imaging unit 120 always keeps shooting, power is consumed continuously.

JP 2012-195732 A

Schmalstieg, Dieter, and Daniel Wagner. "Mobile phones as a platform for augmented reality." Connections 1 (2009): 3.

As described above, augmented reality (AR) can be realized by using the techniques described in Patent Document 1 and Non-Patent Document 1, but there is no sense of unity regarding display. In other words, again, the prior art has the following drawbacks.
(Disadvantage 1) The AR device 100 visible from the user's eyes 400 cannot display an image with a sense of unity with respect to an actual landscape.
(Disadvantage 2) Even if it is possible to display an image with a sense of unity, if the user subsequently moves the AR device 100, the image cannot be displayed with a sense of unity after the movement. .
(Disadvantage 3) The user is tired because the AR device 100 must be held in the air.
(Defect 4) Since the imaging unit 120 always keeps shooting, power is consumed continuously.

The present technology has been made in view of such a situation, and is intended to display an image with a sense of unity. Also, the user does not have to hold the AR device. Further, since shooting is not performed during the AR operation, power consumption can be reduced.

An image display method according to one aspect of the present technology is an image display method for displaying only a specific area from an image on an image display device, the image data storing step for storing the image in a memory, and the image display device. An area that is moved by the same amount as the movement amount measured in the position measurement step with respect to the specific area at the time immediately before the position measurement step that measures the movement amount is set as the specific area at the current time. As described above, a region updating step for changing the specific region and a display step for displaying only the specific region in the image on the image display device are included. Thereby, it is possible to display an image with a sense of unity.

When an impact is detected with respect to the image display device, a process start step for performing a process of starting display on the image display device, or when the movement of the image display device toward the display surface is detected, the image display You may further include at least any one step of the process end step which performs the process which complete | finishes the display to an apparatus. Thereby, the display can be automatically started or ended.

An initial stage for calculating a position at the time of starting the display on the image display device by analyzing the image taken by the imaging means installed in the image display device before the time of starting the display on the image display device A position calculation step may be further included. Thereby, it is possible to display an image with a sense of unity.

The specific area may be updated in the area update step under a condition that the specific area is the same size as the display unit of the image display device. Thereby, it is possible to display an image with a sense of unity.

The image of the specific area displayed on the image display device in the display step is rotated by 90 degrees or -90 degrees and displayed in accordance with the inclination of the image display device at the time of starting the display on the image display device. Anyway. Thereby, it is possible to display an image with a sense of unity.

An image display device according to an aspect of the present technology is an image display device that displays only a specific area from an image, and includes a memory unit that stores the image and a position measurement unit that measures a movement amount of the image display device. The specific area is changed so that the area moved by the same amount as the movement amount measured by the position measurement unit with respect to the specific area at the immediately preceding time is set as the specific area at the current time. And a display control unit that causes the image display device to display only the specific region of the image. Thereby, it is possible to display an image with a sense of unity.

An image display program according to an aspect of the present technology is an image display program for displaying only a specific area from an image on an image display device, the image data storing step for storing the image in a memory, and the image display device An area that is moved by the same amount as the movement amount measured in the position measurement step with respect to the specific area at the time immediately before the position measurement step that measures the movement amount is set as the specific area at the current time. As described above, a region updating step for changing the specific region and a display step for displaying only the specific region in the image on the image display device are included. Thereby, it is possible to display an image with a sense of unity.

In one aspect of the present technology, the display is an image display in which only a specific region is displayed on the image display device, the image is stored in a memory, the movement amount of the image display device is measured, and the time immediately before The specific area is changed so that the area moved by the same amount as the measured movement amount is the specific area at the current time, and the specific area of the image is changed. Only the image is displayed on the image display device. Thereby, it is possible to display an image with a sense of unity.

According to one aspect of the present technology, an image with a sense of unity can be displayed. Also, the user can avoid holding the AR device. Further, it is possible not to perform shooting during the AR operation.

It is a figure for demonstrating the conventional apparatus. It is a figure for demonstrating the display state without the sense of unity by the conventional apparatus. It is a figure for demonstrating the ideal display method with a sense of unity. It is a figure for demonstrating the apparatus of the Example of this invention. It is a figure for demonstrating the paper surface made into object. It is a figure for demonstrating the image and information which are downloaded to the apparatus of the Example of this invention. It is a figure for demonstrating the positional relationship of each site | part installed in the apparatus of the Example of this invention. It is a flowchart explaining the process with the apparatus of the Example of this invention. It is a flowchart explaining the process with the apparatus of the Example of this invention. It is a flowchart explaining the process with the apparatus of the Example of this invention. It is a figure explaining the process with the apparatus of the Example of this invention. It is a figure explaining the process with the apparatus of the Example of this invention. It is a figure explaining the process with the apparatus of the Example of this invention. It is a figure explaining the usage method in the Example of this invention.

The advantages of the present invention are first described. The advantages of the present invention are as follows when each of the above-mentioned conventional defects (defects 1) to (defects 4) is dealt with.
(Advantage 1) Since the AR device is grounded on the paper surface, only an appropriate part of the image is displayed regardless of the position of the user's eyes, thereby displaying an image with a sense of unity with respect to the actual landscape. Can be made. In other words, the image display unit can appear as if it is transparent glass.
(Advantage 2) When the user moves the AR device, by moving the area cut out from the image by the same amount as that amount, an image with a sense of unity can be displayed after the movement.
(Advantage 3) Since the AR device is operated while being grounded on the paper surface, the user does not need to hold the AR device.
(Advantage 4) During the AR operation, since shooting is not performed, an operation with low power consumption is possible.
In addition, there are the following advantages.
(Advantage 5) Since the contact with the paper surface is automatically detected by detecting the impact on the AR device, the user does not need to explicitly give an instruction to start the operation of the AR device (that is, display start). .
(Advantage 6) When the user moves the AR device away from the page, this operation is detected by the acceleration sensor in the AR device and the AR operation is terminated. Therefore, the user explicitly ends the operation of the AR device ( That is, it is not necessary to instruct the display end).

FIG. 4 shows an AR device 10 which is an embodiment of the present invention. The present invention relates to an image display device that provides an AR experience to a user, and will hereinafter be referred to as an AR device. The AR device 10 includes a central processing unit (CPU, central processing unit) 11, an operation unit 12, an image display unit 13, a memory 14, an imaging unit 15, an acceleration sensor 16, and a network interface unit 17. And a bus 18 so that data can be exchanged between 11 to 17. The hardware configuration of such an AR device can employ a compatible computer or a successor of IBM's personal computer “PC / AT (Person C omp pu t e r / A d va nc dT e c h no o g o y y)”. Of course, you may employ | adopt the computer provided with the other architecture. Usually, an operating system is stored in the memory 14, and a program for performing the processing of the embodiment of the present invention (hereinafter referred to as AR operation) is executed on this operating system. . A program for performing the AR operation is also stored in the memory 14 in advance. The CPU 11 can execute various arithmetic processes. From the operation unit 12, the user can give various commands to the AR device 10. The operation unit 12 may be integrated with the image display device 13 and may be input via a touch panel. Images and information can be displayed on the image display unit 13. In addition to the program that performs the AR operation, the memory 14 downloads data calculated by the CPU 11, an image captured by the imaging unit 15, acceleration information from the acceleration sensor 16, and data on the network via the network interface unit 17. Can be stored. The imaging unit 15 includes, for example, a lens and an image sensor, and can capture an outside world and generate an image. The AR device 10 can be connected to a network such as the Internet via the network interface unit 17 and can access various information on the Internet. The acceleration sensor 16 is a triaxial sensor. That is, it is possible to detect acceleration in the x-axis, y-axis, and z-axis directions. In general, the acceleration sensor is affected by gravity, but since the influence of gravity can be canceled by subtracting from the value of the acceleration sensor, the following description will be made assuming that there is no influence of gravity.

A specific use procedure using the AR device 10 will be described.

The user first prepares an object (paper surface) on which an AR operation is desired. A two-dimensional bar code (QR code) is drawn at the corner of the page. FIG. 5 shows an example of a paper surface. Reference numeral 20 in FIG. 5 denotes a paper surface, and a two-dimensional barcode 21 is written at a corner. It is assumed that a photograph of the car is posted on the paper 20 as shown in the figure.

The user instructs the AR device 10 to the AR device 10 via the operation unit 12 to perform preprocessing of the AR operation, and directs the AR device 10 toward the QR code 21. Upon receiving this command, the AR device 10 performs imaging using the imaging unit 15. Then, the QR code 21 in the photographed image is recognized, and the image data 30 with information on the network designated by the QR code 21 is downloaded. The image data 30 with information is the same size as the paper surface 20 and is in the form of “the same image 31 as the paper surface 20 and the information 32 is superimposed on the image”. An example of the image data 30 with information, and the image 31 and information 32 constituting the image data 30 is shown in FIG. In this embodiment, the image data with information 30 will be described as a still image, but may be a moving image such as an animation GIF.

Next, the user holds the AR device 10 at a position separated from the paper surface 20 by several tens of centimeters. Then, the AR device 10 is commanded to the AR device 10 via the operation unit 12 to start the AR operation. Then, the AR device 10 is brought close to the paper surface 20 and finally the AR device 10 is grounded to the paper surface 20.

When receiving the command to start the AR operation, the AR device 10 performs continuous shooting by the shooting unit 15. Since it is approaching the paper surface 20, the captured image will not be in focus after a while. Stop continuous shooting when the subject is out of focus. Thereafter, no shooting is performed during the AR operation.

Further, the AR device 10 continues to detect the output value of the acceleration sensor 16 and the time when the output value becomes discontinuous (the time when the AR device 10 receives an impact, that is, the AR device 10 is grounded to the paper surface 20. Time). The positional relationship between the AR device 10 and the paper surface 20 at the time of discontinuity is obtained by analyzing an image that has been photographed before it is out of focus. An area corresponding to the obtained positional relationship between the AR device 10 and the paper surface 20 is cut out from the image data 30 with information downloaded in the preprocessing and displayed on the image display unit 13. As a result, the image display unit 13 in the AR device 10 placed on the paper surface 20 can cause the user to have the illusion that the paper surface 20 is directly viewed through the transparent glass. I feel a sense of unity. And it feels as if the information 32 (strictly, a part of the information 32) is written on the transparent glass portion (the situation of the AR device 10 (before movement) in FIG. 14, details will be described later).

After the AR device 10 is grounded on the paper surface 20, the user can move the AR device 10 along the paper surface 20. The AR device 10 detects the amount of movement by the acceleration sensor 16, changes the cutout region by the same amount as the amount of movement, and performs display on the image display unit 13. Thereby, even after the movement, the image display unit 13 in the AR device 10 placed on the paper surface 20 can cause the user to have the illusion that the paper surface 20 is directly viewed through the transparent glass. Therefore, the user feels a sense of unity. Then, it feels as if the information 32 (strictly, a part of the information 32) is written on the transparent glass portion (the situation of the AR device 10 (after movement) in FIG. 14, details will be described later).

Finally, the user moves the AR device 10 away from the paper surface 20. The AR device 10 continues to detect the output value of the acceleration sensor 16 and detects the acceleration in the direction of the image display unit 13 (that is, detects the time when the AR device 10 is separated from the paper surface 20). When the acceleration is detected, the display on the image display unit 13 is terminated and the AR operation is also terminated.

The above is a specific usage procedure and a simplified description of the operation of the AR device 10 according to the procedure. In this way, it is possible to receive the advantages of (Advantage 1) to (Advantage 6).

The process flow will be described in detail below.

In the following, in the coordinate system related to the image, the center of the image is the origin, the horizontal direction is the x axis (left side is positive), and the vertical direction is the y axis (lower side is positive). For convenience of explanation, with respect to the image display unit 13, the axis in the vertical direction is further referred to as the z-axis. The direction from the image display unit 13 toward the back side of the image display unit 13 is the positive z-axis direction. Further, it is assumed that an imaging unit 15 is installed on the back side of the image display unit 13. Assume that the center of the image display unit 13 and the center of the imaging unit 15 are the same. Exactly, although the positions of the center of the image display unit 13 and the center of the imaging unit 15 are different in the z-axis direction by the thickness of the AR device, the thickness is usually thin and is ignored. The size of the image display unit 13 is horizontal WD and vertical HD. This is shown in FIG.

If the center of the image display device 13 and the center of the imaging unit 15 are different, an offset may be added for the calculation of the position described later, so that the center of the image display unit 13 and the center of the imaging unit 15 are added. The generality is not lost even if it is described as the same position.

FIG. 8 shows a processing flow of the embodiment of the present invention.

In step 1 of FIG. 8, an instruction for preprocessing from the user is awaited. That is, the process continues to wait in step 1 until the user gives an instruction to perform preprocessing of the AR operation via the operation unit 12. When an instruction to perform preprocessing is received from the user, the process proceeds to step 2.

In the stage of proceeding to step 2, the user points the AR device 10 toward the QR code 21. In step 2, shooting is performed by the imaging unit 15, and the captured image is sent to the CPU 11 via the bus 18. The CPU 11 detects and analyzes the QR code 21 in this image. The detection and analysis of the QR code is a known method, and the description thereof is omitted. In the QR code 21, the location of the image data with information 30 on the network is recorded. From the analysis result of the QR code 21, the image data 30 with information on the network designated by the QR code 21 is downloaded. The download is performed via the network interface unit 17, and the downloaded image data with information 30 is stored in the memory 14 via the bus 18. Note that the image data with information 30 prepared in advance on the network has the same size as the paper surface 20 and has a format of “the same image 31 as the paper surface 20 and information 32 superimposed on the image”. It is data.

Here, it is assumed that a two-dimensional function P (x, y) is introduced to represent the paper surface 20. That is, an image in which a value of P (x, y) is set at an arbitrary position (x, y) is the paper surface 20 (see FIG. 5). Here, x = −W / 2 to W / 2, and y = −H / 2 to H / 2. The size of the paper surface 20 is assumed to be horizontal W and vertical H. The unit of x and y is, for example, millimeter.

Since the image 31 is the same as the paper surface 20, a value of P (x, y) is set at an arbitrary position (x, y) of the image 31. The image data with information 30 is assumed to be Q (x, y). x = -W / 2 to W / 2, and y = -H / 2 to H / 2. Q (x, y) is data obtained by superimposing information 32 on P (x, y). Therefore, Q (x, y) = P (x, y) at the position (x, y) where the information 32 does not exist (see FIG. 6).

After step 2, proceed to step 3.

In step 3, an AR operation start instruction from the user is awaited. That is, the process continues to wait in step 3 until the user gives an instruction to start the AR operation via the operation unit 12. When an instruction to start the AR operation is received from the user, the process proceeds to Step 4.

In step 4, 0 is set to a numerical parameter t indicating time. Then, the process proceeds to Step 5.

In the stage of proceeding to Step 5, the user holds the AR device 10 at a position separated from the paper surface 20 by several tens of centimeters. Then, in the AR device 10, the processes after step 5 are performed. While these processes are being performed, the user brings the AR device 10 closer to the paper surface 20, and finally the paper surface 20. An operation of grounding the AR device 10 is performed. In step 5, shooting is performed by the imaging unit 15, and the captured image is sent to the CPU 11 via the bus 18. Since this image is an image taken at time t, it is assumed that I (t). Then, the process proceeds to Step 6.

In step 6, the CPU 11 determines whether the image I (t) is an in-focus image. Specifically, the high frequency component of the image I (t) is examined, and if it is greater than or equal to a predetermined threshold, it is determined that the image is in focus. If it is less than the threshold value, it is determined that the image is out of focus (an out-of-focus image). If the image is in focus, the process proceeds to step 7. Otherwise, go to Step 10 (FIG. 9).

In step 7, only the image 31 (that is, P (x, y)) of the image data with information 30 in the memory 14 is sent to the CPU 11 via the bus 18. Then, the CPU 11 performs matching between the image I (t) and the image 31 to obtain the positional relationship (projection transformation relationship) between the image I (t) and the image 31. That is, a 3 × 3 matrix H3 that satisfies Equation 1 is obtained.

Here, I (t, x, y) is the value of the position (x, y) of the image I (t). That is, a process for obtaining H3 to be the image 31 by transforming the image I (t) with H3. This 3 × 3 matrix H3 is called a homography matrix (projection transformation matrix). As a method for matching two images (calculation of positional relationship), a known method such as a matching method using feature points may be used, and a detailed description thereof will be omitted. After step 7, proceed to step 8.

In step 8, the CPU 11 determines what positional relationship the center (x, y) = (0, 0) of the photographed image I (t) is with respect to the image 31 (that is, the paper surface 20). Ask. Specifically, (X (t), Y (t), dX (t), dY (t)) is obtained. Specifically, (X (t), Y (t), dX (t), dY (t)) is calculated by Equation 2 and Equation 3. Here, H3 is the homography matrix (projection transformation matrix) obtained in the immediately preceding step 7.

Thereby, the center of the image I (t) photographed at time t (that is, the center of the imaging unit 15) captures the position of (X (t), Y (t)) of the image 31 (that is, the paper surface 20). It can be said that he was doing. Then, it can be seen that the positive direction of the x-axis of the captured image I (t) is the direction of (dX (t), dY (t)) of the image 31 (that is, the paper surface 20). To be precise, it can be seen that a vector having a unit length in the positive direction of the x-axis becomes a vector (dX (t), dY (t)) on the image 31 (that is, the paper surface 20).

(X (t), Y (t), dX (t), dY (t)) obtained by the CPU 11 is stored in the memory 14 via the bus 18. This completes the processing in step 8, and then proceeds to step 9.

In step 9, it waits for unit time (for example, 1/30 second). Since the unit time has been advanced, t is incremented by one. After step 9, the process returns to step 5 to perform the process for the updated t.

At the stage of proceeding to step 10, the AR device 10 is too close to the paper surface 20 by the user, and the imaging unit 15 cannot be focused. In step 10, t1 = t is set in order to store the time at which focusing could not be performed. Then, the process proceeds to Step 11.

In step 11, it is detected whether the value of the acceleration sensor 16 in the z-axis direction becomes discontinuous. If it is not discontinuous, the process proceeds to step 12. If discontinuous, the process proceeds to step 13.

In step 12, it waits for unit time (for example, 1/30 second). Since the unit time has been advanced, t is incremented by one. Then, the process returns to step 11.

Proceeding to step 13 means that the value of the acceleration sensor 16 in the z-axis direction has detected a discontinuity. That is, it means that the AR device 10 has contacted (grounded) the paper surface 20. In step 13, t2 = t in order to store the time when the AR device 10 contacts (grounds) the paper surface 20. Then, the process proceeds to step 14.

In step 14, the positional relationship between the image display unit 13 and the paper surface 20 at the time of contact (grounding) is extrapolated. This means that (X (t), Y (t), dX (t), dY (t)) (where t is an integer from 0 to t1-1) stored in the memory 14 is transferred to the bus 18. Then, the CPU 11 extrapolates using these values (X (t), Y (t), dX (t), dY (t)). Here, extrapolation means the value (X (t2)) at time t2 using the values of (X (t), Y (t), dX (t), dY (t)) from 0 to t1-1. , Y (t2), dX (t2), dY (t2)). Various methods exist for extrapolation. For example, extrapolation may be performed using a linear regression model. That is, A1 and B1 that minimize Equation 4 are obtained. Then, X (t2) = A1 × t2 + B1 may be satisfied. This is shown in FIG. Y (t2), dX (t2), and dY (t2) may be obtained similarly.

After step 14, proceed to step 15.

In step 15, the position of the AR device 10 is divided into cases, and coordinate conversion is performed so as to be in an appropriate direction. In the embodiment of the present invention, a smartphone or a tablet PC is assumed as the AR device 10. The user may use the smartphone or tablet PC in the horizontal direction or may use it in the vertical direction. Therefore, it is necessary to detect the direction in which the AR device 10 was in contact with the paper surface 20. FIG. 12 shows all cases.
FIG. 12A shows a case where the axes of the paper surface 20 and the AR device 10 (that is, the image display unit 13) are in the same direction.
FIG. 12A shows a case where the AR device 10 (that is, the image display unit 13) is rotated 90 degrees with respect to the paper surface 20. That is, the case where the positive direction of the x-axis of the paper surface 20 is the same as the positive direction of the y-axis of the image display unit 13 is shown.
FIG. 12C shows a case where the AR device 10 (that is, the image display unit 13) is rotated by −90 degrees with respect to the paper surface 20. That is, the case where the positive direction of the x-axis of the paper surface 20 is the same as the negative direction of the y-axis of the image display unit 13 is shown.
Note that 40 described in FIG. 12 is “paper surface 20 (or image data with information 30)”. Reference numeral 41 denotes “the coordinate origin of the paper surface 20 (or image data with information 30)”. Reference numeral 42 denotes “the x axis of the paper surface 20 (or the image data 30 with information 30)”. 43 is the “y-axis of the paper surface 20 (or image data 30 with information 30)”. Reference numeral 44 denotes an “image display unit 13”. Reference numeral 45 denotes “the coordinate origin of the image display unit 13”. Reference numeral 46 denotes “the x-axis of the image display unit 13”. 47 is the “y-axis of the image display unit 13”. The vectors (X (t2), Y (t2)) written in FIG. 12 are vectors in the coordinate system of the paper surface 20 (or image data 30 with information).

Note that the fact that it is rotated 180 degrees means that the AR device 10 is used with the top and bottom reversed, and since this is an abnormal usage method, the AR processing is forcibly terminated. Further, since the use of the AR device 10 at an angle is also an abnormal usage method, the AR process is forcibly terminated.

As described above, since the positive direction of the x axis of the image I (t) taken at time t is the direction of (dX (t), dY (t)) on the paper surface 20, the extrapolated value. (DX (t2), dY (t2)), (a), (b), (c), and “other” shown in FIG. 12 can be classified. In other words, in step 15, the CPU 11 performs the following processing using (dX (t2), dY (t2)) obtained in step 14.
Case 1: “Absolute value of dY (t2)” <“Absolute value of dX (t2)”, 0 <dX (t2), and “Absolute value of dY (t2)” ÷ dX (t2) <Δ Then, since it is determined as (a) in FIG. 12, since the directions of the axes of the paper surface 20 and the image display unit 13 match, no processing is performed.
Case 2: “Absolute value of dX (t2)” <“Absolute value of dY (t2)”, 0> dY (t2), and “Absolute value of dX (t2)” ÷ (−dY (t2) ) <Δ, it is determined as (A) in FIG. 12, and the image data with information 30 stored in the memory 14 is rotated by 90 degrees and overwritten. Further, since the positional relationship between the image display unit 13 and the paper surface 20 at the time of contact needs to be rotated by 90 degrees, (X (t2), Y (t2)) = (− Y (t2), X (t2)) And
Case 3: “absolute value of dX (t2)” <“absolute value of dY (t2)”, 0 <dY (t2), and “absolute value of dX (t2)” ÷ dY (t2) <Δ Then, since it is determined as (c) in FIG. 12, the image data with information 30 stored in the memory 14 is rotated by −90 degrees and overwritten. Further, since the positional relationship between the image display unit 13 and the paper surface 20 at the time of contact needs to be rotated by −90 degrees, (X (t2), Y (t2)) = (Y (t2), −X (t2) ).
Case 4: If none of the above cases 1, 2, and 3 is met, the message “The device is tilted too much with respect to the paper surface” is displayed on the image display unit 13 and forced. finish.
Here, Δ is a minute value and is a threshold value for continuing the AR operation without causing an error even if the AR device 10 is slightly inclined.

FIGS. 13A, 13A, and 13C show diagrams for the cases 1, 2, and 3, respectively. In this figure, the axis after rotation (the axis after updating in step 15) is shown. Note that 50 described in FIG. 13 is “image data 30 with information”. Reference numeral 51 denotes “the coordinate origin of the image data with information 30”. Reference numeral 52 denotes “x-axis updated by image rotation in step 15 of the image data with information 30”. 53 is “y-axis updated by image rotation in step 15 of image data with information 30”. Reference numerals 44 to 47 in FIG. 13 are the same as those shown in FIG. “Updated by step 15 (X (t2), Y (t2))” written in FIG. 13 is the coordinate system updated by step 15 of the image data with information 30 (updated x-axis, y-axis).

After step 15, except for case 4, the process proceeds to step 16 (FIG. 10).

In step 16, a new parameter s representing time is set to 0. Further, values (X (t2), Y (t2)) are substituted for values (Xoffset, Yoffset) indicating the positional relationship of the image display unit 13 with respect to the paper surface 20 at the first time, and these values are stored in the memory 14. Keep it. Further, this value is also stored in the memory 14 as (XX (0), YY (0)) = (Xoffset, Yoffset). Then, the process proceeds to Step 17.

In step 17, an appropriate image at the current time s is displayed on the image display unit 13. That is, the image data with information 30 and (XX (s), YY (s)) are sent from the memory 14 via the bus 18 to the CPU 11. In the CPU 11, the (WD, HD) area centered on (XX (s), YY (s)) is cut out from the image data with information 30 and sent to the image display unit 13. The image display unit 13 displays this image. As a result, the image display unit 13 in the AR device 10 placed on the paper surface 20 can cause the user to have the illusion that the paper surface 20 is directly viewed through the transparent glass. I feel a sense of unity. And it feels as if the information 32 (strictly, a part of the information 32) is written on the transparent glass portion. This is because the center of the image display unit 13 is located at (XX (s), YY (s)) with respect to the paper surface 20, and from the image data 30 with information of the same size as the paper surface 20, This is because “the size of the image display unit 13 (WD, HD)” centering on XX (s), YY (s)) is cut out and displayed. After step 17, proceed to step 18.

In step 18, it waits for unit time (for example, 1/30 second). Since the unit time has advanced, s is incremented by one. Then, the x-axis value (referred to as ax (s)) and the y-axis value (referred to as ay (s)) of the acceleration sensor 16 at the current time are stored in the memory 14 via the bus 18. Then, the process proceeds to Step 19.

In step 19, the position information is updated. That is, the values of ax (s) and ay (s) and (Xoffset, Yoffset) at all times from s = 1 to the present stored in the memory are sent to the CPU 11 via the bus 18. In the CPU 11, the time from the first time (s = 1) to the present time is integrated twice with ax (s), and a value obtained by adding Xoffset is further calculated as XX (s). The time from the first time (s = 1) to the present time is integrated twice for ay (s), and a value obtained by adding Yoffset is calculated to be YY (s). The position change can be obtained by integrating the value of the acceleration sensor twice. The values of (XX (s), YY (s)) are stored in the memory 14 via the bus 18. After step 19, proceed to step 20.

In step 20, the position is finely adjusted by the user. The integration in step 19 may accumulate errors over time, and the positional displacement of the image display unit 13 with respect to the paper surface 20 may become significant. Therefore, the user can explicitly make corrections. Specifically, the user inputs a value of (Xoffset, Yoffset) from the operation unit 12. (Xoffset, Yoffset) input from the operation unit 12 overwrites the value of (Xoffset, Yoffset) stored in the memory 14 via the bus 18. By this update, an appropriate area whose position is corrected can be displayed at the next display. After step 20, the process proceeds to step 21.

In step 21, the end of the AR operation is detected. That is, it is determined whether the z-axis value of the acceleration sensor 16 is a negative value. A negative value means that the AR device 10 is separated from the paper surface 20. If it is not negative, it can be seen that the AR device continues to be grounded on the paper surface 20. Therefore, in step 21, if the value of the z-axis of the acceleration sensor 16 is a negative value, the process proceeds to step 22. Otherwise, return to step 17 to continue the AR operation.

In step 22, the display on the image display unit 13 is terminated and the AR operation is terminated.

This is the end of the description of the embodiment of the present invention.

In the present invention, as shown in FIG. 14, the AR device 10 is used while being grounded with respect to the paper surface 20. Therefore, regardless of the position of the user's eyes, the image display unit 13 displays an appropriate area in the paper surface 20 in the same size as the paper surface 20, so that the actual landscape (the paper surface 20) is displayed. It is possible to display an image with a bodily sensation. In other words, the image display unit can appear as if it is transparent glass. Even if the user moves the AR device 10, the cut area from the image data with information 30 is changed and displayed by the same amount as the moved amount, so that the image display unit is made of transparent glass. You can keep it looking as if it were.

In the above-described embodiment of the present invention, when the user moves the AR device 10 on the paper surface 20, only the parallel movement is permitted, and it is not assumed that the AR device 10 is rotated. If rotation is also permitted, a gyro sensor may be provided in the AR device 10 to detect the inclination of the AR device 10 with respect to the paper surface 20. Then, a region to be cut out from the image data with information 30 is determined based on the values of (XX (s), YY (s)) and the inclination detected by the gyro sensor, and the region portion is displayed as an image. What is necessary is just to display in the part 13.

In the embodiment of the present invention described above, the image data with information 30 is downloaded as a QR code. However, the image data with information 30 may be acquired by the following method. That is, the paper surface 20 is photographed and an object drawn in the paper surface 20 is recognized. Then, information on the recognized object may be searched on the Internet, and the searched image data 30 may be created by adding the searched information.

As mentioned above, although embodiment of this invention was explained in full detail, the concrete structure is not restricted to this embodiment, The design change etc. of the range which does not deviate from the summary of this invention are included.

Furthermore, this technique can also be set as the following structures.

[1]
An image display method for displaying only a specific area from an image on an image display device,
An image data storage step for storing the image in a memory;
A position measuring step for measuring the amount of movement of the image display device;
An area for changing the specific area so that an area moved by the same amount as the movement amount measured in the position measurement step with respect to the specific area at the immediately preceding time is set as the specific area at the current time. An update step;
A display step of causing the image display device to display only the specific area of the image.
[2]
A process start step for performing a process of starting display on the image display device when an impact is detected on the image display device;
Or
A process end step for performing a process of ending the display on the image display device when movement of the image display device toward the display surface is detected;
The image display method according to [1], further including at least one of the steps.
[3]
Before the time to start displaying on the image display device,
The method according to [1] or [2], further including an initial position calculating step of analyzing an image captured by an imaging unit installed in the image display device and calculating a position at a time when display on the image display device is started. Image display method.
[4]
[1] to [3], wherein the specific area is updated in the area update step under a condition that the specific area is the same size as a display unit of the image display device. Image display method.
[5]
The image of the specific area displayed on the image display device in the display step is rotated by 90 degrees or -90 degrees and displayed in accordance with the inclination of the image display device at the time of starting the display on the image display device. The image display method according to any one of [1] to [4].
[6]
An image display device that displays only a specific area from an image,
A memory unit for storing the image;
A position measuring unit for measuring the movement amount of the image display device;
An area for changing the specific area so that an area moved by the same amount as the movement amount measured by the position measurement unit with respect to the specific area at the immediately preceding time is set as the specific area at the current time. An update section;
An image display device comprising: a display control unit that causes the image display device to display only the specific region of the image.
[7]
An image display program for displaying only a specific area from an image on an image display device,
An image data storage step for storing the image in a memory;
A position measuring step for measuring the amount of movement of the image display device;
An area for changing the specific area so that an area moved by the same amount as the movement amount measured in the position measurement step with respect to the specific area at the immediately preceding time is set as the specific area at the current time. An update step;
An image display program for causing a computer to execute a process including a display step of causing the image display device to display only the specific area of the image.

10. Image display device (AR device)
DESCRIPTION OF SYMBOLS 11 ... Central processing part 12 ... Operation part 13 ... Image display part 14 ... Memory 15 ... Imaging part 16 ... Acceleration sensor 17 ... Network interface part 18 ... Bus 20 ... Paper surface 21 ... Two-dimensional barcode 30 ... Image data 31 with information ... Image 32 ... Information 100 ... Conventional image display device (conventional AR device)
110 ... Image display unit 120 ... Imaging unit 200 ... Paper surface 300 ... Image 310 ... Information 400 ... User's eyes

Claims (7)

An image display method for displaying only a specific area from an image on an image display device,
An image data storage step for storing the image in a memory;
A position measuring step for measuring the amount of movement of the image display device;
An area for changing the specific area so that an area moved by the same amount as the movement amount measured in the position measurement step with respect to the specific area at the immediately preceding time is set as the specific area at the current time. An update step;
A display step of causing the image display device to display only the specific area of the image. A process start step for performing a process of starting display on the image display device when an impact is detected on the image display device;
Or
A process end step for performing a process of ending the display on the image display device when movement of the image display device toward the display surface is detected;
The image display method according to claim 1, further comprising at least one step of: Before the time to start displaying on the image display device,
3. The initial position calculating step according to claim 1, further comprising an initial position calculating step of analyzing an image captured by an imaging means installed in the image display device and calculating a position at a time when display on the image display device is started. Image display method. 3. The specific area is updated in the area update step with a condition that the specific area is the same size as a display unit of the image display device. Item 4. The image display method according to Item 3. The image of the specific area displayed on the image display device in the display step is rotated by 90 degrees or -90 degrees and displayed in accordance with the inclination of the image display device at the time of starting the display on the image display device. The image display method according to claim 1, 2, 3, or 4. An image display device that displays only a specific area from an image,
A memory unit for storing the image;
A position measuring unit for measuring the movement amount of the image display device;
An area for changing the specific area so that an area moved by the same amount as the movement amount measured by the position measurement unit with respect to the specific area at the immediately preceding time is set as the specific area at the current time. An update section;
An image display device comprising: a display control unit that causes the image display device to display only the specific region of the image. An image display program for displaying only a specific area from an image on an image display device,
An image data storage step for storing the image in a memory;
A position measuring step for measuring the amount of movement of the image display device;
An area for changing the specific area so that an area moved by the same amount as the movement amount measured in the position measurement step with respect to the specific area at the immediately preceding time is set as the specific area at the current time. An update step;
The image display program which makes a computer perform the process including the display step which displays only the said specific area | region among the said images on the said image display apparatus.

Images