JP2016023962A - Method for measuring subject size, device for measuring subject size, program for measuring subject size, image display method, image display device and image display program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform image display which produces a sense of unity between an actual scenery viewed by a user and a display on an image display device.SOLUTION: An image display device (AR device) 10 is used by being brought in contact with respect to a paper surface 20. An image display part 13 displays an appropriate region in the paper surface 20 in the same size as the paper surface 20 regardless of the position of the user's eyes, and therefore, can perform image display with a sense of unity with respect to an actual scenery (paper surface 20). In other words, the image display part 13 can display while acting as if the image display part 13 is transparent glass. The region displayed by the image display part 13 can be made appropriate by utilizing at least one piece of information among an imaging region of an image I which imaged the paper surface 20, a contact position P of an imaging part 15 to the paper surface 20, and a size (WD and HD) of the image display part 13.SELECTED DRAWING: Figure 13

Description

The present technology relates to a method, an apparatus, and a program for measuring a size of an object to be imaged and measuring a size of the object to be imaged, and a method, an apparatus, and an apparatus for displaying an image edited using a captured image as a material. And related to the program.

In recent years, applications using augmented reality (AR) have been realized. For example, when a target object such as paper is photographed using a smartphone or tablet PC, the application recognizes the target object and displays an appropriate image. More specifically, it is an application called “Tokyo AR” provided by the Tokyo head office of Chunichi Shimbun. Also, FIG. 1 of Non-Patent Document 1 describes an example of a similar application. Further, Patent Document 1 discloses a similar application.

There is also a technology for automatically recognizing a face from a single image with a face and creating a moving moving mouth and eyes. More specifically, this is a technology developed by Motion Portrait Co., Ltd. By using this technology, the following applications using AR can be considered.

1 is a conventional AR device (specifically, a smartphone or a tablet PC). The AR device 100 includes an image display unit 110 and an imaging unit 120. The imaging unit 120 is installed on the opposite side of the image display unit 110 and is not visible in the figure. For example, the paper surface 20 is photographed by the imaging unit 120. It is assumed that a person 30 is drawn on the paper surface 20. Here, the paper surface is, for example, a specific page of a magazine, and a photograph of a person is printed on the specific page. When the user uses the imaging unit 120 to photograph the paper surface 20, the AR device 100 analyzes the image and detects a human face portion. Further, the face image is deformed to create a moving image of moving eyes and mouth. Then, the created moving image is displayed on the image display unit 110. The image display unit 110 in FIG. 1 illustrates a situation where such a moving image is being reproduced.

Reference numeral 40 in FIG. 1 indicates the eyes of the user. FIG. 1 is a scene in which the user is looking at the image display unit 110 immediately after photographing the paper surface 20.

A view seen from the user's eyes 40 is shown in FIG. As shown in FIG. 2, the AR device 100 is visible to the user. On the image display unit 110 in the AR device 100, a captured image (more precisely, a moving image in which a face portion of the captured image is deformed) can be seen. In addition, a real world landscape can be seen outside the AR device 100. That is, the paper surface 20 can be seen. Furthermore, a part of the person 30 drawn on the paper surface 20 can also be seen. In FIG. 2, the paper surface 20 and the image displayed in the image display unit 110 have no sense of unity. This is because the size of the target person is different. That is, the size of the person in the image display unit 110 is different from the size of the person 30 drawn on the real world paper 20.

As described above, the AR device 100 visible from the user's eyes 40 cannot display an image with a sense of unity with respect to an actual landscape.

JP 2012-195732 A

Schmalstieg, Dieter, and Daniel Wagner. "Mobile phones as a platform for augmented reality." Connections 1 (2009): 3.

As described above, augmented reality (AR) can be realized by using the techniques described in Patent Document 1 and Non-Patent Document 1, but there is no sense of unity regarding display. In other words, the AR device 100 that can be seen from the user's eyes 40 cannot display an image with a sense of unity with respect to an actual landscape.

The present technology has been made in view of such circumstances, and is intended to display an image with a sense of unity with respect to an actual landscape.

The method for measuring the size of the subject according to one aspect of the present technology is as follows. A photographing step of photographing a target object and obtaining a photographed image with an imaging device placed at the photographing position, and moving the imaging device between the photographing position and the target object, A measurement step for measuring the distance to the target object; and a calculation step for calculating the size of the target object based on the viewing angle of the imaging device and the distance. Thereby, the size of the subject can be measured.

A method for displaying an image of one aspect of the present technology is as follows. In a method of displaying an image using a device in which an image pickup unit and an image display unit are integrated, shooting is performed by shooting a target object with the image pickup unit of the device placed at a shooting position. A step of cutting out the captured image or a part of an image using the captured image as a material as necessary, and the image cut out in the clipping step when the device is grounded to the target object. A display step of displaying on the image display unit, and in the clipping step, at least the size of the target object, the position when the device is grounded to the target object, and the size of the image display unit A region to be cut out is determined using one piece of information. Thereby, it is possible to display an image with a sense of unity.

In the method of displaying the image, in order to determine the size of the target object, the apparatus is further moved between the shooting position and the target object, so that the shooting position and the target object are moved. A measurement step for measuring the distance to the object, and a calculation step for calculating the size of the target object based on the viewing angle of the imaging unit and the distance are provided. Thereby, it is possible to display an image with a sense of unity.

An apparatus for measuring the size of a subject according to one aspect of the present technology is as follows. The image capturing unit placed at the image capturing position captures the target object to obtain a captured image, and the image capturing unit is moved between the image capturing position and the target object, thereby capturing the image. A measurement processing unit that measures a position and a distance to the target object; and a calculation processing unit that calculates a size of the target object based on a viewing angle of the imaging unit and the distance. Thereby, the size of the subject can be measured.

In addition, an apparatus that displays an image of one aspect of the present technology is as follows. In an apparatus in which an imaging unit and an image display unit are integrated, a captured image acquisition unit that captures a target object by the imaging unit of the apparatus placed at a shooting position and obtains a captured image, and the captured image or necessary And a cutout processing unit that cuts out a part of the image using the captured image as a material, and the image cut out by the cutout processing unit is displayed on the image display unit when the device is grounded to the target object. An image display control unit that performs processing to perform at least one of the size of the target object, the position when the device is grounded to the target object, and the size of the image display unit. A region to be cut out is determined using one piece of information. Thereby, it is possible to display an image with a sense of unity.

In the device for displaying the image, in order to determine the size of the target object, the imaging unit is further moved between the shooting position and the target object, so that the shooting position and the target object are moved. A measurement processing unit that measures the distance to the target object, and a calculation processing unit that calculates the size of the target object based on the viewing angle of the imaging unit and the distance are provided. Thereby, it is possible to display an image with a sense of unity.

A program for measuring the size of a subject according to one aspect of the present technology is as follows. A photographing step of photographing a target object and obtaining a photographed image with an imaging device placed at the photographing position, and moving the imaging device between the photographing position and the target object, A measurement step for measuring the distance to the target object; and a calculation step for calculating the size of the target object based on the viewing angle of the imaging device and the distance. Thereby, the size of the subject can be measured.

A program for displaying an image of one aspect of the present technology is as follows. In a program for displaying an image using a device in which an imaging unit and an image display unit are integrated, the imaging unit of the device placed at a shooting position captures a target object and obtains a captured image. A step of cutting out the captured image or a part of an image using the captured image as a material as necessary, and the image cut out in the clipping step when the device is grounded to the target object. A display step of displaying on the image display unit, and in the clipping step, at least the size of the target object, the position when the device is grounded to the target object, and the size of the image display unit A region to be cut out is determined using one piece of information. Thereby, it is possible to display an image with a sense of unity.

In the program for displaying the image, in order to determine the size of the target object, the apparatus is further moved between the shooting position and the target object, so that the shooting position and the target are moved. A measurement step for measuring the distance to the object, and a calculation step for calculating the size of the target object based on the viewing angle of the imaging unit and the distance are provided. Thereby, it is possible to display an image with a sense of unity.

One aspect of the present technology is that an imaging device placed at a shooting position captures a target object to obtain a captured image, and the imaging device is moved between the shooting position and the target object. And measuring the distance to the shooting position and the target object, and calculating the size of the target object based on the viewing angle of the imaging device and the distance. Thereby, the size of the subject can be measured.

In addition, according to one aspect of the present technology, when an image is displayed using an apparatus in which an imaging unit and an image display unit are integrated, the target object is captured by the imaging unit of the apparatus placed at a capturing position. Then, obtaining the captured image, cutting out a part of the image using the captured image or, if necessary, the captured image as a material, and the apparatus is grounded to the target object. In the cutting process, the size of the target object, the position when the device is grounded to the target object, and the size of the image display unit are displayed. A region to be cut out is determined using at least one piece of information. Thereby, it is possible to display an image with a sense of unity.

Then, in order to determine the size of the target object, the apparatus is further moved between the shooting position and the target object, thereby measuring the distance between the shooting position and the target object. And calculating the size of the target object based on the viewing angle of the imaging unit and the distance. Thereby, it is possible to display an image with a sense of unity.

According to one aspect of the present technology, the size of a subject can be measured. Alternatively, an image with a sense of unity can be displayed.

It is a figure explaining the usage example of the conventional apparatus. It is a figure for demonstrating the display state without the sense of unity by the conventional apparatus. It is a figure for demonstrating the apparatus of the Example of this invention. It is a figure for demonstrating the positional relationship of each site | part installed in the apparatus of the Example of this invention. It is a figure explaining the usage example of the apparatus of the Example of this invention. It is a flowchart explaining the process with the apparatus of the Example of this invention. It is a flowchart explaining the process with the apparatus of the Example of this invention. It is explanatory drawing about the image image | photographed in the Example of this invention. It is explanatory drawing about the image image | photographed in the Example of this invention. It is explanatory drawing regarding the position estimation performed in the Example of this invention. It is a figure for demonstrating the positional relationship of the paper surface and apparatus in the Example of this invention. It is explanatory drawing about the image displayed in the Example of this invention. It is a figure for demonstrating the display state with a sense of unity by the apparatus of the Example of this invention.

The advantages of the present invention are first described. The embodiment of the present invention relates to an image display device that provides an AR experience to a user, and will be referred to as an AR device hereinafter.
(Advantage 1) Since the AR device is operated while being grounded on the paper surface, only an appropriate part of the image is displayed regardless of the position of the user's eyes, thereby displaying an image with a sense of unity with respect to the actual landscape. Can be made. In other words, the image display unit can appear as if it is transparent glass.
In addition, there are the following advantages.
(Advantage 2) Since the contact with the paper surface is automatically detected by detecting the impact on the AR device, the user does not need to explicitly give an instruction to start the operation of the AR device (that is, display start). .
(Advantage 3) When the user moves the AR device away from the paper surface, this operation is detected by the acceleration sensor in the AR device and the AR operation is terminated, so that the user explicitly ends the operation of the AR device ( That is, it is not necessary to instruct the display end).

FIG. 3 shows an AR device 10 which is an embodiment of the present invention. The AR device 10 includes a central processing unit (CPU, central processing unit) 11, an operation unit 12, an image display unit 13, a memory 14, an imaging unit 15, an acceleration sensor 16, and a network interface unit 17. And a bus 18 so that data can be exchanged between 11 to 17. The hardware configuration of such an AR device can employ a compatible computer or a successor of IBM's personal computer “PC / AT (Person C omp pu t e r / A d va nc dT e c h no o g o y y)”. Of course, you may employ | adopt the computer provided with the other architecture. Usually, an operating system is stored in the memory 14, and a program for performing the processing of the embodiment of the present invention (hereinafter referred to as AR operation) is executed on this operating system. . A program for performing the AR operation is also stored in the memory 14 in advance. If not stored in the memory 14, a program for performing the AR operation may be downloaded from the Internet via the network interface unit 17 described later. The CPU 11 can execute various arithmetic processes. From the operation unit 12, the user can give various commands to the AR device 10. The operation unit 12 may be integrated with the image display device 13 and may be input via a touch panel. Images and information can be displayed on the image display unit 13. In addition to the program that performs the AR operation, the memory 14 downloads data calculated by the CPU 11, an image captured by the imaging unit 15, acceleration information from the acceleration sensor 16, and data on the network via the network interface unit 17. Can be stored. The imaging unit 15 includes, for example, a lens and an image sensor, and can capture an outside world and generate an image. The AR device 10 can be connected to a network such as the Internet via the network interface unit 17 and can access various information on the Internet. The acceleration sensor 16 is usually a three-axis sensor, but in the embodiment of the present invention, only acceleration information in the z-axis direction is used. In general, the acceleration sensor is affected by gravity, but since the influence of gravity can be canceled by subtracting from the value of the acceleration sensor, the following description will be made assuming that there is no influence of gravity.

In the following, in the coordinate system related to the image, the center of the image is the origin, the horizontal direction is the x axis (right side is positive), and the vertical direction is the y axis (lower side is positive). For convenience of explanation, with respect to the image display unit 13, the axis in the vertical direction is further referred to as the z-axis. The direction from the image display unit 13 toward the back side of the image display unit 13 is the positive z-axis direction. Further, it is assumed that an imaging unit 15 is installed on the back side of the image display unit 13. Assume that the center of the image display unit 13 and the center of the imaging unit 15 are the same. Exactly, although the positions of the center of the image display unit 13 and the center of the imaging unit 15 are different in the z-axis direction by the thickness of the AR device, the thickness is usually thin and is ignored. The size of the image display unit 13 is horizontal WD and vertical HD. This is shown in FIG.

The imaging unit 15 can capture the side opposite to the image display unit 13. The imaging unit 15 performs shooting with an angle of view of Ah in the horizontal direction (x-axis direction) and an angle of view of Av in the vertical direction (y-axis direction). The angles of view Ah and Av are assumed to be known. The image to be shot is an image having horizontal W pixels and vertical H pixels.

If the center of the image display device 13 and the center of the imaging unit 15 are different, an offset may be added for the calculation of the position described later, so that the center of the image display unit 13 and the center of the imaging unit 15 are added. The generality is not lost even if it is described as the same position.

A specific use procedure using the AR device 10 will be described.

The user first prepares an object (paper surface) on which an AR operation is desired. It is assumed that a person is drawn on this paper. FIG. 5 shows an example of a paper surface. 5 in FIG. 5 is a paper surface, and 30 is a person drawn on the paper surface.

The user instructs the AR device 10 to perform an AR operation on the AR device 10 via the operation unit 12 and directs the AR device 10 toward the paper surface 20. Upon receiving this command, the AR device 10 starts shooting with the imaging unit 15. Then, the AR device 10 gives an instruction to the user so that the AR device 10 approaches the paper surface 20. The user who has received the instruction moves the AR device 10 in the direction indicated by the arrow 50 in FIG. 5, and finally causes the AR device 10 to contact the paper surface 20. When the AR device 10 is grounded, the AR device 10 automatically displays a moving image of the face of the person 30 on the image display unit 13. As will be described later, this moving image is an image having a sense of unity with the actual landscape. That is, the image display unit 13 can appear as if it is a transparent glass. Furthermore, when the user removes the AR device 10 from the paper surface, the AR operation is automatically terminated.

The above is a specific use procedure. In this way, it is possible to receive the advantages of (Advantage 1) to (Advantage 3).

The process flow will be described in detail below.

6 and 7 show the processing flow of the embodiment of the present invention.

The user instructs the AR device 10 via the operation unit 12 to perform an AR operation on the AR device 10. Thereby, the AR apparatus 10 first performs step S1 of FIG.

In step S <b> 1, shooting is performed by the imaging unit 15, and the captured image is stored as I in the memory 14 via the bus 18. If the image I is already stored in the memory 14, it is overwritten. Then, the process proceeds to step S2.

In step S2, the image I stored in the memory 14 is taken out and analyzed by the CPU 11. By this analysis, a face is detected from the image I. Note that the technique for recognizing a face from an image is a known technique, and a detailed description of the analysis process is omitted. Then, the process proceeds to step S3.

In step S3, it is determined whether a face has been detected in the immediately preceding step. If the face cannot be detected, the process returns to step S1 in order to take a picture again. If a face can be detected, the process proceeds to step S4 in order to proceed with the process. Note that it is assumed that the shooting in step S1 is performed by the user shooting toward the paper surface 20. Since the person 30 is drawn on the paper surface 20, a normal face can be detected in step S2. Therefore, it is not possible to return from step S3 to step S1. In the unlikely event that a face cannot be detected, the process returns to step S1 to restart the AR operation from the beginning.

In step S4, the user is instructed to move the AR device 10 toward the paper surface 20. Specifically, “Please move this device toward the page” is displayed on the image display unit 13. Then, the process proceeds to step S5.

In step S5, 0 is set to a numerical parameter t indicating time. Then, the process proceeds to step S6.

At the stage of proceeding to step S6, the user holds the AR device 10 at a position away from the paper surface 20. Then, the processes after step S6 are performed in the AR device 10, and while these processes are being performed, the user brings the AR device 10 closer to the paper surface 20, and finally, on the paper surface 20. An operation of grounding the AR device 10 is performed.

In step S6, the image pickup unit 15 performs shooting. The captured image is sent to the CPU 11 via the bus 18. Since this image is an image taken at time t, it is assumed that I (t). Also, the acceleration value in the z-axis direction from the acceleration sensor 16 is sent to the CPU 11 via the bus 18. Since the acceleration value is a value at time t, it is set to a (t). Then, the process proceeds to step S7.

In step S7, the CPU 11 determines whether the image I (t) is an in-focus image. Specifically, the high frequency component of the image I (t) is examined, and if it is greater than or equal to a predetermined threshold, it is determined that the image is in focus. If it is less than the threshold value, it is determined that the image is out of focus (an out-of-focus image). If the image is in focus, the process proceeds to step S8. Otherwise, the process proceeds to step S11 (FIG. 7).

In step S8, the image I in the memory 14 is sent to the CPU 11 via the bus 18. Then, the CPU 11 performs matching between the image I and the image I (t), and obtains the positional relationship between the image I and the image I (t). That is, three variables S, X (t), and Y (t) that satisfy Equation 1 are obtained.

Here, I (x1, y1) is a pixel value at the position (x1, y1) of the image I. I (t, x, y) is a pixel value at the position (x, y) of the image I (t). In other words, the image I (t) is transformed by Equation 1 to calculate S, X (t), and Y (t) such that the image I is obtained. In Expression 1, an image obtained by enlarging / reducing the image I (t) by S times and further translating by X (t) in the x-axis direction and Y (t) in the y-axis direction becomes the image I. It means that. This is shown in FIG. FIG. 8A shows an image I, and FIG. 8A shows an image I (t). Each of the images in FIG. 8 is composed of horizontal W pixels and vertical H pixels. Since the user brings the AR device 10 close to the paper surface 20, the image I (t) is an enlarged image with respect to the image I. This enlargement factor is S. The projected image at an arbitrary position (x, y) of the image I (t) is the same projected image as the position (x1, y1) of the image I. As a method for matching two images (calculation of positional relationship), a known method such as a matching method using feature points may be used, and a detailed description thereof will be omitted.

After step S8, the process proceeds to step S9.

In step S9, the CPU 11 determines what positional relationship the center (x, y) = (0, 0) of the photographed image I (t) is with respect to the image I. In Expression 1, since (x, y) = (0, 0) may be substituted, the result is (X (t), Y (t)). That is, in step S9, among S, X (t), and Y (t) obtained in the immediately preceding step, only S is discarded and X (t) and Y (t) are held. After step S9, the process proceeds to step S10.

In step S10, it waits for unit time (for example, 1/30 second). Since the unit time has been advanced, t is incremented by one. After step S10, the process returns to step S6, and the process for the updated t is performed.

During the processing from step S6 to step S10, the user brings the AR device 10 close to the paper surface 20. Therefore, after a while, only a blurred image (an image out of focus) can be taken as an image taken too close to the subject. When such an image is taken, the process proceeds to step S11.

In step S11, the current value of t is set in the numerical parameter t1 in order to store the time when the focused image cannot be captured. Then, the process proceeds to step S12.

In step S12, it is detected whether the value of the acceleration sensor 16 in the z-axis direction is discontinuous. If not discontinuous, the process proceeds to step S13, and if discontinuous, the process proceeds to step S15.

In step S13, it waits for unit time (for example, 1/30 second). Since the unit time has been advanced, t is incremented by one. Then, the process proceeds to step S14.

In step S <b> 14, the acceleration value in the z-axis direction from the acceleration sensor 16 is sent to the CPU 11 via the bus 18. Since the acceleration value is a value at time t, it is set to a (t). Then, the process returns to step S12.

Proceeding to step S15 means that the value of the acceleration sensor 16 in the z-axis direction has detected a discontinuity. That is, it means that the AR device 10 has contacted (grounded) the paper surface 20. In step S15, in order to store the time when the AR device 10 contacts (grounds) the paper surface 20, the current value of t is set in the numerical parameter t2. Then, the process proceeds to step S16.

The data calculated in the process so far will be described. An image I taken of the paper surface 20 on which the person 30 is drawn is taken at time t = 0. The center of each image I (t) taken by the imaging unit 15 from time t = 0 to t = t1-1 corresponds to (X (t), Y (t)) in the image I. Further, the acceleration value in the z-axis direction at each time from time t = 0 to t = t2 is a (t).

In step S16, a distance L between the AR device 10 and the paper surface 20 when the image I is captured is obtained. That is, the CPU 11 integrates a (t) twice from t = 0 to t = t2, and sets the value to L. Since the time when the image I is captured is t = 0 and the time when the AR device 10 is in contact (ground) with the paper surface 20 is t = t2, if the acceleration a (t) in this period is integrated twice, the distance L can be obtained.

By obtaining the distance L, the size of the subject photographed by the image I can be known. That is, it is possible to know which range on the paper surface 20 has been shot. As shown in FIG. 9, the position of the imaging unit 15 at the time when the image I was captured is separated from the paper surface 20 by a distance L. Then, the imaging unit 15 performs imaging with an angle of view of Ah in the horizontal direction (x-axis direction) and an angle of view of Av in the vertical direction (y-axis direction). Therefore, it can be seen that a rectangular region having a length of 2L × tan (Ah / 2) in the horizontal direction and a length of 2L × tan (Av / 2) in the vertical direction is photographed on the paper surface 20.

After step S16, the process proceeds to step S17.

In step S17, the positional relationship of the center of the image display unit 13 with respect to the image I at the time of contact (that is, time t = t2) is extrapolated. Let the extrapolated position be (X (t2), Y (t2)). This is extrapolated by the CPU 11 using values (X (t), Y (t)) from t = 0 to t = t1-1. Various methods exist for extrapolation. For example, extrapolation may be performed using a linear regression model. That is, A1 and B1 that minimize Equation 2 are obtained. Then, X (t2) = A1 × t2 + B1 may be satisfied. This is shown in FIG. What is necessary is just to obtain | require similarly about Y (t2).

After time t = t1, because of close-up shooting, the image is not focused and is not acquired. However, if an image is acquired, the center of the image at time t = t2 is (X ( It can be said that it corresponds to t2) and Y (t2)).

A position on the paper surface 20 corresponding to the position (X (t2), Y (t2)) in the image I is defined as P (see FIG. 9). That is, the projected image of the subject at the position P on the paper surface 20 is the position (X (t2), Y (t2)) in the image I. At time t = t2, if the image pickup unit 15 is shooting, it is estimated that the center of the shot image is shooting the position P. Considering that the AR device 10 is in contact with the paper surface 20 at time t = t2, the position of the imaging unit 15 matches the position P on the paper surface 20. This is shown in FIG.

After step S17, the process proceeds to step S18.

In step S18, the face in the image I is deformed to create the moving image Ia. Specifically, the CPU 11 may apply the technology developed by the above-mentioned Motion Portrait Co., Ltd. Similar to the image I, the moving image Ia is an image group of horizontal W pixels and vertical H pixels. Then, the process proceeds to step S19.

In step S19, a moving image Ib is created from the moving image Ia by cutting out a rectangular region of horizontal WC pixels and vertical HC pixels centered on the position (X (t2), Y (t2)) (see FIG. 12). Here, WC and HC are values shown in Equation 3.

After step S19, the process proceeds to step S20.

In step S20, the moving image Ib is displayed on the image display unit 13 in the AR device 10 (playing a moving image).

The size of the image (moving image Ib) displayed here will be described. On the paper surface 20, the shooting range of the image I is 2L × tan (Ah / 2) in the horizontal direction and 2L × tan (Av / 2) in the vertical direction (described above). The size of the image display unit 13 is horizontal WD and vertical HD. The size of the moving image Ia is the same as that of the image I, and is W in the horizontal direction and H in the vertical direction. Therefore, when the horizontal WC and vertical HC sizes shown in Expression 3 are cut out from the moving image Ia and displayed on the image display unit 13, the size is the same as the size on the paper surface 20.

Next, the position of the displayed image (moving image Ib) will be described. The center of the moving image Ib is the position (X (t2), Y (t2)) in the moving image Ia (that is, the image I). The position on the paper surface 20 corresponding to the position (X (t2), Y (t2)) in the image I (that is, the moving image Ia) is P. Since the center of the image display unit 13 and the imaging unit 15 are at the same position, the position of the center of the image display unit 13 of the AR device 10 that is grounded is the position P on the paper surface 20. Therefore, the center position of the moving image Ib matches the center position of the grounded image display unit 13 on the paper surface 20.

As apparent from the description of the size and position of the moving image Ib, the moving image displayed on the image display unit 13 as viewed from the user is a real-world landscape (a person 30 drawn on the paper surface 20) that is visible outside the AR device 10. An image with a sense of unity. In other words, the image display unit 13 can appear as if it is a transparent glass. FIG. 13 shows a scene viewed from the user. As shown in FIG. 13, the AR device 10 is visible to the user. On the image display unit 13 in the AR device 10, a captured image (more precisely, a moving image in which a face portion is deformed in the captured image) can be seen. In addition, a real world landscape can be seen outside the AR device 10. That is, the paper surface 20 can be seen. Furthermore, a part of the person 30 drawn on the paper surface 20 can also be seen. In FIG. 13, the paper surface 20 and the image displayed in the image display unit 13 have a sense of unity. This is because the size of the target person is the same. That is, the size of the person in the image display unit 13 is the same as the size of the person 30 drawn on the paper 20 in the real world.

After step S20, the process proceeds to step S21.

In step S21, the value of the acceleration sensor 16 in the z-axis direction is detected. If the acceleration value is 0, it can be determined that the AR device 10 continues to be grounded on the paper surface 20, so the process returns to step S20 to continue the reproduction of the moving image Ib. If the acceleration value is not 0, it can be determined that the user has separated the AR device 10 from the paper surface 20, and the process proceeds to step S22.

In step S22, since the user has separated the AR device 10 from the paper surface 20, the display on the image display unit 13 is terminated and the AR operation is terminated.

This is the end of the description of the embodiment of the present invention.

The important points in the above embodiment will be described below.
(Important Point 1) Photographing is performed by the imaging unit 15 of the AR device 10, and an image I is obtained. The size of the shooting range of the image I is unknown simply by shooting. Therefore, the distance L is obtained by moving the AR device 10 between the position where the image I is obtained and the paper surface 20. By using this distance L and the angle of view (horizontal direction Ah and vertical direction Av) of the imaging unit 15, the size of the shooting range of the image I (2L × tan (Ah / 2) in the horizontal direction) Now, a length of 2L × tan (Av / 2) in the vertical direction) is obtained. In this way, the size of the shooting range can be obtained.
(Important Point 2) Further, the same size as the size of the image display unit 13 (horizontal WD, vertical HD) is set to the size of the shooting range of the image I (2L × tan (Ah / 2) in the horizontal direction). Now, it is cut out from data of 2L × tan (Av / 2)) in the vertical direction. As a result, the size of the display object on the image display unit 13 and the size of the real-world landscape seen outside the AR device 100 match.
(Important Point 3) Further, the position to be cut out from the image I is determined by obtaining the position (the aforementioned position P) where the AR device 10 is grounded. As a result, the position of the display object on the image display unit 13 and the position of the real-world landscape seen outside the AR device 100 coincide.
By executing at least one of the above three important points, an unprecedented merit can be obtained. And, by executing all the three important points, the maximum merit can be obtained.

In the present invention, the AR device 10 is grounded and used with respect to the paper surface 20. For this reason, regardless of the position of the user's eyes, the image display unit 13 displays an appropriate area of the image I obtained by photographing the paper surface 20 in the same size as the paper surface 20, so that an actual landscape (paper surface 20) is displayed. Image display with a sense of unity. That is, the image display unit 13 can appear as if it is a transparent glass.

The area displayed on the image display unit 13 is at least one of the shooting range of the image I obtained by shooting the paper surface 20, the grounding position P of the paper surface 20 of the imaging unit 15, and the size (WD and HD) of the image display unit 13. By using one piece of information, it can be made an appropriate area.

In the embodiment of the present invention, the position (X (t2), Y (t2)) is obtained from the images I and I (t), but the acceleration sensor from time t = 0 to t = t2 is obtained. You may obtain | require using the value of the x-axis direction and y-axis direction.

Further, since an error may occur in the above processing, the user may be able to manually fine-tune the size and position to be cut out from the moving image Ia after the grounding.

Moreover, although the said Example of the said invention was an example which displays the moving image which deform | transformed the face, this invention is not necessarily limited to a face. For example, the moving image Ia may be created by shooting a paper on which a landscape photograph is posted and creating a moving image in which the color of the landscape is gradually changed to red. As a result, the user can see a moving image that gradually becomes dusk.

As mentioned above, although embodiment of this invention was explained in full detail, the concrete structure is not restricted to this embodiment, The design change etc. of the range which does not deviate from the summary of this invention are included.

Furthermore, this technique can also be set as the following structures.

[1]
A shooting step of shooting a target object and obtaining a shot image with an imaging device placed at the shooting position;
A measurement step of measuring the distance between the shooting position and the target object by moving the imaging device between the shooting position and the target object;
A method for measuring the size of an object, comprising: a calculation step of calculating the size of the target object based on a viewing angle of the imaging device and the distance.
[2]
In a method for displaying an image using an apparatus in which an imaging unit and an image display unit are integrated,
In the imaging unit of the apparatus placed at the shooting position, a shooting step of shooting a target object and obtaining a shot image;
A cutout step of cutting out a part of an image using the captured image or, if necessary, the captured image as a material;
A display step of displaying the image cut out in the cutout step on the image display unit when the device is grounded to the target object,
In the clipping step, a region to be clipped is determined using at least one information of the size of the target object, the position when the device is grounded to the target object, and the size of the image display unit. Characteristic image display method.
[3]
In [2] above,
In order to determine the size of the target object,
further,
A measurement step of measuring the distance between the shooting position and the target object by moving the device between the shooting position and the target object;
An image display method comprising: a calculation step for calculating a size of the target object based on a viewing angle of the imaging unit and the distance.
[4]
A captured image acquisition unit that captures a target object and captures a captured image with an imaging unit placed at the capturing position;
A measurement processing unit that measures the distance between the shooting position and the target object by moving the imaging unit between the shooting position and the target object;
An apparatus for measuring the size of a subject, comprising: a calculation processing unit that calculates a size of the target object based on a viewing angle of the imaging unit and the distance.
[5]
In an apparatus in which an imaging unit and an image display unit are integrated,
In the imaging unit of the apparatus placed at the shooting position, a captured image acquisition unit that captures a target object and obtains a captured image;
A cutout processing unit that cuts out a part of an image using the captured image or the captured image as a material as necessary;
An image display control unit that performs processing to display the image cut out by the cut-out processing unit on the image display unit when the device is grounded to the target object;
The cutout processing unit determines a region to be cut out using at least one information of the size of the target object, the position when the device is grounded to the target object, and the size of the image display unit. An apparatus for displaying an image characterized by the above.
[6]
In the above [5],
In order to determine the size of the target object,
further,
A measurement processing unit that measures the distance between the shooting position and the target object by moving the imaging unit between the shooting position and the target object;
An apparatus for displaying an image, comprising: a calculation processing unit that calculates a size of the target object based on a viewing angle of the imaging unit and the distance.
[7]
A shooting step of shooting a target object and obtaining a shot image with an imaging device placed at the shooting position;
A measurement step of measuring the distance between the shooting position and the target object by moving the imaging device between the shooting position and the target object;
A program to be executed by a computer for measuring the size of a subject, comprising a calculation step of calculating the size of the target object based on the viewing angle of the imaging device and the distance.
[8]
In a program for displaying an image using a device in which an imaging unit and an image display unit are integrated,
In the imaging unit of the apparatus placed at the shooting position, a shooting step of shooting a target object and obtaining a shot image;
A cutout step of cutting out a part of an image using the captured image or, if necessary, the captured image as a material;
A display step of displaying the image cut out in the cutout step on the image display unit when the device is grounded to the target object,
In the clipping step, a region to be clipped is determined using at least one information of the size of the target object, the position when the device is grounded to the target object, and the size of the image display unit. A program to be executed by a computer for displaying a characteristic image.
[9]
In the above [8],
In order to determine the size of the target object,
further,
A measurement step of measuring the distance between the shooting position and the target object by moving the device between the shooting position and the target object;
A program to be executed by a computer for displaying an image, characterized in that a calculation step for calculating the size of the target object is provided according to the viewing angle of the imaging unit and the distance.

10. Image display device (AR device)
DESCRIPTION OF SYMBOLS 11 ... Central processing part 12 ... Operation part 13 ... Image display part 14 ... Memory 15 ... Imaging part 16 ... Acceleration sensor 17 ... Network interface part 18 ... Bus 20 ... Paper surface 30 ... Person 40 drawn on paper surface ... User's eyes 50 ... Direction of moving image display device (AR device) 100 ... Conventional image display device (conventional AR device)
110: Image display unit 120: Imaging unit

Claims (9)

A shooting step of shooting a target object and obtaining a shot image with an imaging device placed at the shooting position;
A measurement step of measuring the distance between the shooting position and the target object by moving the imaging device between the shooting position and the target object;
A method for measuring the size of an object, comprising: a calculation step of calculating the size of the target object based on a viewing angle of the imaging device and the distance. In a method for displaying an image using an apparatus in which an imaging unit and an image display unit are integrated,
In the imaging unit of the apparatus placed at the shooting position, a shooting step of shooting a target object and obtaining a shot image;
A cutout step of cutting out a part of an image using the captured image or, if necessary, the captured image as a material;
A display step of displaying the image cut out in the cutout step on the image display unit when the device is grounded to the target object,
In the clipping step, a region to be clipped is determined using at least one information of the size of the target object, the position when the device is grounded to the target object, and the size of the image display unit. Characteristic image display method. In claim 2,
In order to determine the size of the target object,
further,
A measurement step of measuring the distance between the shooting position and the target object by moving the device between the shooting position and the target object;
An image display method comprising: a calculation step for calculating a size of the target object based on a viewing angle of the imaging unit and the distance. A captured image acquisition unit that captures a target object and captures a captured image with an imaging unit placed at the capturing position;
A measurement processing unit that measures the distance between the shooting position and the target object by moving the imaging unit between the shooting position and the target object;
An apparatus for measuring the size of a subject, comprising: a calculation processing unit that calculates a size of the target object based on a viewing angle of the imaging unit and the distance. In an apparatus in which an imaging unit and an image display unit are integrated,
In the imaging unit of the apparatus placed at the shooting position, a captured image acquisition unit that captures a target object and obtains a captured image;
A cutout processing unit that cuts out a part of an image using the captured image or the captured image as a material as necessary;
An image display control unit that performs processing to display the image cut out by the cut-out processing unit on the image display unit when the device is grounded to the target object;
The cutout processing unit determines a region to be cut out using at least one information of the size of the target object, the position when the device is grounded to the target object, and the size of the image display unit. An apparatus for displaying an image characterized by the above. In claim 5,
In order to determine the size of the target object,
further,
A measurement processing unit that measures the distance between the shooting position and the target object by moving the imaging unit between the shooting position and the target object;
An apparatus for displaying an image, comprising: a calculation processing unit that calculates a size of the target object based on a viewing angle of the imaging unit and the distance. A shooting step of shooting a target object and obtaining a shot image with an imaging device placed at the shooting position;
A measurement step of measuring the distance between the shooting position and the target object by moving the imaging device between the shooting position and the target object;
A program to be executed by a computer for measuring the size of a subject, comprising a calculation step of calculating the size of the target object based on the viewing angle of the imaging device and the distance. In a program for displaying an image using a device in which an imaging unit and an image display unit are integrated,
In the imaging unit of the apparatus placed at the shooting position, a shooting step of shooting a target object and obtaining a shot image;
A cutout step of cutting out a part of an image using the captured image or, if necessary, the captured image as a material;
A display step of displaying the image cut out in the cutout step on the image display unit when the device is grounded to the target object,
In the clipping step, a region to be clipped is determined using at least one information of the size of the target object, the position when the device is grounded to the target object, and the size of the image display unit. A program to be executed by a computer for displaying a characteristic image. In claim 8,
In order to determine the size of the target object,
further,
A measurement step of measuring the distance between the shooting position and the target object by moving the device between the shooting position and the target object;
A program to be executed by a computer for displaying an image, characterized in that a calculation step for calculating the size of the target object is provided according to the viewing angle of the imaging unit and the distance.

Images