JP7414197B2 - Virtual gauge display program and device - Google Patents

Description

The present invention relates to a virtual gauge display program and device that displays a virtual gauge for measuring or estimating the size of an object, and more particularly to a virtual gauge display program that displays a virtual gauge alongside an object using augmented reality technology. and devices.

Before loading cargo into the cargo hold of an aircraft, etc., it is necessary to measure the size of the cargo to confirm that the size of the cargo can be accommodated in the cargo hold. In the conventional method, when measuring cargo, a worker measures the size using a contour gauge. The contour gauge has an inverted L-shape and is equipped with an aluminum support sized to correspond to the cargo hold of an aircraft. The pillar was over 3 meters long, and the center of gravity of the inverted L-shaped contour gauge was located at the top, making it difficult to handle and creating a workload. Furthermore, contour gauges with poles longer than approximately 3 meters can only be stored at the edge of the warehouse, and workers must go to the edge of the vast warehouse to retrieve the contour gauges every time they measure air cargo, etc. Moreover, the loss of working time was also large.

On the other hand, since 2000, with the spread of smartphones, applications using augmented reality (AR) have become generally available. In particular, in 2017 Apple Inc. Application development has progressed since the AR development tool ``ARKit'' by Google and the AR development tool ``ARCore'' by Google LLC.

Further, technologies related to AR are proposed in Patent Documents 1 and 2, for example. Patent Document 1 describes a "portable device configured to provide an augmented reality experience, the device having a display screen configured to display a real-world scene; an image capture device configured to capture image data representative of a real-world scene; an image recognition logic circuit configured to analyze the image data representative of the real-world scene; and image generation logic responsive to image recognition logic and configured to incorporate additional images into the real-world scene.''

Cited Document 2 states, ``The game device acquires a captured image from a camera, and repeatedly performs recognition processing on the acquired captured image to recognize a predetermined object included in the captured image. The device repeatedly generates and displays a composite image that combines the image of the virtual object generated using the processing result of the recognition process and the captured image that is the target of the recognition process. If the recognition process fails, the captured image that was the target of the previously successful recognition process and the image of the virtual object generated using the results of the recognition process are combined.'' (Abstract) Are listed.

Special Publication No. 2008-510254 Japanese Patent Application Publication No. 2011-076589

Patent Document 1 incorporates and displays an additional image on a display screen that displays the real world, and Patent Document 2 synthesizes and displays a virtual object with a captured image. However, with AR technology that uses additional images, virtual objects, etc., workers cannot measure or estimate the size or dimensions of real objects such as cargo.

Therefore, an object of the present invention is to provide a virtual gauge display program and apparatus that display a virtual gauge for measuring or estimating the size of an object using AR.

Each aspect of the present invention is as follows.
[Aspect 1]
A virtual gauge display program that causes a virtual gauge display device to display a virtual gauge together with a real space image using augmented reality,
The virtual gauge display device includes a photographing unit that photographs the real space image, a display unit that displays the real space image and the virtual gauge, a storage unit that stores the virtual gauge, and an input regarding the arrangement of the virtual gauge. an input unit that receives the image capturing unit, the display unit, the storage unit, and a control unit that controls the input unit,
The virtual gauge display program includes:
a photographing step of causing the photographing unit to photograph the real space image;
a plane detection step of causing the control unit to detect a plane from the real space image;
a coordinate specifying step of causing the control section to specify the coordinates of the virtual gauge on the plane based on the input from the input section;
a virtual gauge display step of causing the display unit to display the virtual gauge on the plane together with the real space image based on the coordinates of the virtual gauge on the plane;
A virtual gauge display program that runs .

[Aspect 2]
In the virtual gauge display program according to aspect 1,
In the virtual gauge display step, the virtual gauge display program causes the display unit to display the lower end of the virtual gauge at coordinates on the plane.
[Aspect 3]
In the virtual gauge display program according to aspect 1 or 2,
The virtual gauge is a virtual gauge display program, wherein the virtual gauge includes a vertical portion extending vertically with respect to the plane and a horizontal portion extending horizontally with respect to the plane.
[Aspect 4]
In the virtual gauge display program according to any one of aspects 1 to 3,
The storage unit stores a plurality of virtual gauges having mutually different shapes,
The virtual gauge display program is a virtual gauge display program that causes the display unit to execute a virtual gauge selection step of displaying the plurality of virtual gauges in a selectable manner.

[Aspect 5]
In the virtual gauge display program according to any one of aspects 1 to 4,
The virtual gauge display program causes the display unit to execute a positioning mark display step of displaying a positioning mark for arranging the virtual gauge on the plane.
[Aspect 6]
In the virtual gauge display program according to aspect 5,
The virtual gauge display program causes the control unit to generate a virtual line extending from the imaging direction of the imaging unit to the plane, and causes the display unit to display the positioning mark at the intersection of the plane and the virtual line. Virtual gauge display program.
[Aspect 7]
In the virtual gauge display program according to aspect 6,
The virtual gauge display program causes the control unit to determine that the intersection of the plane and the virtual line exists in the center area of the screen of the display unit, and when the intersection exists in the center area of the screen, the control unit A virtual gauge display program that displays the positioning mark in the center area of the screen.

[Aspect 8]
In the virtual gauge display program according to any one of aspects 1 to 7,
The virtual gauge display program causes the control unit to correct the position and/or orientation of the virtual gauge when the real space image changes.
[Aspect 9]
In the virtual gauge display program according to any one of aspects 1 to 8,
The virtual gauge display program causes the display unit to execute a rotation display step of displaying the virtual gauge rotatably about a direction perpendicular to the plane.
[Aspect 10]
In the virtual gauge display program according to any one of aspects 1 to 9,
The virtual gauge display program is a virtual gauge display program that causes the display unit to display a state in which the virtual gauge has moved on the plane in response to an input from the input unit.

[Aspect 11]
In the virtual gauge display program according to any one of aspects 1 to 10,
The virtual gauge display program is a virtual gauge display program that causes the display unit to display the virtual gauge having a scale.
[Aspect 12]
In the virtual gauge display program according to any one of aspects 1 to 11,
The virtual gauge display device is a virtual gauge display program that is a portable device.

[Aspect 13]
A virtual gauge display device that displays a virtual gauge together with a real space image using augmented reality,
The virtual gauge display device includes a photographing unit that photographs the real space image, a display unit that displays the real space image and the virtual gauge, a storage unit that stores the virtual gauge, and an input regarding the arrangement of the virtual gauge. an input unit that receives the input unit, and a control unit that controls the imaging unit, the display unit, the storage unit, and the input unit,
The photographing unit photographs the real space image,
The control unit detects a plane from the real space image,
Based on the input from the input unit, the control unit specifies the coordinates of the virtual gauge on the plane,
The display unit displays the virtual gauge on the plane together with the real space image based on the coordinates of the virtual gauge on the plane.

[Aspect 14]
The virtual gauge display device according to aspect 13,
The control unit is a virtual gauge display device that displays on the display unit such that a lower end of the virtual gauge is placed on the plane.
[Aspect 15]
In the virtual gauge display device according to any one of aspects 13 to 14,
The control unit is a virtual gauge display device that causes the display unit to display a positioning mark for arranging the virtual gauge on the plane.
[Aspect 16]
In the virtual gauge display device according to aspect 15,
The control unit generates a virtual line extending from the imaging direction of the imaging unit to the plane, and causes the display unit to display the positioning mark at an intersection of the plane and the virtual line.

The virtual gauge display program and device of the present invention can measure or estimate the size of a real object using a virtual gauge displayed in AR.

FIG. 1 is a block diagram showing a virtual gauge display device according to the present embodiment. FIG. 2 is a hierarchical diagram showing a development environment for a virtual gauge display program according to the present embodiment. 3 is a flowchart regarding a virtual gauge display program according to the present embodiment. 2 is a screen configuration diagram showing a list of virtual gauges of the device in FIG. 1. FIG. 2 is a screen configuration diagram showing a real space image around the device in FIG. 1. FIG. 6 is a screen configuration diagram that displays positioning marks on the real space image of FIG. 5. FIG. FIG. 3 is a screen configuration diagram that displays a virtual gauge on a real space image. 8 is a screen configuration diagram showing a rotated virtual gauge with respect to the real space image of FIG. 7; FIG. It is a schematic diagram which shows an example of the shape of a virtual gauge.

A virtual gauge display program and device according to an embodiment of the present invention will be described with reference to the drawings. Although the virtual gauge display program and device of this embodiment display virtual gauges side by side for cargo such as air cargo, the present invention is not limited to cargo and can display virtual gauges side by side for any object. Can be done.

[overview]
The virtual gauge display program and device according to the present embodiment photograph the real space with a camera of a device (tablet terminal or mobile device such as a smartphone), associate the real space with the virtual space using AR, and create a virtual gauge of the same size as the real thing. A virtual gauge (virtual contour gauge) is displayed by appearing (synthesizing) at three-dimensional coordinates on a real space image. From the operator's perspective, it appears as if a real contour gauge is appearing on the screen of the device (portable device). Then, by comparing the virtual gauge with the cargo such as air cargo, the worker can measure or estimate the size of the cargo, and the worker can determine whether the cargo is large enough to be loaded into the cargo hold of the aircraft, etc. You can check whether it is true or not.

A virtual gauge display device 1 according to an embodiment of the present invention will be explained using the block diagram of FIG. As shown in FIG. 1, the virtual gauge display device 1 includes a photographing unit 12 such as a digital camera for photographing real space, a storage unit 14 such as a hard disk or memory for storing various data, and a display unit 14 for displaying videos and/or images. a display section 16 such as a liquid crystal display, a communication section 18 that communicates with external equipment via a communication network, an input section 19 such as a touch panel that receives input from the user, and a control section 20 such as a CPU that controls each section. It is equipped with An AR application (virtual gauge display program) 30 is operably installed in the control unit 20. The AR application 30 includes an AR engine 30a. The AR engine 30a can be constructed using functions provided by ARkit, ARCore, or ARToolKit.

FIG. 2 shows an example of a hierarchical structure configuring the development environment of the virtual gauge display program and the device 1. This hierarchical structure consists of a hardware layer 60 consisting of the various parts shown in Figure 1, an operating system layer 50 such as iOS or Android, a software development kit (SDK) layer 40 such as iOS SDK or Android SDK, and an AR application layer 40. and an application layer 30 including 30. The SDK layer 40 is used when developing the AR application 30 and the like. The AR application 30 is configured using functions provided by AR Kit, ARCore, or ARToolKit provided in the SDK layer 40. The application layer 30 is preferably provided with a graphics engine such as Unity (Unity 3D) that provides an integrated development environment (IDE). AR Fundation is an AR development package that preferably runs on Unity and provides basic functions related to AR. SQLite is a database that is embedded in an application and is preferably loaded into AR Foundation. The data of the 3D model (virtual gauge described below) is preferably stored in SQLite.

[Outline of processing]
An overview of the processing executed by the virtual gauge display program of this embodiment will be explained.
In the first step, when the AR application is started, a virtual space is created so as to match (correspond to) the real space with the shooting position of the virtual gauge display device 1 as the origin (0, 0, 0). At this time, preferably, the space is created so that 1 m of the real space corresponds to 1 unit of the virtual space.
In the second step, the self-position of the virtual gauge display device 1 in the virtual space is estimated or determined by motion tracking and image processing.
In the third step, the position of a reference plane (for example, a horizontal plane) is detected from the surrounding image by image processing, and the position of the plane is arranged in the virtual space.
The fourth step is to obtain the coordinates of the point where the virtual line created from the position and orientation of the camera intersects with the plane placed in the third step.
In the fifth step, the virtual gauge is arranged so that the lower end of the virtual gauge (virtual contour gauge) is located at the coordinates obtained in the fourth step.
The size of the virtual gauge is preferably set in advance so that 1 unit=1 m, so that it is displayed in the same size as the one in real space.

Next, more detailed processing of the virtual gauge display program and device 1 according to this embodiment will be explained using the flowchart of FIG. 3. The virtual gauge display device 1 of this embodiment can preferably be a mobile device such as a smartphone or a tablet terminal in which the AR application 30 is installed. When estimating the object size, the worker operates the input unit 19 to start the AR application. Next, in step S301, the control unit 20 acquires information on selectable virtual gauges (virtual contour gauges) from the database (DB) provided in the storage unit 14, and as shown in the list screen of FIG. Then, a list of a plurality of virtual gauges G having different shapes is displayed on the display unit 16. In step S302, while the screen of FIG. Virtual gauge selection information is input.

In step S303, the control unit 20 requests the operator to confirm whether the selection of the virtual gauge is correct. This confirmation is preferably performed by displaying the selected virtual gauge on the display unit 16 and requesting an input of “Use” or “Do not use” via the touch panel (input unit 19). In step S303, when the control unit 20 confirms the input of "use", the process moves to step S304. In step S302, if the control unit 20 detects an input of "do not use" or does not detect an input of "use" within a predetermined time, the process returns to step S301.

In step S304, the control unit 20 activates the photographing unit (camera) 12. In step S305, the control unit 20 displays the camera image (surrounding real space image) photographed by the photographing unit 12 on the screen of the display unit 16, and captures data of the camera image.

In step S306, the control unit 20 uses the AR engine 30a to generate a virtual space corresponding to the photographed surrounding real space image. In step S307, the control unit 20 performs motion tracking on a plurality of surrounding real space images, and uses the AR engine 30a to estimate (generate) the coordinates of the self position (self viewpoint position) with respect to the virtual space. During motion tracking, the control unit 20 uses the AR engine 30a to extract the coordinates of a plurality of feature points for each of the plurality of surrounding real space images, and calculates the self-position (self-viewpoint position) from the coordinates of the plurality of feature points. ) is estimated (generated).

In step S308, as shown in FIG. 5, the operator moves the virtual gauge display device 1 and uses the photographing unit 12 to photograph a real space image of the surroundings.

In step S309, the control unit 20 generates a virtual line vector extending from the position of the imaging unit (origin (0, 0, 0)) in the imaging direction.

In step S310, the control unit 20 executes plane detection using the AR engine 30a based on the surrounding real space image obtained in step S308 and the distances to the plurality of feature points, and detects a plane in the real space (e.g. , horizontal plane). In step S311, the control unit 20 determines whether the plane detection is successful. The success of plane detection is determined by the control unit 20 based on whether the state (arrangement) of the plane calculated by plane detection satisfies a predetermined condition. In step S311, if the control unit 20 determines that the plane detection has been successful, the process moves to step S312, and if the control unit 20 determines that the plane detection has failed, the process returns to step S308.

In step S312, the control unit 20 arranges a plane in the virtual space (sets the coordinates of the plane with respect to the virtual space). In step S313, the control unit 20 determines that while the real space image is being displayed on the screen of the display unit 16, an intersection point between the plane and the virtual line exists on the screen of the display unit 16 (preferably in the center area of the screen). Determine whether or not to do so. In step S313, if the control unit 20 determines that an intersection exists, the process moves to step S314, and if the control unit 20 determines that no intersection exists, the process returns to step S308.

In step S314, the control unit 20 displays the positioning mark M in the center area of the screen of the display unit 16, as shown in FIG. The positioning mark M indicates the position (coordinates) on the plane where the lower end of the virtual gauge is installed. Note that the central area of the screen is an area excluding the peripheral area of the screen, and can be an area where the lower end of the virtual gauge can be placed.

In step S315, the control unit 20 determines whether an input such as a tap on the positioning mark M by the operator is detected via the input unit 19. In step S315, when the control unit 20 detects an input such as a tap, it specifies the coordinates of the positioning mark M and proceeds to step S316, and if the control unit 20 does not detect an input such as a tap within a predetermined time, step Return to S308.

In step S316, the control unit 20 uses the AR engine 30a to add virtual coordinates to the coordinates on the plane where the positioning mark M was displayed, with respect to the real space image displayed by the display unit 16, as shown in FIG. A virtual gauge G appears or is displayed so that the lower end of the gauge is located.

In step S317, when the surrounding image changes as the virtual gauge display device (portable device) 1 moves, the control unit 20 can correct the position and/or orientation of the virtual gauge using the AR engine 30a. Further, as shown in FIG. 8, the control unit 20 uses the AR engine 30a to display a state in which the virtual gauge is rotated in an arbitrary direction around a vertical line with respect to the plane, in response to the input from the input unit 19. You can also. Further, the control unit 20 can also display a state in which the virtual gauge is moved on a plane using the AR engine 30a in response to an input from the input unit 19.

In this embodiment, the control unit 20 can preferably fix and display the relative size of the virtual gauge G with respect to the cargo (real object) C. In this embodiment, the control unit 20 can preferably display the virtual gauge G in a size corresponding to the real gauge (contour gauge) in an overlapping manner on the surrounding screen.

As shown in FIG. 9, the virtual gauge G has a vertical portion (vertical portion) Gv extending in the vertical direction (substantially vertical direction) and a horizontal portion (horizontal portion) Gh extending in the horizontal direction (substantially horizontal direction). have When arranging the virtual gauge on the detection plane, the control unit 20 arranges the vertical portion Gv in a direction substantially perpendicular to the detection plane, and arranges the horizontal portion Gh in a direction substantially parallel to the detection plane. do. A length scale may be displayed on the vertical and/or horizontal portions of the virtual gauge. As a result, the operator can measure or estimate the vertical size of the cargo C by the vertical portion Gv displayed vertically on the side of the cargo, and the horizontal size Gh displayed horizontally on the upper side of the cargo. The lateral size of cargo C can be measured or estimated.

By using the virtual gauge display program and device according to the present embodiment, the operator only needs to hold the virtual gauge display device such as a tablet terminal, without having to hold a real contour gauge that is unbalanced and heavy. , the workload is reduced. In addition, workers can measure or estimate the size of cargo using a virtual gauge display device such as a tablet at hand, without having to go to the edge of the warehouse to retrieve a contour gauge that is about 3 meters long. You can prevent time loss.

[Other application examples]
The virtual gauge display program and device according to this embodiment can be applied to measurements of things that are difficult to measure by directly applying a ruler, and to measurements of the distance between two points on a large object. For example, the following Examples 1 to 3 can be considered.
Example 1: When loading cargo stacked on pallets into the cargo space of a trailer (truck) using a forklift in a warehouse, it is possible to check with a virtual gauge whether the cargo stacked on the pallets will fit into the cargo space.
Example 2: It is possible to measure the width of passageways and the size of objects in places that are difficult for people to access, such as at high places, by applying a virtual gauge from a short distance.
Example 3: It is possible to determine whether the height, size, shape, etc. of an object from the bottom are normal by comparing it with a virtual gauge.

1 Virtual gauge display device 12 Photographing unit 14 Storage unit 16 Display unit 17 Distance measurement unit 18 Communication unit 19 Input unit 20 Control unit 30 AR application (virtual gauge display program)
30a AR engine

Claims (16)

A virtual gauge display program that causes a virtual gauge display device to display a virtual gauge together with a real space image using augmented reality,
The virtual gauge display device includes a photographing unit that photographs the real space image, a display unit that displays the real space image and the virtual gauge, a storage unit that stores the virtual gauge, and an input regarding the arrangement of the virtual gauge. an input unit that receives the image capturing unit, the display unit, the storage unit, and a control unit that controls the input unit,
The virtual gauge display program includes:
a photographing step of causing the photographing unit to photograph the real space image;
a plane detection step of causing the control unit to detect a plane from the real space image;
a coordinate specifying step of causing the control section to specify the coordinates of the virtual gauge on the plane based on the input from the input section;
a virtual gauge display step of causing the display unit to display the virtual gauge on the plane together with the real space image based on the coordinates of the virtual gauge on the plane;
A virtual gauge display program that runs . The virtual gauge display program according to claim 1,
In the virtual gauge display step, the virtual gauge display program causes the display unit to display the lower end of the virtual gauge at coordinates on the plane. The virtual gauge display program according to claim 1 or 2,
The virtual gauge is a virtual gauge display program, wherein the virtual gauge includes a vertical portion extending in the vertical direction with respect to the plane, and a horizontal portion extending in the horizontal direction with respect to the plane. The virtual gauge display program according to any one of claims 1 to 3,
The storage unit stores a plurality of virtual gauges having mutually different shapes,
The virtual gauge display program is a virtual gauge display program that causes the display unit to execute a virtual gauge selection step of displaying the plurality of virtual gauges in a selectable manner. The virtual gauge display program according to any one of claims 1 to 4,
The virtual gauge display program causes the display unit to execute a positioning mark display step of displaying a positioning mark for arranging the virtual gauge on the plane. The virtual gauge display program according to claim 5,
The virtual gauge display program causes the control unit to generate a virtual line extending from the imaging direction of the imaging unit to the plane, and causes the display unit to display the positioning mark at the intersection of the plane and the virtual line. Virtual gauge display program. The virtual gauge display program according to claim 6,
The virtual gauge display program causes the control unit to determine that the intersection of the plane and the virtual line exists in the center area of the screen of the display unit, and when the intersection exists in the center area of the screen, the control unit A virtual gauge display program that displays the positioning mark in the center area of the screen. The virtual gauge display program according to any one of claims 1 to 7,
The virtual gauge display program causes the control unit to correct the position and/or orientation of the virtual gauge when the real space image changes. The virtual gauge display program according to any one of claims 1 to 8,
The virtual gauge display program causes the display unit to execute a rotation display step of displaying the virtual gauge rotatably around a direction perpendicular to the plane. The virtual gauge display program according to any one of claims 1 to 9,
The virtual gauge display program is a virtual gauge display program that causes the display unit to display a state in which the virtual gauge has moved on the plane in response to an input from the input unit. The virtual gauge display program according to any one of claims 1 to 10,
The virtual gauge display program is a virtual gauge display program that causes the display unit to display the virtual gauge having a scale. The virtual gauge display program according to any one of claims 1 to 11,
The virtual gauge display device is a virtual gauge display program that is a portable device. A virtual gauge display device that displays a virtual gauge together with a real space image using augmented reality,
The virtual gauge display device includes a photographing unit that photographs the real space image, a display unit that displays the real space image and the virtual gauge, a storage unit that stores the virtual gauge, and an input regarding the arrangement of the virtual gauge. an input unit that receives the image capturing unit, the display unit, the storage unit, and a control unit that controls the input unit,
The photographing unit photographs the real space image,
The control unit detects a plane from the real space image,
Based on the input from the input unit, the control unit specifies the coordinates of the virtual gauge on the plane,
The display unit displays the virtual gauge on the plane together with the real space image based on the coordinates of the virtual gauge on the plane. The virtual gauge display device according to claim 13,
The control unit is a virtual gauge display device that displays on the display unit such that a lower end of the virtual gauge is placed on the plane. The virtual gauge display device according to any one of claims 13 to 14,
The control unit is a virtual gauge display device that causes the display unit to display a positioning mark for arranging the virtual gauge on the plane. The virtual gauge display device according to claim 15,
The control unit generates a virtual line extending from the imaging direction of the imaging unit to the plane, and causes the display unit to display the positioning mark at an intersection of the plane and the virtual line.