KR102159721B1 - Communication apparatus providing augmented reality service using amr space position information, and control method thereof - Google Patents

Abstract

The present invention relates to a control method of a communication terminal providing an augmented reality (AR) service by utilizing autonomous mobile robot (AMR) space location information. According to the present invention, the control method of the communication terminal providing the AR service by utilizing AMR space location information comprises the steps of: photographing AR marker using a provided photographing camera; estimating an initial position and an initial posture of the camera based on the photographed image of the AR marker; receiving position information of the AMR from a server; determining whether the AMR exists in a camera photographing area based on location information of the AMR received in the previous step, the initial location and the initial posture of the camera estimated in the previous step, and movement information obtained through a previously provided movement sensor; and displaying a virtual object corresponding to the AMR using an AR technique when the AMR exists in the camera photographing area as a result of the determination of the previous step.

Description

Communication terminal that provides AR service using AMR spatial location information and its control method {COMMUNICATION APPARATUS PROVIDING AUGMENTED REALITY SERVICE USING AMR SPACE POSITION INFORMATION, AND CONTROL METHOD THEREOF}

The present invention relates to a communication terminal and a control method thereof, and more particularly, to a communication terminal that provides an AR service using AMR spatial location information, and a control method thereof.

AGV (Automatic Guided Vehicle) is a term referring to an industrial vehicle that automatically moves without a driver, and is used as an official term in the American National Standards Institute (ANSI).

AGV has established itself as a solution that automatically moves materials in manufacturing facilities and distribution centers, and recently, products with the term AMR (Autonomous Mobile Robot) are receiving great attention.

Some companies even say that AMR will replace AGV and bring about a new era for logistics movement. For reference, the relationship between AMR and AGV is as follows.

1) AMR is a subset of AGV.

AMR claims to be differentiated from existing AGVs and implements a completely new material movement, but these are also a subset of AGVs. It is a kind of AGV that focuses on search technology, payload function, and routing function.

2) AGV uses the type of search that is most suitable for the application.

The difference between AMR and AGV is the search method, where in the past AGVs followed a path guided by wires or magnets buried in the facility floor, but today AGVs use the type of search that best suits the application.

Traditional wire guidance is sometimes used, but laser or camera technology can also be used. The AGV's actual navigation path is determined by software that tells where to go, which connects both vehicle management and onboard software to determine the best route, and if the route is blocked by a person or another AGV, it changes to another route. Instruct.

3) AGV can use dynamic guidance path.

The AGV follows the guided path, but does not always go through the same path from A to B. In general, AGVs have a variety of information that can move from one location to another, just like a city street. Of these, AGV software intelligently selects based on traffic, distance traveled and other variables and routes each machine to the most efficient path. Of course, the AGV must travel along a predetermined path. This is in accordance with the ANSI safety standard, and compared to a car, it is like traveling only on large roads, not alleyways, for safety. On the contrary, AMR, which moves light items quickly, seems to move more diligently as it finds a road between it, like a bicycle going through an alley.

4) AGV is the most traditional collaborative robot.

AGVs have been operating in the workplace with people for decades. Probably one of the first'cooperative robots'. Since the AGV uses a relatively fixed guide path, it is possible to predict the movement. This means it can be safer. In addition, AGV guarantees safety according to ANSI safety standards. Furthermore, in recent years, various AGVs designed with human considerations have been introduced. A typical hybrid AGV is a hybrid AGV that follows the movement of people, not the path, and when the operator stops to select a product, the AGV also stops along with the product.

These AMR and AGVs have their own advantages and disadvantages. Whichever technology you use, you need to make sure that it suits your needs and chooses. You can simply compare AGV to a truck and an AMR to a bicycle. If the AGV carries a lot of heavier loads at once, AMR can move lighter loads using shortcuts. Therefore, in some cases AGV and in some cases AMR are the more suitable choices.

Incidentally, it can be referred to as Autonomous Mobile Robots (AMR) in terms of finding a target and avoiding obstacles by itself with the help of sensors such as riders, cameras, and GPS in factories or distribution centers.

However, in the related art, in order to know the current working state of the AMR, the operator has to go directly to the location where the AMR is located.

As described above, the space where AMR is generally used can be a large factory, etc.. Even if the AMR is actually visible from a distance, it is not possible to know the exact location of the AMR, as well as the current working status or operation status of the AMR. There is a problem that cannot be identified as.

Registered Utility Model No. 20-0389099

The present invention was conceived to solve the above-described conventional problem, and its object is a communication terminal capable of easily checking the work/operation state of an AMR using an augmented reality technique without getting close to the AMR, and It is to provide a control method.

In order to achieve the above object, a communication terminal providing an AR service using AMR spatial location information according to the present invention includes: a photographing camera; An estimating unit for estimating an initial position and an initial posture of the camera based on a photographed image of an Augmented Reality (AR) marker photographed by the photographing camera; An AMR information receiving unit for receiving location information of AMR (Autonomous Mobile Robot) from the server; A motion sensor that detects motion and generates motion information; Based on the location information of the AMR received by the AMR information receiving unit, the initial position and initial posture of the camera estimated by the estimating unit, and whether the AMR exists in the camera photographing area based on the motion information obtained through the motion sensor A determination unit that determines whether or not; And a display control unit for displaying a virtual object corresponding to the AMR using an augmented reality technique when the AMR is present in the camera photographing area as a result of the determination by the determination unit.

In addition, in order to achieve the above object, a control method of a communication terminal that provides an AR service using AMR spatial location information according to the present invention includes the steps of photographing an Augmented Reality (AR) marker using an equipped photographing camera. ; Estimating an initial position and an initial posture of the camera based on the captured image of the captured AR marker; Receiving location information of an Autonomous Mobile Robot (AMR) from a server; Based on the position information of the AMR received in the step, the initial position and initial posture of the camera estimated in the step, and whether the AMR exists in the camera photographing area based on the motion information acquired through the already equipped motion sensor. Determining; When the AMR is present in the camera photographing area as a result of the determination of the step, displaying a virtual object corresponding to the AMR using an augmented reality technique.

As described above, according to the present invention, even if the AMR device is not nearby, information related to the AMR device can be checked using an augmented reality technique using a user's communication terminal.

In particular, not only displaying virtual objects, but only when the AMR device enters the camera shooting area using position/position estimation using AR markers, virtual objects or various information corresponding to the AMR are displayed using an augmented reality technique. By expressing, it becomes possible to more accurately express information on the related AMR.

1 is a schematic configuration diagram of an entire system including a communication terminal according to an embodiment of the present invention,
2 is a functional block diagram of the communication terminal of FIG. 1;
3 is an example of an AR marker,
4 is a view showing a process of analyzing the correlation of the position / posture between the AR marker and the photographing camera (communication terminal),
5 is an example of an AMR map,
6 is a view showing a process of determining whether an AMR is included in a camera photographing area using position/position estimation and AMR position/position,
7 is a control flow diagram of a communication terminal according to an embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

Hereinafter, each embodiment according to the present invention is only one example for aiding understanding of the present invention, and the present invention is not limited to these embodiments. In particular, the present invention may be configured with a combination of at least one or more of individual configurations, individual functions, or individual steps included in each embodiment.

In particular, for convenience, some claims in the claims include alphabets such as'(a)', but these alphabets do not prescribe the order of each step.

A schematic configuration of the entire system including the communication terminal 100 according to an embodiment of the present invention is as shown in FIG. 1.

In the drawing, the AMR 200 (Autonomous Mobile Robot) is a device that performs a predetermined task while moving itself in a predetermined space (for example, in a factory).

The AMR 200 may transmit its own location information and/or posture information while continuously communicating with the server 300.

At this time, the AMR 200 may transmit its current location to the server 300 on a preset map (aka AMR 200 map), and for this purpose, various motion sensors 140 may be included in the AMR 200.

The motion sensor 140 may include an acceleration sensor, a gyro sensor, an angular velocity sensor, a gravity sensor, a tilt sensor, and the like.

That is, the AMR 200 can determine the current position and posture on the AMR 200 map by applying the determined motion using the motion sensor 140 from the preset initial position, and the determined own position Information and posture information (ie, information on which direction it is facing) can be transmitted to the server 300.

Determining a path that has been moved using the motion sensor 140 and determining a posture itself corresponds to a known technique, so a more detailed description thereof will be omitted.

Meanwhile, as described above, the server 300 may receive location information and posture information of each AMR 200 received from the AMR 200 as described above, as well as work progress information.

In addition, the information received from the AMR 200 may be transmitted to the communication terminal 100, and further, previously stored information corresponding to the AMR 200 may be extracted and transmitted to the communication terminal 100.

In addition, when there is a request from the communication terminal 100, a virtual object to be applied to the augmented reality (ie, a virtual object corresponding to the AMR 200) may be transmitted to the communication terminal 100.

Meanwhile, the communication terminal 100 determines its current position and posture by photographing and analyzing an externally attached or printed AR marker, and also after determining the current position and posture of the AMR 200 from the server 300 , A specific function is performed according to whether the AMR 200 is present in the photographing area.

An example of a specific functional block of the communication terminal 100 is as shown in FIG. 2.

As shown in the figure, the communication terminal 100 includes a photographing camera 110, an estimation unit 120, an AMR information receiving unit 130, a motion sensor 140, a motion sensor 150, a display control unit 160, and It may be configured to include a storage unit 170.

First, the storage unit 170 stores various data and applications required for the operation of the communication terminal 100, and further performs a function of storing information newly generated or received from the outside during the operation of the communication terminal 100. do.

The photographing camera 110 performs a function of photographing an external subject, and may include, for example, a lens and a charge coupled device (CCD), and the structure of the photographing camera 110 itself is known. Since it corresponds to the technology, a more detailed description will be omitted.

In particular, the photographing camera 110 may photograph a predetermined AR (Augmented Reality) marker as described later, and may continuously photograph the real world for augmented reality.

The estimating unit 120 performs a function of estimating an initial position and an initial posture of the camera based on a photographed image of an Augmented Reality (AR) marker photographed by the photographing camera 110.

An example of an AR marker is as shown in FIG. 3.

3(a) is an example of such an AR marker, and FIG. 3(b) is an example showing a process of recognizing such an AR marker by dividing it into white and black.

When AR marker information, which is information about the shape or location of each AR marker, is provided in the storage unit 170, the estimating unit 120 includes previously provided AR marker information and photographed by the photographing camera 110. The initial position and initial posture of the camera can be estimated by comparing the captured images of the AR marker.

Specifically, first, the estimating unit 120 may extract a marker identifier from a photographed image of a photographed AR marker.

This may correspond to a process of extracting information included in the barcode by recognizing the barcode.

Subsequently, the estimating unit 120 extracts position information corresponding to the extracted marker identifier from the already provided AR marker information.

For example, when each marker identifier and location information are matched to the AR marker information provided with the A device, the estimating unit 120 may check the location information matched to the specific marker identifier by using the corresponding AR marker information.

In addition, the estimating unit 120 may estimate the initial position and initial posture of the camera by comparing the shape information of the corresponding AR marker included in the AR marker information with the captured image of the captured AR marker. In this case, it goes without saying that the location information of the AR marker, which was previously identified, may be used.

For example, when the shape information of the previously stored AR marker and the photographed image of the actually photographed AR marker are compared, the distance and direction to the AR marker, and the tilt of the photographing camera 110 can be known.

That is, the photographed image may be photographed smaller according to the distance compared to the shape of the original AR marker, and a perspective method may be applied according to the photographing angle of the photographing camera 110 so that some regions may be distorted than the shape of the original AR marker. , Using this, it is possible to know the relative position and direction of the AR marker of the photographing camera 110, and the degree of inclination.

Fig. 4 is a diagram showing this example.

In FIG. 4, an example in which an AR marker placed on a plane is photographed by a photographing camera 110 provided in the communication terminal 100 from the top to the bottom, that is, in a downward diagonal direction.

When the correlation between the state of the photographing camera 110 and the AR marker is confirmed, the position and posture of the photographing camera 110 may be determined based on the location information of the AR marker extracted previously.

Of course, even if the shape information of the AR marker is not previously stored, a correlation between the photographing camera 110 and the photographed AR marker may be determined through analysis according to a previously stored algorithm.

That is, the role of the estimating unit 120 is to estimate the initial position and initial posture of the photographing camera 110 (this may mean the initial position and initial posture of the communication terminal 100) using the captured AR marker image. It is to perform the function that it does.

The location information corresponding to the above-described marker identifier may be location information on a previously provided AMR 200 map.

That is, the location information corresponding to the marker identifier is not absolute coordinates such as GPS coordinates, but may correspond to a relative position on a predetermined map (for example, the AMR 200 map).

For this, the AMR 200 map must be provided in advance, and the estimating unit 120 must know a reference point for determining the relative position information on the AMR 200 map.

An example in which the AMR 200 map and the paths to which the various AMRs 200 travel are shown in the corresponding AMR 200 is illustrated in FIG. 5.

The AMR information receiving unit 130 performs a function of receiving location information and/or posture information of the AMR 200 from the server 300.

As mentioned above, the server 300 may receive the location information and attitude information of the AMR 200 from the AMR 200, and provide the corresponding information according to the request of the communication terminal 100, AMR information The receiving unit 130 performs this function.

The motion sensor 140 performs a function of detecting a motion of the communication terminal 100. The motion sensor 140 provided in the communication terminal 100 may include various types of sensors such as the motion sensor 140 provided in the AMR 200 above.

The motion sensor 150 includes location information and/or posture information of the AMR 200 received by the AMR information receiving unit 130, the initial position and initial posture of the camera estimated by the estimating unit 120, and the motion sensor 140. ) Performs a function of determining whether the AMR 200 exists in the camera photographing area based on the motion information acquired through ).

For example, if you know the height of the photographing camera 110 from the ground, the position on the AMR 200 map, and the photographing direction, and also know the current position and posture of the AMR 200, the lens of the photographing camera 110 is By extending the direction and calculating, it is possible to determine whether the AMR 200 currently exists in the camera photographing area. Of course, in this case, it is assumed that information on the size of the AMR 200 is also known in advance or received from the server 300.

This process is illustrated in FIG. 6.

In FIG. 6, when an extension line passing through the viewport from the human eye is drawn, a distant plate can be formed, and it is possible to determine whether the AMR 200 exists in the plate area.

The display control unit 160 performs a function of displaying a virtual object corresponding to the AMR 200 in an augmented reality technique when the AMR 200 is present in the camera photographing area as a result of the determination of the motion sensor 150.

That is, in a state in which a real-world photographed image captured by the current photographing camera 110 is displayed (displayed on a display unit not shown), the display control unit 160 indicates that the AMR 200 is present in the camera photographing area. If it is determined by 150, the virtual object corresponding to the corresponding AMR 200 can be displayed using an augmented reality technique (ie, a virtual object is displayed together with a real-world image).

Here, the virtual object corresponding to the AMR 200 may be a pre-equipped 3D image corresponding to the AMR 200 or may include information corresponding to the AMR 200 received from the server 300.

For example, when information on the workload of the AMR 200 is received from the server 300, the display control unit 160 displays the virtual object corresponding to the AMR 200 using an augmented reality technique. It is also possible to display the workload of 200.

Hereinafter, the overall control flow of the communication terminal 100 according to an embodiment of the present invention will be described with reference to FIG. 7.

First, the communication terminal 100 photographs an AR marker according to a user's manipulation, and analyzes the captured AR marker image.

Through this image analysis, the communication terminal 100 checks the location of the AR marker.

At this time, information on each AR marker (for example, location information on the AMR 200 map to which the AR marker is attached) and identification information obtained from the AR marker may be compared to determine location information corresponding to the captured AR marker. .

Subsequently, the communication terminal 100 checks the position and posture of the photographing camera 110 through image analysis on the AR marker.

The communication terminal 100 may estimate the location and posture of the current capturing camera 110 (that is, the communication terminal 100) by using the position of the AR marker and the position and posture of the photographing camera 110 as identified above.

Thereafter, the communication terminal 100 may continuously track the position or posture of the communication terminal 100 by the motion sensor 140 already provided.

Meanwhile, the communication terminal 100 requests and receives AMR information corresponding to the AR marker from the server 300.

Here, the AMR information received from the server 300 may include location information and status information of the AMR 200.

Subsequently, the communication terminal 100 includes the AMR 200 in the photographing area of the photographing camera 110 based on the position and posture of the communication terminal 100 and the position information of the AMR 200 received from the server 300. Determine whether it is in.

As a result of the determination, when the AMR 200 exists in the photographing area of the communication terminal 100, a virtual object corresponding to the AMR 200 is displayed using an augmented reality technique.

For example, a virtual 3D object formed similar to the shape of the AMR 200 may be displayed, and further, the current work progress of the AMR 200 may be displayed.

Meanwhile, it goes without saying that the process of performing each of the above-described embodiments may be performed by a program or application stored in a predetermined recording medium (eg, computer-readable). Here, the recording medium includes all of an electronic recording medium such as a random access memory (RAM), a magnetic recording medium such as a hard disk, and an optical recording medium such as a compact disk (CD).

At this time, the program stored in the recording medium may be executed on hardware such as a computer or a smart phone to perform each of the above-described embodiments. In particular, at least one of the functional blocks of the communication terminal 100 according to the present invention described above may be implemented by such a program or application.

In addition, the present invention is not limited to the specific embodiments described above, but can be implemented by various modifications and modifications without departing from the gist of the present invention. It will be apparent that such modifications and modifications are included in the present invention if they fall within the appended claims.

100: communication terminal 200: AMR
300: server 110: shooting camera
120: estimation unit 130: AMR information receiving unit
140: motion sensor 150: determination unit
160: display control unit 170: storage unit

Claims (14)

In a control method performed by a communication terminal providing an AR service,
(a) photographing an Augmented Reality (AR) marker using the provided photographing camera;
(b) estimating the initial position and initial posture of the camera based on the captured image of the captured AR marker;
(c) receiving location information of AMR (Autonomous Mobile Robot) from the server;
(d) the position information of the AMR received in step (c), the initial position and initial posture of the camera estimated in step (b), and the own motion information obtained by itself through a previously equipped motion sensor. Determining whether the AMR exists in a camera photographing area based on the present invention;
(e) If the AMR is present in the camera photographing area as a result of the determination in step (d), using the AMR spatial location information, characterized in that it comprises the step of displaying a virtual object corresponding to the AMR using an augmented reality technique. A control method of a communication terminal providing AR service. The method of claim 1,
In the step (b), the AR service is provided using AMR spatial location information, characterized in that the initial position and initial posture of the camera are estimated by comparing the previously provided AR marker information with the captured image of the captured AR marker. The control method of the communication terminal. The method of claim 2,
In the step (b), a marker identifier is extracted from the photographed image of the captured AR marker, location information corresponding to the extracted marker identifier is extracted from the already provided AR marker information, and the corresponding information included in the AR marker information A control method of a communication terminal providing an AR service using AMR spatial location information, characterized in that the initial position and initial posture of the camera are estimated by comparing shape information of the AR marker with the captured image of the captured AR marker. The method of claim 3,
The location information corresponding to the marker identifier is location information on an already provided AMR map. A control method of a communication terminal providing an AR service using AMR spatial location information. The method of claim 4,
The virtual object corresponding to the AMR in step (e) utilizes AMR spatial location information, characterized in that it includes a 3D image corresponding to the AMR or information corresponding to the AMR received from the server. Thus, a control method of a communication terminal providing an AR service. The method of claim 1,
In the step (c), location information and posture information of AMR (Autonomous Mobile Robot) are received from the server,
In step (d), the position information and posture information of the AMR received in step (c), the initial position and initial posture of the camera estimated in step (b), and acquired through a motion sensor A control method of a communication terminal that provides an AR service using AMR spatial location information to determine whether the AMR exists in a camera photographing area based on motion information. A computer-readable recording medium storing a program for executing the method of any one of claims 1 to 6. An application program stored in a computer-readable recording medium to execute the method of claim 1 in combination with hardware. A shooting camera;
An estimating unit for estimating an initial position and an initial posture of the camera based on a photographed image of an Augmented Reality (AR) marker photographed by the photographing camera;
An AMR information receiver for receiving location information of an AMR (Autonomous Mobile Robot) from the server;
A motion sensor that detects motion and generates motion information;
Based on the location information of the AMR received by the AMR information receiving unit, the initial position and initial posture of the camera estimated by the estimating unit, and whether the AMR is present in the camera photographing area based on the motion information obtained through the motion sensor A determination unit that determines whether or not;
Providing AR service using AMR spatial location information, characterized in that it comprises a display control unit that displays a virtual object corresponding to the AMR using an augmented reality technique when the AMR exists in the camera photographing area as a result of the determination by the determination unit. Communication terminal. The method of claim 9,
The estimating unit provides an AR service using AMR spatial location information, characterized in that the initial position and initial posture of the camera are estimated by comparing the already provided AR marker information with the captured image of the AR marker captured by the photographing camera. Communication terminal. The method of claim 10,
The estimating unit extracts a marker identifier from the photographed image of the captured AR marker, extracts location information corresponding to the extracted marker identifier from already provided AR marker information, and the shape of the corresponding AR marker included in the AR marker information A communication terminal providing an AR service using AMR spatial location information, characterized in that the initial position and initial posture of the camera are estimated by comparing the information with the captured image of the captured AR marker. The method of claim 11,
The location information corresponding to the marker identifier is location information on an already provided AMR map. A communication terminal that provides an AR service using AMR spatial location information. The method of claim 12,
The virtual object corresponding to the AMR includes a 3D image corresponding to the AMR or information corresponding to the AMR received from the server, and provides AR service using AMR spatial location information. terminal. The method of claim 9,
The AMR information receiving unit receives location information and posture information of AMR (Autonomous Mobile Robot) from the server,
The determination unit is based on the position information and posture information of the AMR received by the AMR information receiving unit, the initial position and initial posture of the camera estimated by the estimating unit, and the motion information obtained through a previously equipped motion sensor. A communication terminal that provides an AR service by using AMR spatial location information to determine whether the camera exists within the shooting area.

Images