KR20220112442A - Augmented reality-based logistics management vision picking system and method using wearable smart glasses, a hands-free mobile computing technology - Google Patents

Abstract

A logistics management system of the present invention, in order to manage a history of the production or distribution of a device or article of a subject to be managed, includes: an integrated management server managing an AR code for identifying a corresponding device or article; a communication part communicating with the integrated management server; a camera part capable of photographing the AR code; a display control part for adding augmented reality to the image photographed by the camera part; a display part displaying the image to which augmented reality has been added by the display control part or the image photographed by the camera part; and a mobile communication device provided with an augmented reality control part performing a control operation such that a relevant information image can be displayed in augmented reality on the display part by using the information delivered through the communication part. The mobile communication device also includes a location information module for measuring location information. The augmented reality control part of the mobile communication device transmits the location information along with identification information of the photographed AR code to the integrated management server. The integrated management server delivers relevant information corresponding to the delivered identification information of the AR code along with additional information related with the location information of the mobile communication device to the mobile communication device. Therefore, the present invention is capable of easily providing additional information by using posture information and position information of a mobile communication device.

Description

Augmented reality-based logistics management vision picking system and method using wearable smart glasses, a hands-free mobile computing technology

The present invention relates to a logistics management system using augmented reality, and relates to a logistics management system that can be used for multiple purposes, such as for facility maintenance and for managing and confirming distribution history of agricultural and marine products. In particular, it relates to a system capable of more accurately performing facility management and distribution management by using information on location and direction movement of a mobile communication device through a mobile communication device.

Unless otherwise indicated herein, the contents described in this identification are not prior art to the claims of this application, and the description in this identification is not admitted to be prior art.

A service technology called Augment Reality (AR) is being spread. Augmented reality is a technology for adding and presenting information to a real environment using a computer.

In the augmented reality service, when a virtual object, ie, an AR object, is arranged in three-dimensional coordinates of a three-dimensional space, an AR object or display object displayed on the screen is also displayed. AR objects are displayed large or conversely displayed small according to the relative distance from the user. In this way, the sense of principal is expressed, and the user can receive a deep feeling, and the AR object feels as if it exists in reality.

However, in the case of such an AR system, an AR code displayed on an object for which information is to be checked is photographed, and a corresponding AR object is simply displayed from the AR code included in the photographed image. In the case of using augmented reality to remotely control a device, more functions will have to be implemented.

The present invention is to provide a system for easily managing and displaying additional information to a user even in an environment where an AR code is not attached, while easily providing related information on a device or article to be managed in various environments. do.

The logistics management system of the present invention includes an integrated management server that manages an AR code for identifying a corresponding device or article in order to manage a history of production or distribution of a device or article to be managed, and communication with the integrated management server A communication unit for performing the above, a camera unit capable of photographing the AR code, a display control unit for adding augmented reality to an image photographed by the camera unit, and an image to which augmented reality is added by the display control unit or the camera unit A mobile communication device having a display unit for displaying an image taken by the communicator, and an augmented reality control unit for controlling to display a related information image in augmented reality on the display unit using information transmitted through the communication unit, wherein the The mobile communication device further includes a location information module for measuring location information, and the augmented reality control unit of the mobile communication device transmits the location information together with the identification information of the AR code to be photographed to the integrated management server,

The integrated management server may transmit related information corresponding to the transmitted identification information of the AR code and additional information related to the location information of the mobile communication device to the mobile communication device together.

According to the logistics management system using the augmented reality of the present invention as proposed, there is an advantage in that additional information can be easily provided by using the location information and posture information of the mobile communication device along with the recognition of the AR code.

1 is a view for showing device management by the integrated management server of the present invention.
Figure 2 is a diagram showing the configuration of the logistics management system of the present invention.
3 is a diagram showing the configuration of a mobile communication device constituting the logistics management system using the augmented reality of the present invention.
4 is a diagram illustrating an example displayed on a screen when performing remote control as well as checking information of a managed device using an AR code according to the present invention.
5 and 6 are diagrams for explaining a method of logistics management using location information and posture information of a mobile communication device according to the present invention.
7 is a diagram illustrating an example of showing related information in augmented reality when an AR code attached to a managed device is photographed according to the present invention.
8 and 9 are diagrams showing another embodiment of the logistics management system using augmented reality according to the present invention.
10 and 11 are diagrams showing an example of recognizing a plurality of AR codes and providing information thereto using the logistics management system using augmented reality of the present invention.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "part" for components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves. In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical spirit disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Terms including ordinal numbers such as first, second, etc. may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprises” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

1 is a diagram showing device management by the integrated management server of the present invention. Referring to FIG. 1 , a specific area 9 is, for example, a building or a factory such as an office building or public facilities, and the same floor of the building is shown as an example.

A predetermined building is divided into a plurality of rooms, and management target devices (1,2,3,4,5,6) are located in each room, and in each of the management target devices, a user provides information on the device AR code is attached to make it easy to acquire or remotely control the device.

A user can acquire information on each managed device while carrying a mobile communication device such as a smartphone or tablet PC, and remotely control each managed device using the network-connectable integrated management server 10 . It is possible to control

That is, the mobile communication device possessed by the user displays the augmented reality provided by the integrated management server 10 on the screen, and enables the user to easily obtain information on each managed device, or Remote control of each managed device is possible using the device and the integrated management server.

In particular, the integrated management server 10 checks a user's location using a location information module configured in a mobile communication device, and checks a user's location/posture change using a gyro sensor or an acceleration sensor of the mobile communication device. It precisely specifies the managed device to be checked, or provides more accurately the information the user wants to check.

Figure 2 is a diagram showing the configuration of the logistics management system of the present invention.

Referring to FIG. 2 , the system of the present invention produces an AR code and uses the AR code, for example, a distribution sensor 60 , a producer 70 and a wholesaler 80 to the integrated management server 10 . It includes an AR platform server 50 that mediates access to, and an integrated management server 10 that stores and manages the AR codes produced by the AR platform server 50 and data stored in advance for each AR code.

In addition, the mobile communication device 10 that uses the AR code to check information on goods such as agricultural and fishery products or to check information on equipment, etc., accesses the AR platform server 50 or the integrated management server 10 ) can be accessed directly.

Using the logistics management system of the present invention, when the producer 70 registers and generates an AR code using the AR platform server 50, and distributes goods using the registered AR code, the wholesaler 80 ) or by registering additional information in the corresponding AR code in the distribution sensor 60, the consumer may be provided with product history information, distribution process information, etc. in augmented reality using the AR code.

In addition, according to the present invention, by using the location information and posture information of the mobile communication device that captures the AR code, the equipment or product to be checked is accurately specified, and information on a plurality of AR codes and information are provided. to make this useful.

A mobile communication device constituting the augmented reality system of the present invention will be first described.

3 is a diagram showing the configuration of a mobile communication device constituting the logistics management system using the augmented reality of the present invention.

Referring to FIG. 3 , the mobile communication device 100 of the present invention captures an AR code, acquires information on the AR code from the integrated management server 10 through a network, and displays it through the display unit 180 .

In detail, the mobile communication device 100, such as a smart phone or tablet PC, includes a camera unit 130 for photographing an AR code attached to a product such as agricultural or fishery products or an equipment subject to management or remote control, and the mobile communication device. A location information module 120 for acquiring location information, a gyro sensor 140 for detecting an angular motion (angular velocity) of a mobile communication device and calculating an axis-based conversion value, and an external device using mobile communication or short-distance communication A communication unit 160 that enables communication with a server, and a display control unit 170 that performs image processing to be displayed through the display unit 180 by adding augmented reality to the image captured by the camera unit 130 and , the augmented reality controller 110 for controlling each configuration so that the AR code is recognized by using the values measured from the respective sensors or information or images according to the AR code are displayed on the display unit 180 .

In addition, the mobile communication device 100 is an acceleration sensor for detecting the amount of change (acceleration) of the moving speed of the mobile communication device, and the object for displaying the AR code or augmented reality recognized by the camera unit 130 It may further include a memory unit 190 in which the like is stored.

According to the present invention, the augmented reality control unit 110 determines whether an AR code is included in the image captured through the camera unit 130 , and when the AR code is read, information corresponding to the AR code In order to obtain , communication with the integrated management server 10 which is an external server is performed through the communication unit 160 .

In addition, the augmented reality control unit 110 receives augmented reality object data to be displayed through the display unit 180 as augmented reality from the integrated management server 10 or augmented reality stored in advance in the memory unit 190 . A real object is displayed on the display unit 180 .

In addition, the augmented reality controller 110 displays a plurality of augmented reality objects on the display unit 180 in the same manner as above even when there are a plurality of AR codes in the image captured by the camera unit 130 .

In addition, in the present invention, a target article or a target device that the user wants to check is more accurately determined by using the location information of the mobile communication device measured by the location information module 120 . That is, the integrated management server 10 manages the location information for each AR code of the management target, and the integrated management server 10 together with the AR code identified in the mobile communication device, the image or You can check the location information recorded in the video. Through this process, the integrated management server 10 may determine together whether the AR code photographed by the mobile communication device 100 is located in a location where the AR code is stored and managed in advance.

And, in the present invention, the recognition/confirmation of the second AR code may be guided to the user after the first AR code is recognized using the gyro sensor 140 of the mobile communication device 100 . A multi-axis gyro sensor may be used as the gyro sensor, and when the user's movement (movement of the mobile communication device), that is, the inclination change is made, the gyro sensor detects the angular motion (angular velocity) of the mobile communication device and X corresponding to the degree of inclination. Outputs the converted value of the axis/Y axis/Z axis.

Therefore, when the spirit of the present invention is implemented for the purpose of managing a plurality of devices, it is possible to guide the positions of the devices to be sequentially performed according to the posture of the operator (user). That is, after the first AR code is photographed to inspect the first managed device, the direction in which the next second managed device is located can be displayed through the display unit 180 of the mobile communication device 100 . In addition, using the gyro sensor, it is also possible to monitor whether the inclination change of the mobile communication device is being performed correctly.

In addition, in the present invention, sequential management and supervision of management target devices using the movement speed of the user (mobile communication device) using the acceleration sensor of the mobile communication device 100 and location information by the location information module We can guide you to make this happen.

These embodiments of the present invention will be described in more detail with the accompanying drawings.

4 is a diagram illustrating an example displayed on a screen when performing remote control as well as checking information of a managed device using an AR code according to the present invention.

First, in order to remotely control as well as when checking information (production information, distribution information, etc.) of a device or product using augmented reality according to the present invention, each management target device is transmitted through short-distance communication or network connection Bidirectional communication can be made with the integrated management server 10 (refer to FIG. 1).

In this case, when the user shoots the first AR code attached to the first managed device 1 using the mobile communication device 100 , the information on the first managed device 1 is converted into augmented reality. A menu 101 for checking and a menu 102 for remotely controlling the operation of the first management target device 1 using a mobile communication device may be displayed on the display unit 180 .

For example, when the user selects the augmented reality information menu 101, the mobile communication device 100 receives information corresponding to the AR code from the integrated management server 10, and the camera unit 130 ), information on the first management target device 1 is displayed in the image captured by the augmented reality. Related information such as production information of the device, operation switch information of the device, management information, and precautions may be displayed.

And, when the user selects the remote control menu 102 , the mobile communication device 100 receives information for remotely controlling the corresponding management target device from the integrated management server 10 . For example, a screen for selecting a switch position of a device, a function description for each switch, and an operation (on/off, etc.) of the device may be displayed on the screen on which the management target device is photographed as augmented reality. In this case, a command for operation control of the management target device input from the mobile communication device 100 is transmitted to the integrated management server 10 , and the integrated management server 10 controls the communication of the connected management target device. Controls actions according to user selection.

5 is a diagram for explaining a method of managing distribution using location information and posture information of a mobile communication device according to the present invention.

First, the management target devices stored in the integrated management server 10 are managed in the order of the first management target device 1 , the second management target device 2 , and the third management target device 3 , and Assume that it is stored as something to be verified.

At this time, the user 20 uses the mobile communication device 100 to photograph the first AR code attached to the first managed device 1 , and check information and current status of the first managed device. do.

If the user 20 skips the second management target device 2 and moves toward the third management target device 3 , as shown in FIG. 6 , the display unit of the mobile communication device 100 At 180, a message 103 of out of order is displayed.

In detail, when the first AR code is recognized by the mobile communication device 100 and device information corresponding to the first AR code is transmitted from the integrated management server 10 to the mobile communication device 100, the The integrated management server 10 checks the location of the mobile communication device 100 and the posture/inclination information. That is, the integrated management server 10 prepares for information transfer corresponding to the second code in the next order after the first AR code is checked, and in the meantime, the movement location of the mobile communication device 100 is checked, Check posture/tilt information.

If it is determined that the position and inclination information of the mobile communication device measured by the location information module or the gyro sensor configured in the mobile communication device 100 does not correspond to the previously stored values, as illustrated in FIG. 6 , A warning message is displayed on the screen of the mobile communication device.

Through this method, the operator (user) guides to check all the target devices to be managed without omission, and the user checks the information about the device displayed on the display unit of the mobile communication device in augmented reality or remotely controls it can do.

On the other hand, in order for the integrated management server 10 to more accurately determine the location information of the mobile communication device 100 , the memory unit 190 of the mobile communication device 100 contains map information where each management target device is located. can be stored.

In this case, when the first AR code information is transmitted from the mobile communication device, the integrated management server 10 transmits, together with the first AR code information, location information of the previously designated management target devices to the mobile communication device. can transmit In addition, the mobile communication device 100 stores the received related information (eg, map data) in the memory unit 190 while displaying the information on the first AR code on the screen in augmented reality. .

In addition, the mobile communication device 100 guides the user to move to the next management target device using the map data stored in the memory unit 190 and the location information identified by the location information module 120 by itself. can do.

7 is a diagram illustrating an example of showing related information in augmented reality when an AR code attached to a managed device is photographed according to the present invention.

When the AR code attached to the managed device is photographed using the mobile communication device 100 , as described above, information on the target device may be displayed in augmented reality or it may be selected to remotely control the target device. An example of providing relevant information with such augmented reality is shown.

For example, the first image 701 and the second image 701 corresponding to the first switch are displayed on the screen so that the user can easily know the positions of the first switch and the second switch.

Then, each of the images 701 and 792, that is, the description messages 710 and 720 for the first and second switches, are sequentially or simultaneously displayed as an augmented reality image on the screen of the mobile communication device.

Through this method, the user can easily check the information of the managed device. The system of the present invention can be applied so that consumers can easily check the production process and distribution process, such as agricultural and marine products. For example, when a consumer takes an AR code attached to an apple box using a mobile communication device, a menu to check production history information about the apple and a menu to check information about distribution history may be presented. In the production process, the image of the cultivation process registered by the producer is displayed, or the image of the distribution process registered by the distributor in the distribution process is displayed on the screen.

At this time, the integrated management server 10 communicating with the mobile communication device may use the location information measured by the location information module of the mobile communication device to accurately determine the user's location, and the integrated management server 10 may further provide the location of the fruits produced by the producer of the corresponding apple based on the user's current location as additional information.

As described above, in the present invention, there is a feature that not only provides information corresponding to AR code recognition, but additionally related information can be additionally provided using the user's location information and the user's posture (tilt) information. .

8 and 9 are diagrams showing another embodiment of the logistics management system using augmented reality according to the present invention.

As shown in FIG. 8 , when a plurality of pipes are disposed in a facility space such as a factory, an AR code 810 may be attached to a specific pipe 801 such as a pipe to be managed.

And, when a pipe/equipment to be additionally managed occurs in a company or factory that manages the pipes, even if an AR code is not necessarily generated and attached, by additional registration in the integrated management server 10 It may be shown to the user in augmented reality.

That is, according to an embodiment of the present invention, when an AR code 810 is produced to manage the first pipe 801 and registered in the integrated management server 10, a user (administrator) can use the AR code ( 810), along with displaying information 901 about the first pipe 801 as augmented reality, and information 902 about the pipe 920 added to management can be displayed together in augmented reality. (Fig. 9).

The integrated management server 10 checks the location information of the mobile communication device 100 together with the information corresponding to the AR code photographed by the mobile communication device 100 and together when the user is located at the corresponding location. More information to be displayed may be stored. Through this method, information to be displayed together with location information is stored in the integrated management server 10 when a device to be managed is remotely added, so that even if a separate AR code is not attached, additional You can provide more information to be displayed as

In this way, since the user's location information and posture information are used, the integrated management server 10 can check whether the management target device is continuously located at a predetermined location, along with information to be additionally provided at the location. have.

10 and 11 are diagrams illustrating an example of recognizing a plurality of AR codes and providing information thereto using the logistics management system using augmented reality of the present invention.

According to the system of the present invention, when a plurality of AR codes are identified in the image captured by the user's mobile communication device, information on the captured AR code is received through the integrated management server 10, Information on items or devices other than the AR code captured can also be acquired.

That is, even if information on a device or article to be managed is additionally registered in the integrated management server 10 through the AR platform server 50, and an AR code corresponding to the device or article is not attached, it is pre-attached. It is possible to provide information on additionally registered devices or articles when the AR code is taken.

As shown in FIG. 10 , when managing a plurality of files, that is, a first AR code 1110 is attached to a first file folder 1101 to be managed, and a second AR code 1110 is attached to a second file folder 1102 . Take the case where (1120) is attached as an example.

In this case, when the user shoots the first AR code 1110 using the camera of the mobile communication device 100 , identification information corresponding to the first AR code 1110 is transmitted to the integrated management server 10 . And, the mobile communication device 100 displays the information transmitted from the integrated management server 10 on the screen as an augmented reality image synthesized.

Also, even when the user shoots both the first AR code 1110 and the second AR code 1120 using the camera of the mobile communication device 100, identification corresponding to the first AR code and the second AR code Information is transmitted to the integrated management server 10 , and the mobile communication device 100 receives information previously stored in relation to the first and second AR codes and displays them on the screen in augmented reality.

In addition, according to this embodiment, using the integrated management server 10 and the AR platform server 50, it is possible to additionally register the third AR code and information corresponding to the third AR code, Even when the third AR code image is not captured by the mobile communication device 100 , information corresponding to the third AR code is displayed in augmented reality.

For example, information to be additionally displayed together with information about the first AR code 1110 and the second AR code 1120 , for example, disposed between the first file folder 1101 and the second file folder 1102 . This is a case where information on the third file folder 1103 that should be added is additionally registered, and the third AR code indicating the information on the third file folder 1103 is not attached to the third file folder 1103 .

At this time, the integrated management server 10 sets the location where the third AR code is to be attached to the space between the first AR code 1110 and the second AR code 1120 , and sets the first AR code by the user. When the AR code 1110 and the second AR code 1120 are photographed together, the integrated management server 10 provides corresponding information identified from the first AR code 1110 and the second AR code 1120 . together with the first AR code and the second AR code, information identified from the third virtual AR code to be attached (information about the third file folder) is transmitted to the mobile communication device 100 .

That is, as shown in FIG. 11 , on the screen of the mobile communication device 100 , the first related information image 1210 identified by the first AR code 1110 and the second AR code 1120 are displayed on the screen. The identified second related information image 1220 is displayed as an augmented reality image in the captured image. And, the third related information image 1230 stored as to be displayed between the first related information image and the second related information image is further displayed in augmented reality on the screen of the mobile communication device.

In addition, the augmented reality controller 110 of the mobile communication device 100 determines the size of the area in which the third related information image 1230 is displayed is the distance between the first AR code 1110 and the second AR code 1120 . In addition, the distance between the first related information image 1210 and the second related information image 1220 is calculated. Then, while the third related information image 1230 is positioned between the first AR code 1110 and the second AR code 1120, the first related information image 1210 and the second related information image 1220 control to be placed between them.

Through this process, in the case of managing an item or device, additional information input on the item or device to be managed and easy confirmation by the user may be made.

Then, the user can check whether the arrangement of the item or device to be managed is arranged according to the positional relationship stored in the integrated management server 10 , and the display area of the third related information image 1230 is abnormal. , it may be determined that the arrangement of the first to third file folders 1101, 1102, and 1103 is incorrect.

According to the embodiments of the present invention as proposed, logistics management can be more efficiently performed using augmented reality, and additional input of information on a device or article to be managed can be performed using an AR platform server or an integrated management server. Not only is it easy, but the user can easily obtain information about the item or device added to the server.

Claims (1)

An integrated management server that manages an AR code for identifying a corresponding device or product in order to manage a history of producing or distributing a device or product to be managed, a communication unit communicating with the integrated management server, and the AR A camera unit capable of taking a code, a display control unit for adding augmented reality to the image photographed by the camera unit, and an image to which augmented reality is added by the display control unit or an image photographed by the camera unit is displayed A mobile communication device having a display unit and an augmented reality control unit for controlling to display a related information image in augmented reality on the display unit using information transmitted through the communication unit,
The mobile communication device further includes a location information module for measuring location information, and the augmented reality control unit of the mobile communication device transmits the location information together with the identification information of the AR code to be photographed to the integrated management server,
Logistics management system using augmented reality, characterized in that the integrated management server transmits the related information corresponding to the transmitted identification information of the AR code and additional information related to the location information of the mobile communication device together to the mobile communication device .

Images