KR102414575B1 - Method and system recognizing marine transportation product using of augmented reality - Google Patents

Abstract

The present invention relates to a method and system for recognizing maritime transportation products using augmented reality, wherein the user terminal acquires an image by driving a photographing unit when a transportation product confirmation application is executed, wherein the user terminal coordinates the obtained image calculating and transmitting a transportation product information request signal including the calculated image coordinates to an ordering company system, wherein the ordering company system inquires a vessel located in the image coordinates, and ordering information of the inquired vessel or Transmitting the transport product information including delivery information to the user terminal, the user terminal may include the step of overlaying the transport product information on the acquired image to display on a screen.

Description

Marine transportation product recognition method and system using augmented reality {METHOD AND SYSTEM RECOGNIZING MARINE TRANSPORTATION PRODUCT USING OF AUGMENTED REALITY}

The present invention relates to a method and system for recognizing maritime transportation products using augmented reality, and more particularly, augmented display of transportation product information of a ship located in the coordinate range of an image obtained through a photographing unit on a screen using augmented reality. It relates to a method and system for recognizing marine transportation products using reality.

Augmented Reality (增强現實: Augmented Reality) creates a new environment by combining the real world that users see with their eyes and the virtual world with additional information. Augmented reality systems that converge real and virtual environments have been researched and developed mainly in the United States and Japan since the late 1990s. Augmented reality, a concept that complements the real world with a virtual world, uses a virtual environment created with computer graphics, but the main character is the real environment. Computer graphics serve to provide additional information necessary for the real environment.

On the other hand, large shipyards manufacture some blocks outside the company considering various factors such as production resource requirements and production cost, but blocks are transported by sea because land transportation is impossible. The transported blocks wait at a place called an anchorage before unloading, and it is very difficult to visually check the blocks anchored at the anchorage in an onshore workshop.

In addition, the unloading manager must accurately understand the anchorage situation to perform necessary tasks such as adjusting the unloading order, so it is required to develop a technology that can accurately recognize marine products anchored in the sea.

Prior art 1: Korean Patent Publication No. 2011-0116842 (published on October 26, 2011)

The present invention has been devised to solve the above problems, and by using augmented reality technology to overlay and display transport product information mounted on a ship anchored in the sea on the ship image, it is possible to accurately recognize marine transport products. An object of the present invention is to provide a method and system for recognizing maritime transportation products using augmented reality.

On the other hand, the technical problems to be achieved by the present invention are not limited to the technical problems mentioned above, and various technical problems may be included within the range apparent to those skilled in the art from the content to be described below.

According to one aspect of the present invention for solving the above problems, the user terminal obtains an image by driving a photographing unit when the transportation product confirmation application is executed, the user terminal calculates the coordinates of the obtained image and , transmitting a transportation product information request signal including the calculated image coordinates to an ordering company system, wherein the ordering company system inquires a ship located in the image coordinates, and retrieves the inquired ship's order information or delivery information There is provided a method for recognizing maritime transport products using augmented reality, comprising the steps of: transmitting the transport product information including the user terminal to the user terminal, the user terminal overlaying the transport product information on the acquired image to display on a screen .

The user terminal calculates the coordinates of the obtained image, and the step of transmitting a transportation product information request signal including the calculated image coordinates to the ordering company system includes: location information, direction information and posture information of the user terminal Measuring, calculating the coordinates of the acquired image using the measured location information, direction information, posture information and user terminal information, and sending a transportation product information request signal including the calculated image coordinates to the ordering company system It may include the step of transmitting to

According to another aspect of the present invention, in a method for a user terminal to recognize a marine transportation product using augmented reality, when a transportation quantity check application is executed, driving a photographing unit to acquire an image; Calculating coordinates, transmitting a transport product information request signal including the calculated image coordinates to an ordering company system, receiving transport product information of a ship located in the image coordinates from the ordering company system, the There is provided a shipping product recognition method using augmented reality comprising the step of overlaying the received transport product information on the acquired image to display on the screen.

Calculating the coordinates of the obtained image may include measuring location information, direction information and posture information of a user terminal, and the obtained image using the measured location information, direction information, posture information and user terminal information. It may include calculating the coordinates of .

According to another aspect of the present invention, in the method for the ordering company system to provide a maritime transportation product recognition service using augmented reality, when a transportation product information request signal including image coordinates is received from a user terminal, the image A method of recognizing maritime transportation products using augmented reality is provided, comprising the steps of inquiring a ship belonging to the coordinates, acquiring transportation product information including order information or delivery information of the inquired ship and transmitting the information to the user terminal. .

The maritime transportation product recognition method using the augmented reality receives AIS information from an AIS (Automatic Identification System) terminal installed on a ship, analyzes the received AIS information, and grasps the ship identification information and the current location information of the ship, The method may further include storing the identified vessel identification information and current location information together with the AIS information reception time.

In addition, the maritime transportation product recognition method using the augmented reality comprises the steps of receiving and storing delivery information including at least one of an order item, a ship in charge of transport, ship identification information, an expected arrival date, a captain's name, and a ship phone number from a supplier may further include.

According to another aspect of the present invention, an AIS terminal that is provided on a ship and transmits the current location information of the ship, receives and stores the current location information of the ship from the AIS terminal, and includes an image coordinate from a user terminal. When an information request signal is received, an ordering company system that inquires a ship belonging to the image coordinates and provides transportation product information including order/delivery information of the inquired ship to the user terminal, a transportation product confirmation application is executed In this case, the photographing unit is driven to acquire an image, the coordinates of the acquired image are calculated, and a transport product information request signal including the calculated image coordinates is transmitted to the ordering company system and transport product information is received, There is provided a marine transportation product recognition system using augmented reality including a user terminal for overlaying the transportation product information on the acquired image.

The maritime transportation product recognition system using the augmented reality receives delivery information including at least one of an order item, a ship in charge of transportation, ship identification information, an expected arrival date, a captain's name, and a ship phone number, and registers it in the ordering company system. It may further include a vendor terminal.

According to another aspect of the present invention, a transport ship information database in which the location information of transport ships is stored, a transport product information database in which order information and delivery information for ordered items are stored, and a transport product information request signal including image coordinates from a user terminal is received, inquires a ship belonging to the image coordinates from the transport ship information database, obtains transport product information including order information and delivery information of the inquired ship from the transport product information database, and sends it to the user terminal An ordering company system including a transport product information providing unit to be transmitted is provided.

The ordering company system includes an AIS receiver that receives AIS information from an AIS terminal installed in each ship, analyzes the received AIS information to determine ship identification information and current location information of the ship, and identifies the identified ship identification information and current location It may further include an AIS analysis unit for storing information in the transport vessel information database together with the AIS information reception time.

The transport product information database is an order information database in which order information including at least one of ordered items, outside company name, contract price, contract date, and stocking request date is stored, order item, transport ship, ship identification information, expected arrival date, captain's name , and may include a delivery information database in which delivery information including at least one of a ship phone number is stored.

The ordering company system may further include an ordering information registration unit that receives ordering information from the ordering company and registers it in the ordering information database, and a delivery information registration unit that receives delivery information from the vendor and registers it in the delivery information database.

According to another aspect of the present invention, a photographing unit for capturing an image, a measuring unit for measuring location information, direction information and posture information, the measured location information, direction information and posture information, and device information An image coordinate calculation unit for calculating the coordinates of the image, a transportation product information request processing unit for requesting and receiving transportation product information of a ship located in the calculated image coordinates, and overlaying the transportation product information on the captured image to display on the screen A user terminal including an augmented reality processing unit is provided.

The measuring unit includes a position measuring unit measuring a current position of the user terminal, a direction information measuring unit measuring direction information including yaw angle information of the user terminal, roll angle information of the user terminal and It may include a posture information measuring unit for measuring posture information including pitch angle information.

The device information may include at least one of angle of view information, focal length information, enlargement information, and reduction information.

According to the present invention, by using the augmented reality technology to display the information of the transportation products mounted on the ship anchored in the sea overlaid on the image of the ship, it is possible to accurately recognize the marine transportation products.

On the other hand, the effects of the present invention are not limited to the above-mentioned effects, and various effects may be included within the range apparent to those skilled in the art from the description below.

1 is a view showing a marine transportation product recognition system using augmented reality according to an embodiment of the present invention.
2 is a block diagram schematically showing the configuration of an ordering company system according to an embodiment of the present invention.
3 is a block diagram schematically showing the configuration of a user terminal according to an embodiment of the present invention.
FIG. 4 is a diagram specifically illustrating the configuration of the measurement unit shown in FIG. 3 .
5 to 8 are for describing in detail the process in which the image coordinate calculator calculates the three-dimensional spatial coordinates of the vertices of the captured video image using the location information, direction information, posture information, and device information of the user terminal. It is a drawing.
9 is a diagram illustrating a method of recognizing a marine transportation product using augmented reality according to an embodiment of the present invention.
10 to 12 are exemplary screen views for explaining a method of recognizing a marine transportation product using augmented reality according to an embodiment of the present invention.

The above-described object and technical configuration of the present invention, and details regarding the operational effects thereof will be more clearly understood by the following detailed description based on the accompanying drawings in the specification of the present invention.

Hereinafter, 'a method and system for recognizing maritime transport products using augmented reality' according to the present invention will be described in detail with reference to the accompanying drawings. The described embodiments are provided so that those skilled in the art can easily understand the technical spirit of the present invention, and the present invention is not limited thereto. In addition, matters expressed in the accompanying drawings may be different from the forms actually implemented in the drawings schematically for easy explanation of the embodiments of the present invention.

On the other hand, each component expressed below is only an example for implementing the present invention. Accordingly, other components may be used in other implementations of the present invention without departing from the spirit and scope of the present invention. In addition, each component may be implemented purely by a configuration of hardware or software, or may be implemented by a combination of various hardware and software components that perform the same function. In addition, two or more components may be implemented together by one piece of hardware or software.

In addition, the expression 'including' certain elements merely refers to the existence of the elements as an expression of 'open type', and should not be construed as excluding additional elements.

1 is a view showing a marine transportation product recognition system using augmented reality according to an embodiment of the present invention.

1, the maritime transportation product recognition system using augmented reality is delivered to the ordering company system 200, which receives the AIS information from the AIS (Automatic Identification System) terminal 100 installed on the ship, the ordering company system 200 It includes a supplier terminal 300 for registering information, and a user terminal 400 in which the transport product confirmation application is stored.

AIS (Automatic Identification System) terminal 100 transmits AIS information. Here, the AIS information may be current location information including vessel identification information, current latitude, current longitude, etc., and is encoded and transmitted using the NMEA-0183 protocol. AIS is an automatic ship identification device, a state-of-the-art device that provides real-time navigation information such as the ship's position, course, and speed. This is a device to prevent collision of ships at sea, and it is a mandatory installation requirement for ships according to the International Maritime Organization (IMO). and progress can be assessed.

The ordering company system 200 analyzes the AIS information transmitted from the AIS terminal 100 and updates the current location information of the vessel in real time or periodically. Since the AIS information is encoded, the ordering company system 200 decodes the AIS information, grasps the current position information of the vessel, and stores the position information.

The ordering company system 200 stores the divided block information for ship manufacturing, determines and places an order among the divided blocks to be produced by an outside company, and stores the ordering information. The ordering information may include ordered items, the name of an outside company, a contract price, a contract date, a warehousing request date, and the like.

When access is requested from the vendor terminal 300 , the ordering company system 200 provides order information placed on the vendor after security authentication of the vendor.

The vendor terminal 300 receives delivery information and registers it in the ordering company system 200 . In other words, when the production of the ordered block is completed, the supplier delivers it to the ordering company by sea transport. It is registered in the company system 200 .

When a shipping product information request signal including image coordinates is received from the user terminal 400, the ordering company system 200 transmits the shipping product information including the order information and delivery information of the ship located in the image coordinates to the user terminal ( 400) is provided.

Meanwhile, the ordering company system 200 may be implemented as a single arithmetic unit or may be embodied in the form of an aggregate device in which two or more arithmetic units are connected to each other. For example, the ordering company system 200 may be implemented as a single server or in a form in which two or more servers are connected.

For a detailed description of the ordering company system 200, reference will be made to FIG. 2 .

When the transport product confirmation application is executed, the user terminal 400 acquires an image by driving the photographing unit, calculates the coordinates of the acquired image, and sends a transport product information request signal including the calculated image coordinates to the ordering company system ( 200) is sent. Here, the transportation product confirmation application may be an application that provides information on transportation products mounted on a ship in augmented reality.

When the transport product information is received from the ordering company system 200 , the user terminal 400 overlays the transport product information on the image acquired through the photographing unit and displays it.

The user terminal 400 having such a configuration is provided with a photographing unit, and is a device capable of installing and executing a transportation product confirmation application, and may be implemented in the form of an electronic device such as a smartphone, tablet, PC, notebook, PDA, etc., In addition, it may be implemented in the form of various electronic devices capable of arithmetic operations and data transmission/reception operations.

A detailed description of the user terminal 400 will be described with reference to FIG. 3 .

2 is a block diagram schematically showing the configuration of an ordering company system according to an embodiment of the present invention.

Referring to FIG. 2 , the ordering company system 200 includes an ordering information database 210 , a delivery information database 220 , a transport ship information database 230 , an ordering information registration unit 240 , a delivery information registration unit 250 , and AIS. It includes a receiving unit 260 , an AIS analyzing unit 270 , and a transport product information providing unit 280 .

The order information database 210 stores order information including order items, names of outside companies, contract prices, contract dates, stocking request dates, and the like.

The delivery information database 220 stores delivery information including order items, transport ship, ship identification information, expected arrival date, captain's name, ship phone number, and the like.

The transport vessel information database 230 stores vessel identification information, message reception time, current latitude, current longitude, and the like.

The order information registration unit 240 receives the order information and registers it in the order information database 210 .

When delivery information registration is requested from a supplier, the delivery information registration unit 250 registers delivery information in the delivery information database 220 . At this time, the delivery information registration unit 250 performs security authentication for the supplier, and registers the delivery information when authenticated.

That is, the delivery information registration unit 250 receives delivery information from the supplier and registers it in the delivery information database 220 .

Although it has been described that the order information database 210 and the delivery information database 220 are separated here, a database (transport product information database) in which order information and delivery information are stored may be configured as one.

The AIS receiver 260 receives AIS information from the AIS terminal installed on the ship and transmits it to the AIS analyzer 270 .

The AIS analysis unit 270 analyzes the AIS information from the AIS receiver 260 to determine the vessel identification information and the current position information of the vessel, and uses the identified vessel identification information and the current position information together with the AIS information reception time to the transport vessel. stored in the information database 230 . Since the AIS information is encoded, the AIS analysis unit 270 decodes the AIS information, grasps the position information of the vessel, and stores the position information in the transport vessel information database 230 .

When the transport product information request signal including the image coordinates is received from the user terminal, the transport product information providing unit 280 inquires a ship belonging to the image coordinates from the transport ship information database 230 . Thereafter, the transportation product information providing unit 280 obtains the order information of the ship inquired from the order information database 210 , and obtains the delivery information of the ship inquired from the delivery information database 220 . Thereafter, the transportation product information providing unit 280 provides transportation product information including the obtained order information and delivery information to the user terminal.

Meanwhile, the ordering information registration unit 240 , the delivery information registration unit 250 , the AIS analysis unit 270 , and the transportation product information providing unit 280 may be implemented by a processor required to execute a program on the computing device, respectively. As such, the ordering information registration unit 240 , the delivery information registration unit 250 , the AIS analysis unit 270 , and the transportation product information providing unit 280 may be implemented by each physically independent configuration, and functionally within one processor. It may be implemented in a form separated by .

In addition, the ordering company system 200 may include at least one arithmetic unit, where the arithmetic unit is a general-purpose central processing unit (CPU), a programmable device device (CPLD, FPGA) implemented to suit a specific purpose, and a custom It may be a semiconductor processing unit (ASIC) or a microcontroller chip.

3 is a block diagram schematically showing the configuration of a user terminal according to an embodiment of the present invention, FIG. 4 is a diagram specifically showing the configuration of the measurement unit shown in FIG. 3, and FIGS. 5 to 8 are the image coordinate calculation unit of the user terminal It is a diagram for explaining in detail the process of calculating the three-dimensional spatial coordinates of the vertices of the captured video image using the location information, direction information, posture information, and device information of

Referring to FIG. 3 , the user terminal 400 includes a communication unit 410 , an input unit 420 , an output unit 430 , a storage unit 440 , a photographing unit 450 , a measurement unit 460 , and an image coordinate calculation unit. 470 , a transport product information request processing unit 480 , an augmented reality processing unit 490 , and a control unit 495 .

The communication unit 410 is a communication means for interconnecting the user terminal 400 through a communication network with the ordering company system so that transportation product information can be provided, for example, a wireless communication module such as mobile communication, satellite communication, the Internet, etc. It may include a wired communication module, a short-distance wireless communication module such as Wi-Fi, and the like.

The input unit 420 is a means for receiving a user request for operation control of the user terminal 400 , and converts the user's request into an electric signal according to the user's manipulation. For example, the input unit 420 receives a transport product confirmation application execution command. When the input unit 420 is implemented in the form of a touch screen, the input unit 420 and the output unit 430 may be the same.

The output unit 430 is a means for providing the operation result or state of the user terminal 400 so that the user can recognize it, and has a visually identifiable display form, for example, a liquid crystal display (LCD) or organic light (OLED). It may be desirable to implement a small flat panel display device such as emitting diodes). In particular, the output unit 430 according to the present invention displays the transportation product information implemented in augmented reality in the augmented reality processing unit 490 on the screen.

The storage unit 440 stores a program necessary for the operation of the augmented reality transportation product information service and data generated during the execution of the program, and basically, the operating program of the user terminal 400 is stored. In particular, the storage unit 440 is stored in the transportation product confirmation application.

The photographing unit 450 is configured to photograph an image according to a user manipulation, and may include, for example, a camera. The photographing unit 450 may photograph an image using a CCD or CMOS device. Meanwhile, in the present embodiment, only the photographing of an image using the CCD or CMOS device provided in the photographing unit 450 has been described, but it may be implemented in the form of receiving an image from an external camera.

The measurement unit 460 measures location information, direction information, posture information, and the like of the user terminal 400 .

The measurement unit 460 includes a position information measurement unit 462 , a direction information measurement unit 464 , and an attitude information measurement unit 466 as shown in FIG. 4 .

The location information measuring unit 462 is a component for measuring the location of the user terminal 400 , and may be, for example, a location measuring device such as a Global Positioning System (GPS), WIPI positioning, or the like. That is, the location information measuring unit 462 receives a signal from a GPS satellite and detects a current location by a GPS sensor or a wireless LAN (WiFi) for detecting a current location in an area where reception of a GPS satellite signal is impossible or reception sensitivity is low. A wireless LAN device for detecting an approximate current location by receiving radio waves from an access point may be included, and the accuracy of location detection may be further improved by mixing two types of sensors.

The direction information measuring unit 464 measures the photographing direction of the user terminal 400 to obtain direction information (azimuth or yaw angle ∂). For example, the direction information measuring unit 464 may include a gyroscope sensor, a geomagnetic sensor, a digital direction sensor, and the like.

The posture information measuring unit 466 measures the roll angle b and the pitch angle v of the user terminal 400 . For example, the posture information measuring unit 466 may be a tilt sensor.

The image coordinate calculation unit 470 calculates the coordinates of the image captured by the photographing unit 450 using the device information of the user terminal 400, the location information measured by the measurement unit 460, the direction information, and the posture information. do. Here, the device information may include, for example, angle of view information according to lens specifications, focal length information, and zoom in/zoom out information by user manipulation. The image coordinates calculated by the image coordinate calculation unit 470 may be the coordinate ranges of latitude and longitude of the area illuminated by the photographing unit 450 .

Hereinafter, a method for the image coordinate calculation unit 470 to calculate the image coordinates will be described in more detail.

5 and 6 , the user terminal 400 has an angle of view 2ω. The angle of view means the width of the background (landscape) that can be put in one frame (screen). That is, as the angle of view increases, the width of the background that can be included in one screen increases. In general, fisheye lenses have an angle of view of 180° or more, and standard lenses have an angle of view of 40° to 60°.

, 롤각(roll angle)은 0°인 것으로 가정하기로 한다. Position information of the user terminal 400 is A ₀ (x ₀ ,y ₀ ,z ₀ ), the height from the ground 1 to the user terminal 400 is h, and the pitch angle of the user terminal 400 is silver

, it is assumed that the roll angle is 0°.

는 자세정보 획득부(466)을 통해 획득할 수 있다.A ₀ (x ₀ ,y ₀ ,z ₀ ) may be obtained through the location information measurement unit 462 , and the height h may be derived through z ₀ . and the pitch angle

may be obtained through the posture information obtaining unit 466 .

는 수학식 1과 같이 계산될 수 있다. 여기서

는 가상영역(도 7의 V)의 중심점과 사용자 단말기(400) 간의 거리를 나타낸다.Then, in FIG.

can be calculated as in Equation 1. here

denotes a distance between the central point of the virtual region (V in FIG. 7 ) and the user terminal 400 .

The virtual area V is a virtual area formed on a plane perpendicular to the gaze direction of the user terminal 400 , and is different from the geographic area R in which an actual subject is photographed. In the case of the video image taken by the user terminal 400, since the horizontal * vertical ratio is determined, as shown in FIGS. 7 and 8 , for the geographical area R to be actually photographed, a predetermined value is placed on an imaginary plane perpendicular to the gaze direction. There is a virtual region V having an aspect ratio of .

Here, if the value calculated through Equation 1 is used, the value of dX in the virtual region V shown in FIG. 7 may be calculated using Equation 2 .

When the dX value is calculated through Equation 2, the dY value in the virtual region V can be calculated. That is, since the horizontal*vertical ratio of the video image photographed by the user terminal 400 is determined, if the horizontal*vertical ratio of the digital image is m:n, the dY value can be calculated through Equation (3).

값을 이용하면 도 8의

값을 계산할 수 있다.

는 A₀와 가상영역(V)의 중심점을 연결하는 수선과, 가상영역(V)에서 가상영역(V)의 중심점으로부터 dY떨어진 지점과 A₀을 연결하는 직선 사이에 형성되는 사이각을 나타낸다.dY value calculated through Equation 3 and Equation 1

Using the value,

value can be calculated.

denotes the angle formed between the vertical line connecting A ₀ and the center point of the virtual area V and the straight line connecting A ₀ and a point dY away from the center point of the virtual area V in the virtual area V.

는 수학식 4를 통해 계산되고, 도 7의

는 수학식 5를 통해 계산된다.Then, the y-coordinate can be calculated through Equations 4 and 5 below. First, in FIG. 8

is calculated through Equation 4,

is calculated through Equation 5.

와

는 가상영역(V)의 중심점으로부터 y축방향으로 실제 지형상에서 이격하여 있는 거리를 각각 나타낸다.Equations 4 and 5 are equations for obtaining the actual distance difference between the video images captured by the user terminal 400 . in other words,

Wow

denotes the distance from the center point of the virtual region V in the y-axis direction on the real topography, respectively.

와

를 이용하면, 도 8에 도시된 실제 촬영되는 지형영역(R)에 대한 y좌표를 구할 수 있다.calculated like this

Wow

By using , it is possible to obtain the y-coordinate of the geographical area R to be actually photographed shown in FIG. 8 .

In the embodiment of the present invention, since it is assumed that the user terminal 400 is photographed in the north direction in the northern hemisphere, y1 and y2 are the same, and y3 and y4 are the same. Accordingly, the y-coordinates (y ₁ , y ₂ , y ₃ , y ₄ ) of the terrain region R for each vertex of the video image captured by the user terminal 400 may be calculated as follows.

,

,

,...)을 이용하면 도 7에 기재된

,

값을 계산할 수 있다. 보다 상세하게는, 도 8의

과

값을 이용하면, A₀를 시작으로 가상영역(V)의 중심점에서 각각 dY씩 y축방향으로 이격하여 있는 지점까지 연결되는 직선의 길이를 계산할 수가 있다. 이렇게 계산된 직선의 길이와Values derived from the above formulas (

,

,

,...) as described in FIG.

,

value can be calculated. More specifically, in FIG.

class

Using the value, it is possible to calculate the length of a straight line from A ₀ to a point spaced apart from the center point of the virtual region V by dY in the y-axis direction. The length of this calculated straight line and

,

값을 계산할 수 있다.With dX

,

value can be calculated.

와

는 가상영역(V)의 꼭지점과 실제 촬영되는 지형영역(R) 사이에 발생되는 x축방향으로의 거리차를 나타낸다.Equations (8) and (9) are equations for obtaining the actual distance difference between the video images captured by the user terminal 400 . in other words,

Wow

denotes a distance difference in the x-axis direction between the vertex of the virtual region V and the actual geographic region R.

와

가 도출되면, 수학식 10을 이용하여 촬영된 영상 이미지에 대한 x좌표를 계산할 수 있다. 사용자 단말기(400)로 촬영된 영상 이미지의 각 꼭지점에 대한 지형영역(R)의 x좌표(x₁,x₂,x₃,x₄)는 다음의 수학식 10과 같다.Through Equation 8 and Equation 9

Wow

When is derived, the x-coordinate of the captured video image may be calculated using Equation 10. The x-coordinate (x ₁ ,x ₂ ,x ₃ ,x ₄ ) of the terrain region R for each vertex of the video image captured by the user terminal 400 is expressed by Equation 10 below.

In the same way, the z-coordinates (z ₁ , z ² , z ₃ , z ₄ ) of the geographic region R for each vertex of the video image captured by the user terminal 400 may be calculated as follows.

As described above, the image coordinate calculation unit 470 according to the present invention includes the position information measured through the position information measurement unit 462 , the direction information measurement unit 464 , and the posture information measurement unit 466 , the direction information, It becomes possible to calculate the three-dimensional spatial coordinates A1, A2, A3, and A4 of the terrain area for each vertex of the captured video image using the posture information and the device information. In addition, latitude and longitude can be calculated by analyzing and processing the three-dimensional spatial coordinates of the obtained video image.

The transportation product information request processing unit 480 transmits a transportation product information request signal including the image coordinates calculated by the image coordinate calculation unit 470 to the ordering company system, and receives the transportation product information from the ordering company system. Transport product information includes order information and delivery information of the vessel.

The augmented reality processing unit 490 overlays the transportation product information on the image captured by the photographing unit 450 and displays it on the screen.

Augmented reality processing unit 490 provides detailed information about the list selected by the user from among the transport product information displayed on the screen in augmented reality.

For example, when the user selects a vessel, the augmented reality processing unit 490 displays a list of products mounted on the vessel in augmented reality, and when the user selects a specific product, an image for the specific product is displayed in augmented reality. do.

In addition, when the user selects a phone number from the transport product information, the augmented reality processing unit 490 makes a phone connection to the phone number.

In addition, when the user selects a company from the transportation product information, the augmented reality processing unit 490 overlays detailed information about the company and displays it on the screen.

Meanwhile, each of the measurement unit 460, the image coordinate calculation unit 470, the transportation product information request processing unit 480, and the augmented reality processing unit 490 may be implemented by a processor required to execute a program on the computing device, etc. . As such, each of the measurement unit 460, the image coordinate calculation unit 470, the transportation product information request processing unit 480, and the augmented reality processing unit 490 may be implemented by each physically independent configuration, and within one processor. It may be implemented in a functionally distinct form.

The control unit 495 controls the overall function of the user terminal 400 by executing the driving program stored in the storage unit 440 according to a user's instruction input through the input unit 420 .

The control unit 495 includes a communication unit 410, an input unit 420, an output unit 430, a storage unit 440, a photographing unit 450, a measurement unit 460, an image coordinate calculation unit 470, and a transport product. It is a configuration for controlling the operation of various components of the user terminal 400 including the information request processing unit 480 and the augmented reality processing unit 490 .

The control unit 495 may include at least one arithmetic unit, where the arithmetic unit is a general-purpose central processing unit (CPU), a programmable device device (CPLD, FPGA) implemented appropriately for a specific purpose, and an application-specific semiconductor operation. It can be a device (ASIC) or a microcontroller chip.

These components that the user terminal 400 may include may be implemented by hardware, software, or a combination thereof, and two or more components may be simultaneously implemented by one piece of hardware or software.

The user terminal 400 of this configuration is a device in which a transportation product confirmation application is installed, and a transportation product information service is possible using augmented reality, and may be implemented in the form of an electronic device such as a smartphone, tablet, PC, notebook, PDA, etc. In addition, it may be implemented in the form of various electronic devices capable of arithmetic operations and data transmission/reception operations.

9 is a view showing a maritime transportation product recognition method using augmented reality according to an embodiment of the present invention, FIGS. 10 to 12 are screens for explaining a maritime transportation product recognition method using augmented reality according to an embodiment of the present invention It is an example diagram.

Referring to FIG. 9 , when the transport product confirmation application is executed (S502), the user terminal drives the photographing unit to acquire an image (S504).

The user terminal calculates the coordinates of the obtained image (S506), and transmits a transportation product information request signal including the calculated image coordinates to the ordering company system (S508). At this time, the user terminal measures location information, direction information, posture information, etc. of the user terminal, and calculates the coordinates of the obtained image using the measured location information, direction information, and posture information.

The ordering company system inquires a ship belonging to the image coordinates (S510), and obtains transportation product information including order information or delivery information of the inquired ship (S512), and transmits it to the user terminal (S514).

The user terminal overlays the transportation product information transmitted from the ordering company system on the acquired image and displays it (S516).

For example, a method of recognizing a shipping product will be described with reference to FIG. 10 .

A case in which the user terminal whose current location is 36.5 north latitude and 125 east longitude captures the same image as in FIG. 10 will be described.

The user terminal obtains the coordinates of the captured image using location information, direction information, posture information, and device information. That is, the user terminal can obtain the range coordinates of the image as ① 36 north latitude, 124 east longitude, ② 36 north latitude, 124.5 east longitude, ③ 36.5 north latitude 124 east longitude, ④ 36.5 north latitude, 124.5 east longitude, as shown in FIG. 11 .

Thereafter, the user terminal requests and receives the shipping product information of the ship located in the coordinate range of the image, that is, 36 to 26.5 north latitude and 124 to 124.5 east longitude, to the ordering company system.

Then, the user terminal may display the transportation product information in augmented reality as shown in FIG. 12 .

The maritime transportation product recognition method using augmented reality according to the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination.

The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - Includes magneto-optical media, and any form of hardware device specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those generated by a compiler. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The teachings of the present principles may be implemented as a combination of hardware and software. In addition, the software may be implemented as an application program actually implemented on the program storage unit. The application program may be uploaded to and executed by a machine comprising any suitable architecture. Preferably, the machine may be implemented on a computer platform having hardware such as one or more central processing units (CPU), a computer processor, random access memory (RAM), and input/output (I/O) interfaces. . The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be part of microinstruction code or part of an application program, or any combination thereof, which may be executed by various processing devices including a CPU. Additionally, various other peripheral devices such as additional data storage and printers may be connected to the computer platform.

It is noted that since some of the constituent system components and methods shown in the accompanying drawings are preferably implemented in software, the actual connections between system components or process functional blocks may vary depending on the manner in which the principles of the present invention are programmed. should be further understood. Given the teachings herein, one skilled in the relevant art will be able to contemplate these and similar implementations or configurations of the principles of the present invention.

As described above, in the method for recognizing maritime transport products using augmented reality according to the present invention and the recording medium recording the same, the configuration and method of the embodiments described above are not limitedly applicable, but the embodiments are various modifications. All or part of each of the embodiments may be selectively combined and configured to achieve this.

As such, those skilled in the art to which the present invention pertains will understand that the present invention may be embodied in other specific forms without changing the technical spirit or essential characteristics thereof. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100: AIS terminal 200: ordering company system
210: order information database 220: delivery information database
230: Transport ship information database 240: Order information register
250: Delivery information register 260: AIS receiver
270: AIS analysis unit 280: transportation product information provision unit
300: vendor terminal 400: user terminal
410: communication unit 420: input unit
430: output unit 440: storage unit
450: photographing unit 460: measuring unit
462: location information measurement unit 464: direction information measurement unit
466: posture information measurement unit 470: image coordinate calculation unit
480: transportation product information request processing unit 490: augmented reality processing unit
495: control

Claims (17)

When the user terminal executes the transportation product confirmation application, the step of driving a photographing unit to obtain an image;
calculating, by the user terminal, the coordinates of the obtained image, and transmitting a transportation product information request signal including the calculated image coordinates to the ordering company system;
transmitting, by the ordering company system, shipping product information including order information or delivery information of the inquired ship to the user terminal, inquiring for a ship located in the image coordinates; and
displaying, by the user terminal, on a screen by overlaying the transport product information on the acquired image;
including,
The user terminal calculating the coordinates of the obtained image, and transmitting the transportation product information request signal including the calculated image coordinates to the ordering company system,
measuring location information, direction information, and posture information of the user terminal;
3D spatial coordinates of a virtual area for each vertex of the obtained image are calculated using the measured location information, direction information, posture information, and user terminal information, and the distance between the virtual area and the actually photographed terrain area calculating three-dimensional spatial coordinates of the terrain area using the difference; and
Sea transportation product recognition method using augmented reality, characterized in that it comprises the step of transmitting a transportation product information request signal including the three-dimensional image coordinates of the geographical area to the ordering company system.
delete In a method for a user terminal to recognize a marine transportation product using augmented reality,
When the transport product confirmation application is executed, driving the photographing unit to obtain an image;
calculating the coordinates of the obtained image;
transmitting a transportation product information request signal including the calculated image coordinates to an ordering company system;
receiving transport product information of a ship located in the image coordinates from the ordering company system; and
displaying the received transport product information on a screen by overlaying the acquired image;
including,
Calculating the coordinates of the obtained image comprises:
measuring location information, direction information, and posture information of the user terminal;
3D spatial coordinates of a virtual area for each vertex of the obtained image are calculated using the measured location information, direction information, posture information, and user terminal information, and the distance between the virtual area and the actually photographed terrain area A method for recognizing maritime transport products using augmented reality, comprising the step of calculating the three-dimensional spatial coordinates of the geographical area using a difference.
delete In the method for the ordering company system to provide a maritime transportation product recognition service using augmented reality,
when a transport product information request signal including image coordinates is received from a user terminal, inquiring a vessel belonging to the image coordinates;
obtaining transport product information including order information or delivery information of the inquired ship and transmitting it to the user terminal; and
Receives AIS information from an AIS (Automatic Identification System) terminal installed on a vessel, analyzes the received AIS information to determine vessel identification information and current position information of the vessel, and sets the identified vessel identification information and current position information to AIS Storing the information in a shipping vessel information database together with the information reception time; and
The step of inquiring the vessel belonging to the image coordinates includes the step of inquiring the vessel belonging to the image coordinates from the transport vessel information database.
delete 6. The method of claim 5,
Shipping product recognition method using augmented reality, further comprising the step of receiving and storing delivery information including at least one of an order item, a ship in charge of transport, ship identification information, an expected arrival date, a captain's name, and a ship phone number from a supplier .
AIS terminal provided on the ship and transmitting the current location information of the ship;
Receives and stores the current location information of the vessel from the AIS terminal, and when a transport product information request signal including image coordinates is received from the user terminal, inquires a vessel belonging to the image coordinates, and places an order / an ordering company system that provides transportation product information including delivery information to the user terminal; and
When the transportation product confirmation application is executed, an image is acquired by driving a photographing unit, the coordinates of the acquired image are calculated, and a transportation product information request signal including the calculated image coordinates is transmitted and transported to the ordering company system a user terminal for receiving product information and overlaying the transport product information on the acquired image;
including,
The user terminal is
Measure own location information, direction information, and posture information, and calculate 3D spatial coordinates of a virtual area for each vertex of the acquired image using the measured location information, direction information, posture information, and user terminal information and calculating the three-dimensional spatial coordinates of the geographic area by using the distance difference between the virtual area and the actual photographed terrain area.
9. The method of claim 8,
Using augmented reality further comprising a supplier terminal that receives delivery information including at least one of order item, transport ship, ship identification information, expected arrival date, captain's name, and ship phone number, and registers it in the ordering company system Sea transport product recognition system.
delete a transport vessel information database in which location information of transport vessels is stored;
a transport product information database in which order information and delivery information for ordered items are stored;
When a transport product information request signal including image coordinates is received from the user terminal, a vessel belonging to the image coordinate is inquired from the transport vessel information database, and order information and delivery information of the inquired vessel from the transport product information database A transport product information providing unit for acquiring transport product information including a and transmitting the information to the user terminal; and
AIS receiver for receiving AIS information from the AIS terminal installed in each ship; and
An AIS analysis unit that analyzes the received AIS information to determine the vessel identification information and the current position information of the vessel, and stores the identified vessel identification information and the current position information together with the AIS information reception time in the transport vessel information database doing,
Buyer system.
delete 12. The method of claim 11,
The transport product information database,
an order information database in which order information including at least one of an order item, an outside company name, a contract price, a contract date, and a storage request date is stored; and
Ordering company system comprising a delivery information database in which delivery information including at least one of an order item, a ship in charge of transport, ship identification information, an expected arrival date, a captain's name, and a ship phone number is stored.
12. The method of claim 11,
The transport product information database includes an order information database and a delivery information database,
The ordering company system may include: an ordering information registration unit for receiving ordering information from an ordering company and registering it in an ordering information database; and
The ordering company system further comprising a delivery information register for receiving delivery information from the supplier and registering it in the delivery information database.
a photographing unit for photographing an image;
a measurement unit for measuring location information, direction information, and posture information;
an image coordinate calculation unit for calculating coordinates of the photographed image by using the measured position information, direction information, posture information, and device information;
a transport product information request processing unit for requesting and receiving transport product information of a ship located in the calculated image coordinates; and
Augmented reality processing unit for overlaying the transport product information on the captured image to display on the screen;
including,
The image coordinate calculation unit,
3D spatial coordinates of a virtual area for each vertex of the photographed image are calculated using the measured location information, direction information, posture information, and device information, and a distance difference between the virtual area and the actually photographed terrain area is calculated. A user terminal, characterized in that for calculating the three-dimensional spatial coordinates of the geographical area using
16. The method of claim 15,
The measurement unit,
a location information measurement unit for measuring the current location of the user terminal;
a direction information measuring unit for measuring direction information including yaw angle information of the user terminal; and
and a posture information measuring unit for measuring posture information including roll angle information and pitch angle information of the user terminal.
16. The method of claim 15,
The device information includes at least one of angle of view information, focal length information, enlargement information, and reduction information.

Images