KR20210125709A - Method And Apparatus for Recognizing Character of Container - Google Patents

Abstract

Disclosed are a method and an apparatus for recognizing a character of a container. When a crane unloads a container on a port, a camera provided in the port or on a lower portion of the crane is used to photograph a side or an upper portion of the container on which container identification information is written when a truck transporting the container passes through a gate. When detecting a character corresponding to a position representing container cargo from a character patch based on a calculated character confidence probability (CP) after detecting the character patch written on the side or the upper portion of the container in the image, a conventional character recognition technique recognizes an identification number string by ignoring an omitted character and continuously listing only recognized characters to fail in container recognition. The present invention, however, provides a method and an apparatus for recognizing a character of a container which uses character prediction for estimating a container identification number by recognizing and remedying an undetected vacant position if some characters are not detected in the container identification number. The apparatus for recognizing a character of a container comprises an image acquisition unit, a character recognition unit, and a data augmentation unit.

Description

Method And Apparatus for Recognizing Character of Container

The present embodiment relates to a container character recognition method and apparatus.

The content described below merely provides background information related to the present embodiment and does not constitute the prior art.

In general, cargo terminal access gates installed in ports go through customs procedures for entering and leaving containers.

Customs procedures are carried out by recognizing characters, symbols, and identification numbers recognized on the outside of the container. In order to recognize characters, symbols and identification numbers from the outside of the container, a number of cameras for photographing the outside of the container in the container monitoring system installed in the port use cameras at each position around the gate when the vehicle loaded with the container passes. Take a picture of the outside of the container.

The container monitoring system is a gate formed so that a vehicle loaded with a container can pass, a sensor means disposed at the gate to detect the entrance and completion of the vehicle, and the vehicle and container by detecting the entry and passage of the vehicle by the sensor means It includes a monitoring means for monitoring the vehicle or the container on the basis of the camera to take a picture, the image taken by the camera.

The monitoring means included in the container monitoring system includes wired and wireless communication means connected to a remote central control center. The camera included in the container monitoring system includes a plurality of lenses to photograph the identification number and character symbol recognized according to the vehicle number and each position of the container.

The container monitoring system uses a sensor means to recognize vehicles and containers entering and passing through the gate, and operates a camera to photograph each location of the vehicle and container to form an image for monitoring.

A conventional container monitoring system implements a predetermined monitoring image by photographing a specific location such as a vehicle number, a driver's seat, a recognized character of a container, and an identification symbol. Recently, images are implemented in two-dimensional or three-dimensional form from images obtained by combining images for each location.

The container monitoring system installs a small number of high-definition, high-speed cameras, detects the vehicle entering and shoots continuously, and then combines the captured images, or installs multiple high-definition cameras and takes pictures when a vehicle enters. use the

In the method of continuously taking pictures by detecting the entrance of a vehicle, a large number of sensors to detect the vehicle and container must be installed, and a separate building must be installed to take pictures uniformly. The method of installing multiple high-definition cameras and taking pictures at once when a vehicle enters requires a large number of high-definition cameras. A large number of sensors and cameras are required to realize a single image, that is, a two-dimensional or three-dimensional monitoring image by photographing the shape of the entire container, not simply to recognize the identification number, character, and symbol recognized from the container. Therefore, there is a problem that installation is complicated and requires a lot of cost to configure the system.

After loading a container on a truck and checking it when passing through the gate, if the container is loaded incorrectly, the problem of reloading and unloading the container occurs. Therefore, when the crane picks up the container, there is a need for a technology that can accurately recognize the characters written on the container.

In this embodiment, when a crane unloads a container in a port, when a truck carrying a container passes through a gate, the upper or side part of the container in which the container identification information is written using the provided camera is photographed, and the container in the image When the character corresponding to the number of digits representing the container cargo is detected from the character patch after detecting the character patch written on the upper or side part, based on the calculated character confidence probability (CP), the conventional character recognition technology ignores the missing character In contrast, container identification failed by recognizing the identification information string in a way that lists only the recognized characters in a row, whereas, if some characters are not detected in the container identification number, character prediction that estimates the container identification information by recognizing and supplementing the undetected empty space An object of the present invention is to provide a container character recognition method and apparatus using

According to an aspect of this embodiment, an image acquisition unit for acquiring an image of a container from a camera installed in a structure under a crane or a port; After detecting a plurality of patches from the container image, and after detecting a character corresponding to container identification information from each of the plurality of patches, the confidence probability (CP) for the character calculated for each character and the character position in the identification number a character recognition unit that combines the detected characters based on , and determines a final character string that includes a missing digit or supplements missing information; and a data augmentation unit (Data Augmentation) for generating a training data set by performing neural network training after receiving the plurality of character patches when it is determined that data reinforcement is necessary and transmitting the data set to the character recognition unit A container character recognition device is provided.

As described above, according to this embodiment, when a crane unloads a container in a port, when a truck carrying a container passes through a gate, the upper or side part of the container in which the container identification information is recorded using the provided camera is photographed and, after detecting the character patch written on the upper part or the side part of the container in the image, the character corresponding to the number of digits representing the container cargo from the character patch is detected based on the calculated character confidence probability (CP), the conventional character recognition technology Container identification failed by recognizing the identification information string by ignoring these missing characters and listing only the recognized characters in a row. On the other hand, if some characters are not detected in the container identification number, the container is identified by estimating and supplementing the undetected vacancy. information can be estimated.

1 is a view showing a container character recognition apparatus according to the present embodiment.
2 is a block diagram schematically showing the internal architecture of the container character recognition apparatus according to the present embodiment.
3 is a view for explaining a container character recognition method according to the present embodiment.
4 is a diagram for explaining a method of increasing/decreasing data according to the present embodiment.
5 is a diagram illustrating a data structure used when increasing/decreasing data according to the present embodiment.
6, 7, 8, and 9 are diagrams for explaining the linear separability of a data set used when increasing/decreasing data according to the present embodiment.

Hereinafter, this embodiment will be described in detail with reference to the accompanying drawings.

1 is a view showing a container character recognition apparatus according to the present embodiment.

The container character recognition apparatus 100 according to the present embodiment is required in the process of the crane unloading or moving the container.

The container character recognition device 100 is applied to an automated port system, and when the crane is brought to the upper part of the container during unloading, a camera attached to the lower part of the crane acquires an image in which the container information is written on the upper part of the container. The movement of the container acquires an image of the outside of the container through a camera provided on a structure in a gate or a port.

The crane picks up the container and moves it from the vessel to the yard and vice versa. The crane knows the position to move the container by using the information pre-stored in the database.

The container character recognition apparatus 100 may identify the container by detecting the characters written on the upper part of the container while the crane unloads or moves the container. The container character recognition device 100 takes a picture of the upper part or the side part through a camera provided in a gate or a port when moving and moving the upper part of the container as the crane goes down when unloading. At least one camera is installed at the lower end of the crane.

2 is a block diagram schematically showing the internal architecture of the container character recognition apparatus according to the present embodiment.

The container character recognition apparatus 100 increases the data volume by performing a character image blender (CIB) on the acquired character patch image in the process of recognizing the characters written on the upper part or the side of the container. The container character recognition apparatus 100 synthesizes an image patch from a label and a latent space to additionally learn.

The container character recognition apparatus 100 performs custom character recognition (CR). The container character recognition apparatus 100 receives an actual image or a false image. The container character recognition apparatus 100 outputs the detected character, the confidence probability of each character, and the position in the identification information of the character. The container character recognition apparatus 100 performs post-processing of the captured image and the detected character. The container character recognition apparatus 100 aligns the detected characters and estimates an appropriate position of the detected characters to enable stronger recognition.

The container character recognition apparatus 100 according to the present embodiment includes an image acquiring unit 210 , a character recognition unit 220 , and a data increasing unit 230 . Components included in the container character recognition apparatus 100 are not necessarily limited thereto.

Each component included in the container character recognition device 100 may be connected to a communication path that connects a software module or a hardware module inside the device, so that they can organically operate with each other. These components communicate using one or more communication buses or signal lines.

Each component of the container character recognition apparatus 100 shown in FIG. 2 means a unit for processing at least one function or operation, and may be implemented as a software module, a hardware module, or a combination of software and hardware.

The image acquisition unit 210 acquires a container image from a camera installed in a structure in a port, such as a lower part of a crane or a gate.

The text recognition unit 220 detects a left text patch and a right text patch from the top of the container, and a text patch on the side of the container from the container image. The character recognition unit 220 detects a character corresponding to the container identification information from each of the character patches. The character recognition unit 220 determines the final character by combining the detected characters based on the character confidence probability (CP) of each character.

The character recognition unit 220 rotates the character patch in a preset direction. The character recognition unit 220 detects the character in the rotated character patch, reads it, and allocates it to an appropriate position in the identification information indicating the container cargo. The character recognition unit 220 determines the order in which the characters detected in the character patch are read using one trend line or two parallel lines. The number of parallel lines is not limited and can be applied. The character recognition unit 220 estimates the correct position in the container identification information for the character detected in the character patch, and synthesizes information from other character patches or other frames in the frame to the unrecognized vacancy and allocates the character.

The character recognition unit 220 compares the confidence probability (CP) value of the character for each digit (index) indicating the container cargo within the frame corresponding to the container image to finally determine the character having a higher confidence probability (CP) value . The character recognition unit 220 finally determines a character having a higher confidence probability (CP) value by comparing the confidence probability (CP) value of the character for each digit (index) indicating the container cargo between the previous frame and the current frame.

The character recognition unit 220 includes a character patch detection unit 222 , a character recognition learning unit 224 , and a character prediction unit 226 . Components included in the character recognition unit 220 are not necessarily limited thereto.

The text patch detection unit 222 detects a text patch on the upper left side of the container, a text patch on the right side, or a text patch on the side of the container from the container image. The character recognition learning unit 224 receives the training data set and performs learning. The character prediction unit 226 learns a character prediction unit that detects a character corresponding to the container identification information in each of the character patches by reflecting the learning result of the character recognition learning unit 224 . The character prediction unit 226 detects a character corresponding to the container identification information in each of the character patches. The character prediction unit 226 determines the final identification information by combining the recognized characters based on the character confidence probability (CP) of each character.

When it is determined that data augmentation is necessary, the data augmentation unit 230 receives the character patch, performs neural network training, generates a training data set, and transmits it to the character recognition unit 220 .

The data increasing unit 230 recognizes the received character patch as a real image. The data increasing unit 230 expresses (eg, 481×1) each number or character in a label of an actual image using a one-hot vector. The data increase unit 230 expresses (eg, 512×1) using a latent space including noise information in order to generate a realistic image. The data increase unit 230 combines a vector expressed using a one-hot vector and a vector expressed using a latent space and passes through a dense layer. Alternatively, the vector expressed using the one-hot vector is combined with the value passed through the dense layer and the vector expressed using the latency space. The data increase unit 230 is a data set for learning based on a vector obtained by combining a value passing through the dense layer or a value passing through the dense layer and a vector expressed using a latent space including noise information. create The location of combining vectors expressed using the latency space including noise information can be changed within the design range.

The data increase unit 230 inputs a value that has passed through the dense layer to a generator, and performs learning based on the value that has passed through the dense layer using the generator. Create fake images.

The data increase unit 230 receives a real image and a fake image using a discriminator, and then determines whether the fake image is a real image or a generated fake image. The data increase unit 230 performs generator learning based on the determination result and generates a data set for character recognition unit learning from the learned generator.

The data increasing unit 230 uses the discriminator (D) to backpropagate the score for determining whether the false image output from the generator (G) is a real image or a generated false image, and a score for determining whether it is real or false. Training is performed to generate a generated false image that is realistic to increase the .

The data increase unit 230 generates a false image using a discriminator loss (D loss) and a generator loss (G loss), and stores the real image and the false image together in a database. Discriminator loss involves edge detection to bring the generator-based false image closer to the real image.

The data increasing unit 230 extracts a character, a color (white or black), and an orientation from a data structure of a real image. The data increasing unit 230 encodes the label using the one-hot vector.

3 is a view for explaining a container character recognition method according to the present embodiment.

The character recognition unit 220 rotates a plurality of character patches (eg, a left character patch and a right character patch) to a correct position. The character recognition unit 220 receives a plurality of character patches (eg, a left character patch and a right character patch) from the character patch detection unit 222 . The character recognition unit 220 rotates a plurality of character patches (eg, a left character patch and a right character patch) in a preset direction (eg, so that characters are read from left to right).

The character recognition unit 220 detects the characters in the rotated character patch (eg, a left character patch, a right character patch) and reads the characters independently.

The character recognition unit 220 detects the characters after division, but aligns the recognized characters because the order of the characters and the space in the middle of the characters cannot be determined. The character recognition unit 220 uses one trend line or two parallel lines to align the characters. The number of parallel lines is not limited and can be applied.

The character recognition unit 220 extracts the center point of each detected character. The character recognition unit 220 recognizes the trend line by using the central point of the character. The trend line means a line to minimize the distribution of the center point of the character.

When the line that minimizes the distribution of the center points of each detected character is one line, the character recognition unit 220 may check whether the characters are in a horizontal direction or a vertical direction by checking the trend line. The character recognition unit 220 calculates the distribution of the central point of the character based on the y-axis of the trend line and divides it into two groups. Separate the characters to determine the order in which they are read. When the trend line detected as the center point of the character is two lines, the character recognition unit 220 calculates a variance value based on the y-axis based on the slope of the trend line and the amount of deviation. By separating the two groups, we can determine the order in which the characters are read.

The character recognition unit 220 finds and assigns the recognized character to a place in the container identification information, and supplements the unrecognized blank in the middle of the character. The character recognition unit 220 compares the confidence probability (CP) value of each digit (index) within a frame (eg, one frame among frame i (Framei) to frame i+1 (Framei+1)) to obtain a higher value. Choose a result with a confidence probability (CP) value. The character recognition unit 220 compares the confidence probability (CP) value of each digit (index) of the character string recognized between frames to finally determine a result having a higher confidence probability (CP) value.

The character patch detection unit 222 detects the image of the upper or side of the container. A character patch is attached to the top or side of the container. The character patch detection unit 222 recognizes two character patches (eg, the left character patch (①) and the right character patch (②)) from the image of the upper or side of the container using a character patch detection model. do.

The character patch detection unit 222 inputs one or more character patches (eg, a left character patch (①) and a right character patch (②)) to the character prediction unit 226 . The character patch detection unit 222 inputs the character patch to the data augmentation unit 230 .

The text prediction unit 226 receives text patch images for each frame that are continuously photographed. The character prediction unit 226 generates a frame at each time step, and receives at least one character patch (eg, a left character patch and a right character patch) for each frame.

The character prediction unit 226 rotates the received character patch (eg, a left character patch, a right character patch) to a correct position, detects a character, and then reads it. (Reading), detects segmentation in units of letters, arranges the order in which the detected letters are read, estimates the empty space among the number of digits representing the container cargo, and places them according to the order (Placement in order).

When detecting a character from the character patch, the character prediction unit 226 calculates a confidence probability (CP) for each detected character. The confidence probability (CP) means a probability value indicating trust in the corresponding character.

A frame for one container is defined as frame i (Framei) to frame i+1 (Framei+1).

As a result of recognizing the left character patch in the frame i, which is the time cell for one container, the character prediction unit 226 recognizes MSCU 5493*3 3, and as a result of recognizing the right character patch, MSDU 5393*6* recognizes it do.

The character prediction unit 226 compares each digit (index) of these two results by elementwise comparison to check a confidence probability (CP) for each patch, and compares each confidence probability to obtain the highest MSDU 5493*6 3, which is a value having a confidence probability (CP), is recognized as a character of frame i.

Thereafter, the character prediction unit 226 recognizes the left character patch in the frame i+1 (Framei+1), which is the time cell for one container, and as a result, M*CU 5*93*8 5 is recognized and the right character patch is recognized. As a result, MSDU 5393*6* recognizes it.

The character prediction unit 226 checks the confidence probability (CP) for each patch by using the two results as elementwise comparison, and compares each confidence probability by comparing each confidence probability to have the highest confidence MSCU 5393*8 5, which is a value with probability (CP), is recognized as a character of frame i+1 (Framei+1).

The character prediction unit 226 compares all the digits of the recognized character string (result) from frame i to frame i+1 (Framei+1) for one container to determine MSCU 539388 5 as a final decision (Final Decision) do.

The character prediction unit 226 combines predicted character sequences in the process of allocating position for a final decision.

The data increasing unit 230 receives a character patch from the character patch detecting unit 222 . The data increasing unit 230 serves to reinforce data when data reinforcement is required for character recognition unit learning. The data increasing unit 230 may be selectively performed. The data increasing unit 230 performs a process for creating a training data set. The data increasing unit 230 allows the character recognition unit to learn additional data to increase the recognition rate. For example, the data increasing unit 230 may perform neural network training on the text patch photographed at night, and then transfer to the text patch photographed during the day. The data increaser 230 may generate generated data based on real data.

4 is a diagram for explaining a method of increasing/decreasing data according to the present embodiment.

The data increasing unit 230 performs a character image blender (CIB). The data augmentation unit 230 is a data augmentation module (DA), and increases a data volume by performing a character image blender (CIB) on an image that is difficult to recognize. The data increaser 230 synthesizes and learns an image patch from a label and a latent space.

The data increasing unit 230 expresses (eg, 481×1) each number or character in a label of a real image using a one-hot vector. The data increase unit 230 expresses the noise information using a latent space (eg, 512×1). The data increase unit 230 combines a vector expressed using a one-hot vector and a vector expressed using a latent space and passes through a dense layer. Alternatively, the one-hot vector is passed through the dense layer and the latency vector is added. The latent vector can change the location of the merging within the design range. The data increasing unit 230 inputs the value passed through the dense layer to the generator. The Generator generates fake images (Generated Image) by performing learning based on the values passed through the Dense Layer.

The discriminator in the data increase unit 230 receives a Real Image and a Fake Image (Generated Image), and then the generated image output from the generator G is Determines whether it is a real image or a generated fake image.

The discriminator (D) backpropagates the score for determining whether the generated image output from the generator (G) is a real image or a generated false image, and increases the score for determining whether the generated image is real or false. Training is performed to generate an image.

The discriminator D may use some or all of a discriminating branch, an edge detection, and a character recognition (CR) branch. The data increase unit 230 generates a false image using a discriminator loss (D loss) and a generator loss (G loss). The data increasing unit 230 stores the real image and the false image together in the database.

5 is a diagram illustrating a data structure used when increasing/decreasing data according to the present embodiment.

The data increasing unit 230 uses a feature in a data structure of a real image when generating a data set for learning.

As shown in FIG. 5 , the data increasing unit 230 is a feature for a Real Image, such as Character, Color (White or Black), and Orientation (up, down, left, right, vertical). direction, horizontal direction).

6, 7, 8, and 9 are diagrams for explaining the linear separability of a data set used when increasing/decreasing data according to the present embodiment.

The data increasing unit 230 uses a Gaussian random generated latent vector for linear separability of a data set for learning. The data increasing unit 230 encodes the label using the one-hot vector. The data increasing unit 230 performs a mapping function of linearizing a one-hot vector or a one-hot vector and a latency space during training. The data increasing unit 230 uses a one-hot encoded label mapping function for linearization of an image that is difficult to recognize.

6, 7, 8, and 9, the data increase unit 230 increases the data volume for the character recognition unit 220 learning.

The above description is merely illustrative of the technical idea of this embodiment, and various modifications and variations will be possible by those skilled in the art to which this embodiment belongs without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the equivalent range should be interpreted as being included in the scope of the present embodiment.

100: container character recognition device
210: image acquisition unit
220: character recognition unit
222: character patch detection unit
224: character recognition learning unit
226: character prediction unit
230: data increase unit

Claims (12)

An image acquisition unit for acquiring an image of a container from a camera installed under a crane or a structure in a port;
After detecting a plurality of patches from the container image, and after detecting a character corresponding to container identification information from each of the plurality of patches, the confidence probability (CP) for the character calculated for each character and the character position in the identification number a character recognition unit that combines the detected characters based on the character recognition unit to determine the final character string including the missing position or supplementing the missing information; and
When it is determined that data reinforcement is necessary, a data augmentation unit that receives the plurality of character patches and performs neural network training to generate a training data set and transmit it to the character recognition unit
Container character recognition device comprising a. According to claim 1,
The character recognition unit,
Container character recognition apparatus, characterized in that receiving the learning data set, performing learning, and detecting the identification information character recognized outside the container from each of the plurality of patches by reflecting the learning result. According to claim 1,
The data increase unit,
Recognizes the received plurality of patches as a real image, expresses each number or character in a label of the real image using a one-hot vector, and displays the real image (Real Image). The noise information corresponding to the image) is expressed using a latent space, and the vector expressed using the one-hot vector is passed through the dense layer, or the one-hot vector and the latent space ( By combining vectors expressed using Latent Space, the value passed through the Dense Layer is input to the Generator to generate Fake Images, and using a Discriminator, the actual image ( After receiving the real image and the fake images, it is determined whether the false image is a real image or a generated fake image, and a training data set for learning the character recognition unit is generated based on the determination result. Container character recognition device, characterized in that. 4. The method of claim 3,
The data increase unit,
By merging the edge detection network in the discriminator (D), the false image output from the generator (G) can generate a patch image suitable for the string condition (eg, a label expressed as a one-hot vector) to be generated. Container character recognition device, characterized in that it backpropagates the score for determining whether it is a real image or a generated false image, and performs learning to generate a fake image like the real one in order to increase the score for determining whether it is real or false. 5. The method of claim 4,
The data increase unit,
Learning a generator and a discriminator using a discriminator loss (D loss) and a generator loss (G loss), generating the false image from the generator, and storing the real image and the false image together in a database Container character recognition device. 4. The method of claim 3,
The data increase unit,
Character, color, and orientation are extracted from the data structure of the real image, and the one-hot vector and the latency vector are synthesized and input to the generator. Container character recognition device. According to claim 1,
The character recognition unit,
Container character recognition apparatus, characterized in that rotating the plurality of character patches in a preset direction. 8. The method of claim 7,
The character recognition unit,
Container character recognition device, characterized in that detecting and reading characters in the plurality of rotated character patches, and detecting identification information indicating container cargo in character units. 9. The method of claim 8,
The character recognition unit,
Container character, characterized in that the reading order of the characters detected in the plurality of character patches is determined using one trend line or two or more parallel lines recognition device. 10. The method of claim 9,
The character recognition unit,
Container character recognition apparatus, characterized in that the vacancy of characters detected in the plurality of character patches and unrecognized characters in the container identification information string is accurately estimated. 11. The method of claim 10,
The character recognition unit,
Container character recognition apparatus, characterized in that the character having a higher confidence probability (CP) value is finally determined by comparing the confidence probability (CP) value of the character for each digit representing the container cargo in the frame corresponding to the container image. 11. The method of claim 10,
The character recognition unit,
Container character recognition apparatus, characterized in that the character having a higher confidence probability (CP) value is finally determined by comparing the confidence probability (CP) value of the character for each digit (index) indicating the container cargo between the previous frame and the current frame.

Images