KR102595542B1 - Tracking device, tracking method, data structure and program of tracking data - Google Patents

Abstract

The tracking device 1200 derives the three-dimensional position of the display area by the stencil (label) based on the first captured image and the position of the first imaging device that captured the first captured image, and determines the three-dimensional position. The information representing is stored in the storage medium in association with the identification information of the thick plate.

Description

Tracking device, tracking method, data structure and program of tracking data

The present invention relates to a tracking device, a tracking method, a data structure and program for tracking data, and is particularly suitable for use for tracking goods. This application claims priority based on Japanese Patent Application No. 2020-198065 filed in Japan on November 30, 2020, and the entire contents thereof are incorporated herein by reference.

Manufactured products such as steel products including heavy plates are moved to various storage locations. So, tracking the product is done. Tracking refers to tracking and specifying the location of an object.

In the technology described in Patent Document 1, tracking of a steel plate is performed by registering the X-Y coordinates of the overhead crane when the steel plate is unloaded together with the steel plate number of the steel plate.

In addition, in the technology described in Patent Document 2, the stationary position of the steel material expressed by three-dimensional coordinates is calculated by a stationary position calculation program based on the crane's movement in the horizontal and vertical directions and operations such as load detection, and , the calculated stationary location information indicating the stationary location is stored in correspondence with the steel identification information of the steel material.

Japanese Patent No. 5109451 Publication Japanese Patent No. 3734419 Publication

However, in the technology described in Patent Document 1, the information used for tracking the steel plate is the X-Y coordinates of the overhead crane. For this reason, tracking can only be performed within the range that can be transported with an overhead crane. In addition, in the technology described in Patent Document 1, the Z coordinate cannot be specified from the

Additionally, in the technology described in Patent Document 2, the three-dimensional coordinates of the steel material are calculated. However, like Patent Document 1, tracking can only be performed within the range that can be transported by crane. In addition, the three-dimensional coordinates of the steel material calculated by the technology described in Patent Document 2 are the three-dimensional coordinates of the stationary location calculated by the stationary location calculation program. Therefore, in the technology described in Patent Document 2, steel materials can only be tracked at stationary locations that are predetermined in the stationary location calculation program.

In an actual manufacturing site, products are transported using various transport devices. Additionally, workers sometimes transport products without using a transport device. In addition, there are cases where manufactured products are placed in a political location where the address of the location has not been determined. Additionally, there are various methods of placing the product. In the technology described in Patent Documents 1 and 2, it is not easy to track such products. These problems also exist in products other than manufactured products.

The present invention was made in consideration of the above problems, and its purpose is to improve the tracking precision of articles.

The tracking device of the present invention is a tracking device for tracking an article, and includes one or more image acquisition means for acquiring a captured image of an area containing display information displayed on the article, and acquiring identification information for the article. Identification information acquisition means, position derivation means for deriving an information display position, which is a three-dimensional position where the display information is displayed, based on the captured image acquired by the image acquisition means, and derivation by the position derivation means. It is characterized by comprising a storage means for storing the information display position on the article in a storage medium in relation to the identification information for the article acquired by the identification information acquisition means.

The tracking method of the present invention is a tracking method for tracking an article, comprising one or more image acquisition steps of acquiring a captured image of an area containing display information displayed on the article, and acquiring identification information of the article. An identification information acquisition process, a position derivation process for deriving an information display position, which is a three-dimensional position where the display information is displayed, based on the captured image acquired by the image acquisition process, and derivation by the position derivation process. and a storage step of storing the information display position on the article in a storage medium in relation to the identification information of the article acquired through the identification information acquisition step.

The data structure of tracking data of the present invention has display information displayed on an article and non-display information not displayed on the article, and the non-display information is an information display position that is a three-dimensional position where the display information is displayed. A tracking device is used to derive the information display position on the article based on a captured image of an area containing display information displayed on the article.

The program of the present invention is a program for causing a computer to execute processing for tracking an article, and includes one or two image acquisition steps of acquiring a captured image of an area containing display information displayed on the article, and An identification information acquisition step of acquiring identification information, a position derivation step of deriving an information display position that is a three-dimensional position where the display information is displayed, based on the captured image acquired by the image acquisition step, and the position Characterized by causing the computer to execute a storage process for storing the information display position on the article derived by the derivation process in a storage medium in relation to the identification information of the article obtained by the identification information acquisition process. .

1 is a diagram illustrating an example of the flow of products in a factory.
Figure 2 is a diagram showing an example of a display area on a thick plate.
FIG. 3A is a diagram showing an example of display content using a stencil.
FIG. 3B is a diagram showing an example of display content by a label.
Figure 4 is a diagram showing an example of how a heavy plate is loaded and unloaded on a transport ship.
Figure 5 is a view showing the heavy plate being hung up on the unloading device as seen from the garter extension installation direction (y-axis direction).
FIG. 6A is a diagram showing a first example of a conveyance mode of a thick plate.
FIG. 6B is a diagram showing a second example of a conveyance mode of a thick plate.
FIG. 6C is a diagram showing a third example of a conveyance mode of a thick plate.
FIG. 6D is a diagram showing a fourth example of a conveyance mode of a thick plate.
FIG. 6E is a diagram showing a fifth example of a conveyance form of a thick plate.
FIG. 7 is a diagram showing an example of how a rear plate is loaded and unloaded on a transport vehicle.
Figure 8 is a diagram showing an example of a state in which a heavy plate is loaded on a transport vehicle.
FIG. 9A is a diagram showing a first example of a heavy plate placed in a place in a factory.
FIG. 9B is a diagram showing a second example of a heavy plate placed in a place in a factory.
FIG. 9C is a diagram showing a third example of a heavy plate placed in a place in a factory.
Figure 10 is a diagram showing an example of how a thick plate is processed in a production facility.
Fig. 11 is a diagram showing an example of the original thick plate and the thick plate after cutting.
Fig. 12 is a diagram showing an example of the functional configuration of the tracking device.
Fig. 13 is a diagram showing an example of a management table.
FIG. 14 is a diagram showing an example of the position of the stencil in the article and the position of the components of each display item in display using the stencil.
Fig. 15 is a diagram showing an example of the position of the label in the article and the position of the components of each display item when displayed by the label.
Fig. 16A is a diagram showing a first example of information stored in the management table.
FIG. 16B is a diagram showing a second example of information stored in the management table.
Figure 17 is a flowchart explaining an example of tracking device processing when tracking a thick plate.
FIG. 18 is a diagram illustrating an example of a method for deriving the three-dimensional position of a display area using a stencil.
FIG. 19 is a diagram illustrating an example of a method for deriving the position in the height direction (z-axis direction) of the display area using a stencil.
FIG. 20 is a diagram illustrating a modified example of a method for deriving the three-dimensional position of a display area using a stencil.
Fig. 21 is a diagram showing an example of the hardware configuration of the tracking device.

BEST MODE FOR CARRYING OUT THE INVENTION One embodiment of the present invention will be described below with reference to the drawings.

In addition, the objects of comparison, such as length, position, size, spacing, etc., are the same, not only when they are strictly identical, but also when they are different within the scope of the invention (for example, within the range of tolerances determined at the time of design). ) are also included.

<Overview of the factory>

In this embodiment, a case in which various types of thick plates (e.g., thick steel plates) manufactured at a plurality of steel mills are tracked in a location including a factory that performs various treatments such as heat treatment and cutting is exemplified. . 1 is a diagram illustrating an example of the flow of products in a factory to which this embodiment is applied.

In Figure 1, heavy plates manufactured at a steel mill are loaded onto a transport ship. Afterwards, the transport ship lands on the quay wall near the factory. Heavy plates are unloaded from transport ships and loaded onto transport vehicles (trucks, etc.). When heavy plates transported by transport vehicles arrive at the factory, they are placed in a location within the factory. The heavy plates placed in the storage area are processed such as heat treatment or cutting (gas cutting) before being placed in the warehouse. After that, the thick plate is shipped from the factory to the consumer.

<Conception>

If the position of the heavy plate being transported as described above is tracked based on the operation of the transport equipment, as in the techniques described in Patent Documents 1 and 2, there are restrictions on the range that can be tracked. Therefore, the present inventors focused on the fact that information managed by the manufacturer of the heavy plate is displayed on the heavy plate for actual product management (position management of each heavy plate). Tracking the display position of this information is equivalent to tracking a thick plate. And, if the information managed by the manufacturer of such a heavy plate can be specified, the heavy plate can be identified. The present embodiment is made based on this idea.

<Marking performed on thick plates>

Figure 2 is a diagram showing an example of a display area on a thick plate. In addition, the x-axis, y-axis, and z-axis shown in FIG. 2 represent the directional relationship between FIG. 2 and FIGS. 3A, 3B, 14 to 15, and 18 to 20 described later, and FIG. 4 and It does not necessarily correspond to the direction of the x-y-z coordinates shown in FIG. 5. Additionally, x-y-z coordinates are, for example, coordinates displayed in the world coordinate system.

In FIG. 2, the display area 201 is a display area using a stencil. In the example shown in FIG. 2, the display area 201 formed by the stencil exists on the surface of the thick plate. Additionally, the broken line shown in FIG. 2 is a virtual line and is not actually displayed. In the example shown in FIG. 2, the label (side label) 202 is affixed to the side surface (plate thickness portion) of the thick plate.

FIG. 3A is a diagram showing an example of display content using a stencil. In addition, as described above, the x-y-z coordinates shown in FIG. 3A indicate the directional relationship with FIG. 2 and the like. In addition, in the x-y-z coordinates shown in FIG. 3A, the symbol with a black circle (●) in a white circle (○) indicates that it is facing from the inside of the paper toward the front side. The notation of these x-y-z coordinates is the same in other drawings.

In the example shown in FIG. 3A, the stencil display includes a mark 301, a price name 302, a standard 303, a size 304, an ID 305, a price code 306, and an order number. (307) is included.

The mark 301 is a mark indicating the manufacturer of the thick plate. The demand price name 302 is information indicating the demand price (buyer) of the heavy plate. The standard 303 is information indicating the standard of the thick plate. The size 304 is information indicating the size of the thick plate (for example, thickness × width × length). The ID 305 is information for uniquely identifying the heavy plate, and in the example shown in Fig. 3A, it is the plate number (identification number of the heavy plate). Therefore, the same ID is not attached to different plates. The demand price code 306 is information designated by the heavy plate manufacturer to attach to the heavy plate based on the demand price. The order number 307 is a part of a number used to identify an order for a heavy plate from a customer.

Mark (301), price name (302), specification (303), size (304), ID (305), price code (306) and order number (307) can be generally displayed for heavy plates. It's information. Additionally, information other than the ID 305 does not need to be included in the display using the stencil. Additionally, information other than the above-described information may be included in the display using the stencil.

Additionally, for example, the number of digits in the demand price code 306 may vary depending on the demand price, or the notation method of the standard 303 may vary depending on the steel mill that manufactures the heavy plate. Additionally, for example, the order number 307 may be attached depending on the demand price or may not be attached. Additionally, the size of the displayed items may also vary depending on the steel mill that manufactures the heavy plate. Additionally, the position of the display area by the stencil may also vary depending on the steel mill that manufactures the thick plate. In this way, the content, location, and size of the mark made by the stencil differ depending on the steel mill that manufactures the thick plate and the price of demand.

Mark (301), price name (302), standard (303), size (304), ID (305), price code (306), and order number (307) are information managed by the heavy plate manufacturer. . Additionally, the mark 301, price name 302, standard 303, size 304, ID 305, price code 306, and order number 307 are information that can be identified by a person. Human-identifiable information refers to information that can be understood by a person when he or she sees the information. Information that requires machine encoding and decoding to create and read information, such as barcodes and two-dimensional codes, is not information that can be identified by humans. Specifically, the mark 301, demand name 302, standard 303, size 304, ID 305, demand code 306, and order number 307 are read and decoded by a machine. is unnecessary information, includes at least one of letters, numbers, and symbols, and does not include information that requires decoding by a machine for reading.

FIG. 3B is a diagram showing an example of display content by a label. In addition, as described above, the x-y-z coordinates shown in FIG. 3B represent the directional relationship with FIG. 2 and the like. In addition, in the x-y-z coordinates shown in FIG. 3B, the symbol with × in the white circle indicates the direction from the front side of the page toward the inner side. The notation of these x-y-z coordinates is the same in other drawings.

In the example shown in FIG. 3B, the label indication includes standard 311, ID 312, size 313, delivery date 314, and demand price code 315. The delivery date 314 is information indicating the delivery date of the thick plate. The standard 311, ID 312, size 313, and price code 315 are the same as the standard 303, size 304, ID 305, and price code 306, respectively. Additionally, information other than the ID 312 does not need to be included in the display using the label. Additionally, information other than the above-described information may be included in the display using the label. The content, location, and size of the label markings, like the stencil markings, also vary depending on the steel mill that manufactures the heavy plates and the price of demand.

However, the meaning of the same display item for the same plate is the same for the stencil display and the label display. Therefore, when the display using the stencil shown in FIG. 3A and the display using the label shown in FIG. 3B are made on the same thick plate, the ID 305 displayed by the stencil and the ID 312 displayed by the label means the same thing. Additionally, the standard 303 displayed by the stencil and the standard 311 displayed on the label mean the same thing. The size 313 and the demand code 315 also mean the same things as the size 304 and the demand code 306, respectively. As described above, the standard 303, ID 305, size 304, and price code 306 are information managed by the heavy plate manufacturer. Therefore, the standard 311, ID 312, size 313, and demand code 315 are also information managed by the heavy plate manufacturer. Additionally, the delivery date 314 of the heavy plate is also information managed by the heavy plate manufacturer. Additionally, the specifications (311), ID (312), size (313), delivery date (314), and price code (315) are information that can be identified by humans.

In addition, in addition to the above-mentioned markings, the thick plate may be marked with, for example, an engraving.

<Transfer type of heavy plate>

<<Transportation by transport ship>>

Figure 4 is a diagram showing an example of how a heavy plate is loaded and unloaded on a transport ship. In FIG. 4 , a case in which heavy plates are loaded and unloaded on the transport ship 400 using the gantry crane 410 is illustrated. FIG. 5 is a diagram showing the thick plate 420 being hung on the unloading device 411 as seen from the direction in which the garter 412 is extended and installed (y-axis direction). The x-y-z coordinates shown in FIGS. 4 and 5 represent the relationship in the directions of FIGS. 4 and 5 .

The gantry crane 410 has an unloading device (trolley, etc.) 411, a garter (traverse seat) 412, and a leg structure 413. The unloading device 411 moves along the extension installation direction (y-axis direction) of the garter 412 in a state where the conveyance object is suspended. FIG. 4 shows the heavy plate 420 being lifted and moved by the unloading device 411. When the thick plate 420 is hung on the unloading device 411 and viewed from the direction in which the garter 412 is extended and installed (y-axis direction), it becomes as shown in FIG. 5 .

4 and 5 illustrate a state in which one thick plate 420 is suspended by the unloading device 411. However, the transport form of the thick plate by the unloading device 411 is not limited to this transport form. FIGS. 6A to 6E are diagrams showing first to fifth examples of conveyance modes of thick plates. 6A to 6E, the unloading device 411 is shown in a more simplified manner.

FIG. 6A illustrates a state in which two thick plates are aligned in the horizontal plane (x-y plane) direction and are simultaneously transported by the unloading device 411. FIG. 6B illustrates a state in which one thick plate is being transported on the top side (negative direction side of the x-axis) of the unloading device 411. FIG. 6C illustrates a state in which one thick plate is being transported on the bottom side (the positive direction side of the x-axis) of the unloading device 411. FIG. 6D illustrates a state in which one thick plate is being transported in the center of the unloading device 411. FIG. 6E illustrates a state in which multiple sheets (two sheets) of thick plates are stacked and simultaneously transported by the unloading device 411. In this way, in this embodiment, there are cases where a plurality of thick plates are transported simultaneously in a stacked state.

As described above, the form of conveyance of the thick plate by the unloading device 411 is not limited. Additionally, the method of hoisting the heavy plate using the unloading device 411 may be a method using the magnetic attraction force of a magnet, a method using a hacker, or other methods.

Since the gantry crane 410 itself can be explained using known technology, detailed description is omitted here.

Heavy plates manufactured at the steel mill are transported to the quay wall and placed in a predetermined location where they can be transported by the gantry crane 410. As shown in FIG. 4, a plurality of thick plates may be placed in a predetermined location in a stacked state. In some cases, the back plates are placed one at a time in a designated location. 4 and the like illustrate a case where the thickness (plate thickness) of a plurality of thick plates is the same for convenience of notation. However, there are cases where the thickness of a plurality of thick plates is different. That is, there are cases where at least one of the thickness, width, and length of the thick plate placed at a predetermined location is different.

Figure 4 illustrates a case where a plurality of thick plates are stacked so that the surface of the thick plate is parallel to the surface on which the thick plate is placed. However, the method of stacking thick plates is not limited to this method. For example, there are cases where a plurality of thick plates are stacked so that the surface of the thick plate is perpendicular or at an angle close to vertical with the surface on which the thick plate is placed. This is the same even in positions other than the predetermined location on the quay wall.

In this embodiment, the first imaging devices 430a to 430f are installed in the garter 412 and the leg structure 413. The number and position of the first imaging devices 430a to 430f are set so that as much of the display area of the thick plate as possible, including the display area 201 by the stencil of the thick plate and the display area by the label 202, is imaged. Additionally, the first imaging device (430g to 430m) is also installed in the unloading device 411. The number and position of the first imaging devices 430a to 430m are indicated by a stencil of the thick plate 420 being transported by the gantry crane 410 in at least one of the first imaging devices 430a to 430m. At least one of the display areas defined by the area 201 and the label 202 is set to be imaged as much as possible.

The first imaging devices 430a to 430m are capable of adjusting imaging conditions (imaging direction, angle of view, zoom magnification, and aperture (F value), etc.). The adjustment of the imaging conditions may be performed by an operator (worker) operating the first imaging device, or may be performed automatically based on instructions from an external device. In addition, for example, among the first imaging devices 430g to 430m installed in the garter 412, the first imaging devices 430i to 430m installed in a position close to the thick plate 420 during transportation may be a fiberscope camera. do. Using a fiberscope camera, for example, it is possible to close up the display area 201 by the stencil and the display area by the label 202 of the thick plate 420 being transported.

Additionally, the captured image captured by the first imaging device is not limited to a still image and may be a moving image. When the captured image captured by the first imaging device is a moving image, an image extracted from the moving image captured by the first imaging device (i.e., a frame image of the moving image) is used in the tracking device 1200 described later. In this case, it is desirable to extract a still image that is as focused as possible on the subject captured by the first imaging device from the moving image captured by the first imaging device. The fact that the captured images captured by the imaging device are not limited to still images also applies to the second to seventh imaging devices described later.

The heavy plate transported from the steel mill is placed in a predetermined location of the gantry crane 410. The area including the display area 201 by the stencil of the thick plate placed at the predetermined location of the gantry crane 410 is imaged by the first imaging devices 430c to 430f. The operator (worker) operates the information processing terminal in the operation room of the gantry crane 410 or the information processing terminal (tablet terminal, etc.) held by the operator (worker) of the gantry crane 410, A preview image display instruction is given to the imaging devices 430c to 430f. The first imaging devices 430c to 430f transmit the preview image to the information processing terminal based on the preview image display instruction. Then, a preview image is displayed by the information processing terminal. The operator (worker) operates the information processing terminal and gives an image capture instruction to the first image pickup device that is capturing a preview image in which the display area by the stencil to be imaged is displayed. When receiving an imaging instruction, the first imaging devices 430c to 430f capture an image of the area including the stencil display area 201. Additionally, the preview image is a live image displayed for confirmation by an operator (worker), and is displayed based on the display instructions of the preview image. The captured image is assumed to be an image captured based on an imaging instruction when a preview image is displayed. In addition, here, the case of displaying a preview image is exemplified, but it is not necessarily necessary to display the preview image.

When a plurality of thick plates are stacked and transported simultaneously, the area including the display area 201 by the stencil cannot be imaged for the thick plate below. In this case, instead of the captured image of the area including the display area 201 by the stencil, the captured image of the area including the display area by the label 202 is captured.

Depending on the position where the thick plate is placed, the display area 201 by the stencil and the display area by the label 202 may not be included in the imaging range of the first imaging devices 430c to 430f. In this case, the operator (worker) operates the unmanned aerial vehicles (drones) 440a to 440b equipped with the first imaging device to select the area (or label) including the display area 201 by the stencil on the desired thick plate. The captured image (area including the display area 202) may be captured using the first imaging device. Furthermore, the area including the display area 201 by the stencil (or the area including the display area by the label 202) of the desired thick plate is captured using the first imaging device carried by the operator (worker). You may capture an image. In addition, the first imaging device carried by the operator (worker) may be, for example, an information terminal device (tablet terminal, smartphone, etc.) provided with a first imaging device.

In addition, the display of the preview image and the capturing of the captured image by the first imaging devices 430c to 430f described above are performed using the first imaging devices 430a to 430b and 430g to 430w other than the first imaging devices 430c to 430f. The same is true for . In addition, when a plurality of thick plates are stacked and conveyed simultaneously, the captured image of the area including the display area by the label 202 is captured by the first imaging devices other than the first imaging devices 430c to 430f. The same is true for (430a to 430b, 430g to 430w). In addition, the imaging method when the display area 201 by the stencil and the display area by the label 202 are not included in the imaging range of the first imaging devices 430c to 430f are the first imaging devices 430c to 430f. The same applies to the first imaging devices 430a to 430b and 430g to 430w other than 430f). Therefore, in the following, detailed descriptions of the first imaging devices 430a to 430b and 430g to 430w other than the first imaging devices 430c to 430f are omitted.

The thick plate is placed in a predetermined location of the gantry crane 410 and then loaded onto the transport ship 400 by the gantry crane 410. There are cases where multiple sheets of heavy plates are stacked and loaded onto the transport ship 400 at the same time. In some cases, heavy plates are loaded one at a time on the transport ship 400. Additionally, at least one of the thickness, width, and length of the heavy plates loaded on the transport ship 400 may be different.

The captured image of the area including the display area 201 by the thick plate and stencil being transported by the gantry crane 410 is captured by the first imaging device 430g to 430m installed on the unloading device 411. .

When the thick plate is loaded on the transport ship 400, a captured image of the area including the display area 201 by the stencil of the thick plate is captured by the first imaging devices 430a to 430b installed on the garter 412 or The image is captured by the first imaging device 430n installed on the transport ship 400. Like the first imaging devices 430a to 430m, the first imaging device 430n installed on the transport ship 400 also adjusts imaging conditions (imaging direction, angle of view, zoom magnification, and aperture (F value), etc.) It is possible. Additionally, in Fig. 4, for convenience of notation, the number of first imaging devices installed in the transport ship 400 is set to one. However, a plurality of first imaging devices are installed in the transport ship 400 so that the area where the thick plate is placed is included in the imaging range.

The transport ship 400 loaded with heavy plates as described above is centered on the quay wall near the factory. As described above, the heavy plate is unloaded from the transport ship 400. As described above, in this embodiment, the heavy plate is loaded using the gantry crane 410. For the heavy plate being transported by the gantry crane 410, an image is captured in the same manner as in the case of loading on the transport ship 400. Additionally, when loaded by the gantry crane 410 and placed in a predetermined location of the gantry crane 410, the heavy plate is imaged in the same manner as in the case where it is transported from the steel mill and placed in a predetermined location of the gantry crane 410. An image is captured.

<<Transportation by transport vehicle>>

The heavy plate placed at the predetermined location of the gantry crane 410 is loaded onto the transport vehicle. FIG. 7 is a diagram showing an example of how a rear plate is loaded and unloaded on a transport vehicle. The heavy plate placed at the predetermined location of the gantry crane 410 is loaded onto the transport vehicle 700 using a crane, etc., not shown. As shown in FIG. 7, there are cases where a plurality of thick plates are stacked and loaded onto the transport vehicle 700 at the same time. However, there are cases where the heavy plates are loaded one by one on the transport vehicle 700. In addition, there are cases where at least one of the thickness, width, and length of the thick plate loaded on the transport vehicle 700 is different. Additionally, the thick plate may be transported and loaded on the transport vehicle 700 as shown in FIGS. 6A to 6E.

FIG. 8 is a diagram showing an example of a state in which a heavy plate is loaded on the transport vehicle 700. First imaging devices 430o to 430p are installed in a structure installed at a standby position of the transport vehicle 700. The number and position of the first imaging devices 430o to 430p are the display area 201 by the stencil of the heavy plate loaded on the transport vehicle 700 and the display area by the label 202, so that as many of the heavy plates as possible are displayed. The area is set to be imaged. Additionally, the first imaging device 430q is installed in the transport vehicle 700. Like the first imaging devices 430a to 430m, the first imaging devices 430o to 430q can also adjust imaging conditions (imaging direction, angle of view, zoom magnification, and aperture (F value), etc.).

The area including the display area 201 by the stencil of the thick plate loaded on the transport vehicle 700 is imaged by the first imaging devices 430o to 430q.

<<Return within the factory>>

When the transport vehicle 700 loaded with the heavy plates as described above arrives at the factory, the heavy plates are unloaded from the transportation vehicle 700. The heavy plates loaded on the transport vehicle 700 are unloaded from the transport vehicle 700 using a crane or the like, not shown. As shown in FIG. 7, there are cases where a plurality of thick plates are stacked and simultaneously unloaded from the transport vehicle 700. However, there are cases where the heavy plates are loaded one at a time from the transport vehicle 700. In addition, there are cases where at least one of the thickness, width, and length of the thick plate unloaded from the transport vehicle 700 is different. Additionally, the thick plate may be transported and unloaded from the transport vehicle 700 as shown in FIGS. 6A to 6E. In addition, a transport equipment (gantry crane 410) used when transporting a heavy plate from the transport ship 400 to a predetermined location of the gantry crane 410, and a transport device (gantry crane 410) used to transport the heavy plate from the transport vehicle 700 to a predetermined location in the factory. The transport equipment used (such as a crane not shown) is a different transport equipment. In this way, a heavy plate transported to a certain storage location by a certain transportation device may be transported to a different storage location by a different transportation device from that certain transportation device and cannot be transported by that certain transportation device. This is the same when conveying a thick plate between different locations, such as between locations in a factory.

FIGS. 9A to 9C are diagrams showing first to third examples of heavy plates placed in a storage area in a factory. The first imaging devices 430r to 430w are installed in a structure installed at a location in the factory. As shown in FIGS. 9A to 9C, heavy plates are placed in various locations in the factory. Therefore, the number and position of the first imaging devices 430r to 430w are such that as much of the display area of the thick plate as possible is captured as the display area by the stencil and the label 202 of the thick plate placed in the place. It is set. Like the first imaging devices 430a to 430m, the first imaging devices 430r to 430w are also capable of adjusting imaging conditions (imaging direction, angle of view, zoom magnification, and aperture (F value), etc.).

The area including the display area 201 by the stencil of the thick plate placed in the storage area in the factory is imaged by the first imaging devices 430r to 430w.

When the thick plate is placed in the storage area for shipment, the area of the thick plate including the display area 201 by the stencil (or the display area by the label 202) is the first imaging device 430r to 430w as described above. ) is captured by

After that, the thick plate is transported to the production facility. Figure 10 is a diagram showing an example of how a thick plate is processed in a production facility. In FIG. 10, production equipment 1000 is, for example, equipment for heat treatment and equipment for gas cutting. FIG. 10 illustrates a case in which thick plates 1002a and 1002b are conveyed using the conveyance roll 1001. In addition, conveyance of the thick plate between the place and the conveyance roll 1001 is realized by using a conveyance device, etc., not shown.

First imaging devices (430x to 430y) are installed in a structure installed on the production line of a factory. The number and position of the first imaging devices 430x to 430y are set so that the display area 201 by the stencil of the thick plates 1002a and 1002b is imaged. Like the first imaging devices 430a to 430m, the first imaging devices 430x to 430y can also adjust imaging conditions (imaging direction, angle of view, zoom magnification, and aperture (F value), etc.). In addition, it is assumed that the thick plates 1002a and 1002b conveyed on the conveyance roll 1001 do not overlap. In addition, the conveyance path of the thick plates 1002a and 1002b on the conveyance roll 1001 is assumed in advance. For this reason, in this embodiment, it is assumed that there is no case in which the display area 201 by the stencil of the thick plates 1002a and 1002b is not imaged by the first imaging devices 430x to 430y. However, the display area 201 formed by the stencil of the thick plates 1002a and 1002b may be imaged using an unmanned aerial vehicle equipped with a first imaging device. In addition, in FIG. 10, the broken line represents the conveyance path of the thick plates 1002a and 1002b on the conveyance roll 1001, and does not exist.

The thick plate 1002a transported from the storage location in the factory is loaded on the transport roll 1001 and transported by the transport roll 1001. Before entering the production equipment 1000, the area including the display area 201 by the stencil of the thick plate 1002a being conveyed on the conveyance roll 1001 is imaged by the first imaging device 430x. In addition, after processing in the production equipment 1000 is completed and exiting the production equipment 1000, the area including the display area 201 by the stencil of the thick plate 1002b being conveyed on the conveyance roll 1001 is , is imaged by the first imaging device 430y.

After that, the thick plate is placed in a storage location as illustrated in FIGS. 9A to 9C in the warehouse. As described above, the area including the display area 201 by the stencil (or the display area by the label 202) of the thick plate placed in the storage area in the factory is imaged by the first imaging devices 430r to 430w. do. At the time of shipment, the heavy plate is moved from the warehouse to the storage location for shipping. The area including the display area 201 by the stencil (or the display area by the label 202) of the thick plate placed in the storage area for shipping is imaged by the first imaging devices 430r to 430w.

Even at each location in a factory, there are cases where multiple sheets of thick plates are placed in a stacked state. In some cases, one piece of heavy plate is placed at each location in the factory. In addition, there are cases where at least one of the thickness, width, and length of the thick plates placed at each location in the factory is different.

Fig. 11 is a diagram showing an example of the original thick plate and the thick plate after cutting. In Figure 11, a case where the thick plate 1100 is cut and the thick plates 1110, 1120, 1130, and 1140 are manufactured is illustrated. After the thick plates 1110, 1120, 1130, and 1140 are manufactured in this way, marking is performed on each of the thick plates 1110, 1120, 1130, and 1140. FIG. 11 shows display areas 1111, 1121, 1131, and 1141 for the thick plates 1110, 1120, 1130, and 1140. Information including the IDs of the thick plates 1110, 1120, 1130, and 1140 is displayed in the display areas 1111, 1121, 1131, and 1141. Display areas 1111, 1121, 1131, and 1141 are displayed using, for example, a stencil. In addition to marking by stencil, at least one of marking by engraving and marking by label may be performed. Additionally, the display in the display areas 1111, 1121, 1131, and 1141 may be a handwritten display.

Additionally, the ID of the original thick plate 1100 may be displayed in the display areas 1111, 1121, 1131, and 1141. Display areas 1111, 1121, 1131, and 1141 are displayed using, for example, a stencil. However, the display in the display areas 1111, 1121, 1131, and 1141 may be a handwritten display. This is the same for display in the display area 201 and display using the label 202. Hereinafter, the display areas 1111, 1121, 1131, and 1141 are assumed to be display areas using stencils.

When the thick plates 1110, 1120, 1130, and 1140 are manufactured, for example, a stencil-based display area 1111, 1121, and 1131 of the desired thick plate is captured using a first imaging device carried by an operator. 1141) may be captured. In addition, the first imaging device carried by the operator (worker) may be, for example, an information terminal device (tablet terminal, smartphone, etc.) provided with a first imaging device. Additionally, the operator (worker) operates the unmanned aerial vehicle equipped with the first imaging device to capture a captured image of the area including the stenciled display areas 1111, 1121, 1131, and 1141 of the desired thick plate. The image may be captured using the first imaging device.

As described above, in this embodiment, at least when placed on the transport ship 400, when loaded from the transport ship 400 and loaded on the transport vehicle 700, and at each head of the gantry crane 410 and the factory. An example of a case where, when placed in place, captured images of the area including the display area by the stencil and the display area by the label 202 of each thick plate are obtained. do.

In this embodiment, the position of the thick plate is automatically tracked by a tracking device using the image captured as described above. Additionally, when the thick plates are conveyed one by one, the display area 201 by the stencil can be imaged, so the captured image of the area including the display area by the label 202 becomes unnecessary. In the following description, when the display using the stencil and the display using the label 202 are collectively referred to as the display using the stencil (label), they are indicated as display using the stencil (label). Additionally, when the stencil display areas 201, 1111, 1121, 1131, and 1141 and the label 202 are collectively referred to, they are indicated as stencil (label) display areas, if necessary.

<Tracking device>

FIG. 12 is a diagram showing an example of the functional configuration of the tracking device 1200. The hardware of the tracking device 1200 is realized by using, for example, an information processing device including CPU, ROM, RAM, HDD, and various interfaces, or dedicated hardware. FIG. 13 is a diagram showing an example of the management table 1300 managed by the tracking device 1200.

As shown in FIG. 12, in this embodiment, the tracking device 1200 includes a first image acquisition unit 1201, a recognition unit 1202, an acquisition unit 1203, and a matching unit 1204, It has a specification unit 1205, a position derivation unit 1206, a storage unit 1207, an output unit 1208, and a second image acquisition unit 1209.

The first image acquisition unit 1201 acquires the first captured image of the area including the display area by the stencil (label). The recognition unit 1202 recognizes the components, positions, etc. of display items in display using a stencil (label) from the first captured image acquired by the first image acquisition unit 1201. The acquisition unit 1203 obtains the information stored in the ID column of the actual item display column in the management table 1300, which will be described later. The collation unit 1204 compares the information recognized by the recognition unit 1202 with the information obtained by the acquisition unit 1203. The specification unit 1205 specifies one thick plate from the thick plates managed in the management table 1300, based on the result of the collation in the collation unit 1204. In this embodiment, the identification information acquisition unit 1210, which acquires the identification information of the thick plate, has a recognition unit 1202, an acquisition unit 1203, a matching unit 1204, and a specification unit 1205. The position derivation unit 1206 derives the x-y-z coordinates (3-dimensional coordinates) of the display area by the stencil (label) as the 3-dimensional position of the thick plate specified in the specification unit 1205. The storage unit 1207 stores information indicating the three-dimensional position of the thick plate in the management table 1300. The output unit 1208 outputs information indicating the three-dimensional position of the thick plate. The second image acquisition unit 1209 acquires a second captured image for specifying the position in the height direction (z-axis direction) of the display area by the stencil (label) in the position derivation unit 1206. An example of details of the tracking device 1200 will be described later with reference to the flowchart in FIG. 17.

In Fig. 13, information used by the tracking device 1200 to track the thick plate is stored in the management table 1300. The information stored in the management table 1300 includes information managed by the heavy plate manufacturer. However, part of the information stored in the management table 1300 may be information that can be utilized in distribution, etc. Here, the information managed by the heavy plate manufacturer is information set or acquired by the heavy plate manufacturer. The information managed by the heavy plate manufacturer includes information that allows operators who are employees of the heavy plate manufacturer to understand the meaning of the information. More specifically, the information managed by the heavy plate manufacturer is, for example, information created by the heavy plate manufacturer and stored in the management table 1300. In this embodiment, the information stored in the management table 1300 is an example of tracking data, and by storing each information in the management table 1300, an example of the data structure of tracking data is realized.

In Fig. 13, information in the same column becomes identification information for the same thick plate (the columns under the same No. become columns where identification information for the same thick plate is respectively stored).

In Fig. 13, information regarding each display item displayed by a stencil (label) is stored in the actual product display column. In the column of the non-actual product display, information other than the information regarding each display item of the display using the stencil (label) is stored. In this embodiment, the information stored in the column for actual product display is an example of display information included in the tracking data, and the information stored in the column for non-actual product display is an example of non-display information included in the tracking data.

An example of information stored in the current item display column of the management table 1300 will be described.

In the actual item display column, information about the stencil is stored in the stencil column.

In the ID column of the stencil column, information indicating the contents and positions of the components of ID305 included in the display by the stencil is stored. Additionally, a component refers to individual information that constitutes a display item, such as letters, numbers, symbols, and marks.

In the demand name column of the stencil column, information indicating the contents and positions of the components of the demand name 302 included in the display by the stencil is stored.

In the size column of the stencil column, information indicating the contents and positions of the components of the size 304 included in the display by the stencil is stored.

In the standard column of the stencil column, information indicating the contents and positions of the components of the standard 303 included in the display using the stencil is stored.

In the demand code column of the stencil column, information indicating the contents and positions of the components of the demand code 306 included in the display by the stencil is stored.

In the order number column of the stencil column, information indicating the contents and positions of the components of the order number 307 included in the display by the stencil is stored.

In addition to the display items in the above stencil display, the contents of display items that may be displayed as display items in the stencil display are also stored in the management table 1300. “···” under the additional information column of the stencil column in FIG. 13 indicates this. Additionally, the display item itself may be added and/or deleted.

In the actual item display column, information about the label is stored in the label column.

In the ID column of the label column, information indicating the contents and positions of the components of the ID 312 included in the display by the label is stored.

In the size column of the label column, information indicating the contents and positions of the components of the size 313 included in the display by the label is stored.

In the standard column of the label column, information indicating the contents and positions of the components of the standard 311 included in the display by the label is stored.

In the delivery date column of the label, information indicating the contents and positions of the components of the delivery date 314 included in the label display is stored.

In the demand code column of the label column, information indicating the contents and positions of the components of the demand code 315 included in the display by the label is stored.

In addition to the display items in the above label display, the contents of display items that may be displayed as display items in the label display are also stored in the management table 1300. The “···” under the demand code column in the label column in Fig. 13 indicates this. Additionally, the display item itself may be added and/or deleted.

Next, an example of information stored in the non-actual item display column of the management table 1300 will be described.

In the position column within the product, information indicating the position of the display area when a predetermined position on the thick plate is set as the origin (reference) is stored.

In the stencil column of the position column within the product, information indicating the position of the display area by the stencil when a predetermined position on the thick plate is set as the origin (reference) is stored.

In the label column of the position column within the product, information indicating the position of the display area by the label when a predetermined position on the thick plate is set as the origin (reference) is stored.

In the item position column, information indicating the three-dimensional position of the thick plate is stored. The information stored in the item position column is information indicating the three-dimensional position of the thick plate specified by the tracking device 1200. Accordingly, the three-dimensional position of the same thick plate specified by the tracking device 1200 may change not only the x and y coordinates but also the z coordinate as tracking progresses (movement of the thick plate). Therefore, the three-dimensional position of the information display position of the same thick plate specified by the tracking device 1200 also changes not only the (The position in the height direction) may also change.

In the registration time column, information indicating the date and time (timing determined from the date and time) when the heavy plate was placed in the current position is stored.

In the parent ID field, information indicating the ID of the original heavy plate is stored. There are cases where heavy plates are cut in factories. The original thick plate refers to the thick plate before cutting. Therefore, when information indicating the ID of the original thick plate is stored in the parent ID column, the information stored in the column of the same column as the parent ID column becomes information about the thick plate after cutting. When the thick plates 1110, 1120, 1130, and 1140 are manufactured as shown in FIG. 11, in the management table 1300, the actual product indication column for the thick plates 1110, 1120, 1130, and 1140 contains the following: Information indicating the contents and positions of components of display items displayed in the display areas 1111, 1121, 1131, and 1141 is stored. Additionally, information indicating the ID of the thick plate 1100 is stored in the parent ID column for the thick plate 1110, 1120, 1130, and 1140. In this way, in this embodiment, the case where information on the thick plate after cutting is managed as a new column (No) in the management table 1300 shown in FIG. 13 is exemplified. In the management table 1300, information about the thick plate after cutting is stored in the actual product display and non-actual product display columns.

Here, for example, there are cases where the thick plate 1110 is cut and another plurality of thick plates are manufactured. In this case, information indicating the ID of the thick plate 1110 is stored in the parent ID column for the plurality of separate thick plates. In the ID column for the thick plate 1110, information indicating the ID of the thick plate 1100 is stored. In this way, by following the information stored in the key ID column, the evolution of each plate can be specified. In this way, for a certain plate, the arrangement of the plate obtained by tracing the information stored in the key ID column is called the out-of-plate series of the plate. For example, the out-of-plate series of the thick plate 1110 becomes the thick plates 1100 and 1110. In addition, in the parent ID column of each plate, not only information indicating the ID of the immediately adjacent plate, but also information indicating the IDs of all plates belonging to the out-of-plate series are stored, so that the out-of-plate series can be known, for example. do. In this case, any ID defined by the manufacturer is used.

In the transport tool ID column, the ID of the transport tool for the thick plate is stored. In this embodiment, the case where the transport tools are the transport ship 400 and the transport vehicle 700 is exemplified.

When the heavy plate is loaded on the transport ship 400, the tracking device 1200 may store information indicating the ID of the transport ship 400 as follows, for example.

First, the tracking device 1200 acquires the ID of the heavy plate and the ID (so-called line number) of the transport ship 400. Then, the tracking device 1200 identifies a row in which information indicating the same ID as the ID of the acquired thick plate is stored from the ID column of the management table 1300. Then, the tracking device 1200 stores information indicating the ID of the acquired transport vessel 400 in the transport tool ID column of a specific row.

For example, a display area using a stencil 201 or a label 202 and information indicating the ID of the transport ship 400 are displayed by the first imaging devices 430a, 430b, and 430n. A captured image including the area may be captured. In this case, the tracking device 1200 may recognize and acquire the ID of the thick plate and the ID of the transport ship 400 from the captured image.

Additionally, an image captured by the first imaging devices 430a, 430b, and 430n including the display area 201 by the stencil or the display area by the label 202, and information indicating the ID of the transport ship 400 are A captured image including the displayed area may be captured. For example, a captured image including a display area 201 by a stencil or a display area by a label 202 is followed by a captured image including an area where information indicating the ID of the transport ship 400 is displayed. It is okay to take pictures. In this case, the tracking device 1200 displays the ID recognized from the captured image including the display area 201 by the stencil or the display area by the label 202, and information indicating the ID of the transport ship 400. The ID recognized from the captured image including the area may be acquired as the ID of the thick plate and the ID of the transport ship 400.

Additionally, for example, the ID of the thick plate and the ID of the transport ship 400 may be input into an information processing terminal (tablet terminal, etc.) possessed by the operator (worker) of the transport ship 400. In this case, the ID of the heavy plate and the ID of the transport ship 400 are transmitted from the information processing terminal to the tracking device 1200, so that the tracking device 1200 transmits the ID of the heavy plate and the ID of the transport ship 400. You can acquire it.

Additionally, when the heavy plate is loaded on the transport vehicle 700, the tracking device 1200 may store information indicating the ID of the transport vehicle 700 as follows, for example.

First, the tracking device 1200 acquires the ID of the rear plate and the ID of the transport vehicle 700 (for example, license plate information). Then, the tracking device 1200 identifies a row in which information indicating the same ID as the ID of the acquired thick plate is stored from the ID column of the management table 1300. Then, the tracking device 1200 stores information indicating the ID of the acquired transport vehicle 700 in the transport tool ID column of a specific row.

For example, a display area using a stencil 201 or a label 202 and an area where information indicating the ID of the transport vehicle 700 is displayed by the first imaging devices 430o to 430q. You may capture a captured image containing . In this case, the tracking device 1200 may recognize and acquire the ID of the thick plate and the ID of the transport vehicle 700 from the captured image.

In addition, the first imaging devices 430o to 430q display a captured image including the display area 201 by the stencil or the display area by the label 202, and information indicating the ID of the transport vehicle 700. You may capture a captured image including the area in question. For example, a captured image including a display area 201 by a stencil or a display area by a label 202, followed by a captured image including an area where information indicating the ID of the transport vehicle 700 is displayed. This may be taken as an image. In this case, the tracking device 1200 has an ID recognized from the captured image including the display area 201 by the stencil or the display area by the label 202, and information indicating the ID of the transport vehicle 700. The ID recognized from the captured image including the displayed area may be acquired as the ID of the rear plate and the ID of the transport vehicle 700.

Additionally, for example, the ID of the thick plate and the ID of the transport vehicle 700 may be input into an information processing terminal (tablet terminal, etc.) possessed by the operator (worker) of the transport vehicle 700. In this case, the ID of the heavy plate and the ID of the transportation vehicle 700 are transmitted from the information processing terminal to the tracking device 1200, so that the tracking device 1200 can track the ID of the heavy plate and the ID of the transportation vehicle 700. You can obtain an ID.

The data structure of the management table 1300 is a data structure in which the information (e.g., product location and registration time) stored in each column of the management table 1300 is updated each time the heavy plate is placed in a new position. It's okay. Additionally, the data structure of the management table 1300 may be a data structure in which information is added. In the latter case, it has the advantage of being able to track the tracking history.

Next, an example of the content stored in the actual product display column of the management table 1300 and the content stored in the position within the product column of the non-actual product display column will be described in more detail.

FIG. 14 is a diagram showing an example of the position of the stencil in the article and the position of the components of each display item in display using the stencil. FIG. 14 illustrates a case where display using a stencil is made at the display position shown in FIG. 2 and the content shown in FIG. 3A.

In FIG. 14, among the eight vertices of the rectangular thick plate, the vertex 203 shown in FIG. 2 is taken as the origin, and the direction along the width direction of the thick plate from the vertex 203 is the positive direction of the y-axis and the longitudinal direction of the thick plate. In the x-y coordinates with the direction along the positive x-axis, the position of the stencil in the article and the position of the components of each display item by the stencil are indicated.

In this embodiment, among the positions of the components of each display item in the stencil display, the four vertices of the circumscribed rectangle are determined by the positions of the maximum value of the x-axis, the minimum value of the x-axis, the maximum value of the y-axis, and the minimum value of the y-axis. The x-y coordinates of the two vertices on the diagonal line are taken as the position of the stencil in the article. In Fig. 14, the x-y coordinates (Ls10, Ws10) and (Ls11, Ws11) of the two vertices 1401a and 1401b of the circumscribed rectangle show the positions of the stencil in the article. In addition, here, to simplify the explanation, the circumscribed rectangle is assumed to be a rectangle parallel to the coordinate axes (x-axis and y-axis in the example shown in Fig. 14) representing the position within the article.

Additionally, the x-y coordinates of the position of the center of gravity (or centroid) of each display item's component (letter, number, symbol, mark) in display using a stencil are taken as the location of the component. In Figure 14, the y-axis coordinates of 5, 6, 3, 4, 9, 7, -, 0, and 1, which are components of the ID 305 included in the display by the stencil, are Ws1, Ws2, Ws3, respectively. They are Ws4, Ws5, Ws6, Ws7, Ws8, and Ws9, and the x-axis coordinates are all Ls1.

As described above, the position of each component of each display item displayed by the stencil is displayed in x-y coordinates.

Fig. 15 is a diagram showing an example of the position of the label in the article and the position of the components of each display item when displayed by the label. FIG. 15 illustrates a case where display by a label is performed at the display position shown in FIG. 2 and the display content shown in FIG. 3B.

In FIG. 15, among the eight vertices of the thick plate of a rectangular parallelepiped, the vertex 204 shown in FIG. 2 is taken as the origin, and the direction along the width direction of the thick plate from the vertex 204 is the positive direction of the y-axis, the plate thickness of the thick plate. In y-z coordinates with the direction along the direction being the positive direction of the z-axis, it indicates the position of the label within the product and the position of the components of each display item by the label.

In this embodiment, the label is assumed to have a rectangular shape. Among the four vertices of the label, the y-z coordinates of two diagonal vertices are taken as the label's position in the article. Additionally, the y-z coordinates of the position of the center of gravity (or centroid) of the component of each display item in display by label are taken as the location of the component. In Figure 15, the y-z coordinates of the two vertices 1501a and 1501b of the label show the position of the label within the article.

As described above, in the management table 1300 shown in FIG. 13, the ID column of the stencil column of the actual product display column contains the contents and positions of the components of the ID 305 included in the stencil display. Information representing is remembered. Therefore, as shown in FIG. 16A, in the example shown in FIG. 14, the ID column of the stencil column of the actual product display column contains the contents of the constituent elements of ID 305 included in the display by stencil, “5”, “6”, “3”, “4”, “9”, “7”, “-”, “0”, and “1” are remembered. Additionally, in the ID column of the stencil column of the actual product display column, (Ls1, Ws1), (Ls2, Ws1), (Ls3, Ws1), (Ls4, Ws1), (Ls5, Ws1), (Ls6, Ws1), (Ls7, Ws1), (Ls8, Ws1), (Ls9, Ws1) are remembered.

Similarly to the ID column of the stencil column, information indicating the content and position of the components of the display item is stored in the other display item columns of the actual item display column. Additionally, the information indicating the position of the components of each display item in the label column is not x-y coordinates, but y-z coordinates.

In addition, as shown in FIG. 16B, in the example shown in FIG. 14, the stencil column of the non-actual product display column and the column of the position within the article contain (Ls10, Ws10) as information indicating the location of the stencil within the article. , (Ls11, Ws11) are remembered.

Similarly to the stencil column of the position within the product column, information indicating the position of the label within the product is stored in the label column of the position within the product column. Additionally, as described above, the information indicating the location of the label within the product is not x-y coordinates, but y-z coordinates.

When creating the management table 1300, for example, the operator may operate the user interface of the tracking device 1200 and input information to be stored in each column of the management table 1300 into the tracking device 1200. . Additionally, the tracking device 1200 may receive information to be stored in each column of the management table 1300 from an external device such as a computer that manages factory operations.

The information stored in the actual product display column of the management table 1300, the position within the product in the non-actual product display column, and the information stored in the parent ID column may be determined after the display for the heavy plate is made. In some cases, the indication is determined before it is made. Therefore, when the information to be stored in the relevant column is determined before display on the back plate is performed, the tracking device 1200 acquires the information to be stored in the relevant column as described above, for example, and manages it. It is stored in each column of the table 1300.

On the other hand, the information stored in the item location and registration time column of the non-actual product display column may not be determined before the display on the plate is made. Therefore, the tracking device 1200 acquires the information to be stored in the relevant column, for example, at the timing when the information to be stored in the relevant column is obtained, and stores it in each column of the management table 1300. As described above, the date and time when the information stored in the item location column was changed is stored in the registration time column. In this case, the date and time when the back plate was placed in the current position is registered in the registration time column.

Additionally, for items in the column marked as non-actual goods, the information is stored in the column of that item only if the information for that item is acquired, and if the information for that item is not acquired, it is stored in the column of that item. The information is not remembered. Additionally, if the information stored in the item position field is not updated even after a predetermined time has passed from the date and time stored in the registration time field, the tracking device 1200 determines that the identification (tracking) of the position of the back plate has been lost. In this case, it is desirable to reset the information stored in the item location and registration time column. Additionally, at the same location, a later plate with an earlier date and time stored in the registration time column will not move before a later plate with a later date and time stored in the registration time column (i.e., when there is overlap of plates, There are no cases where the plate underneath suddenly disappears). For this reason, the tracking device 1200 preferably performs the following when resetting the information stored in the item location and registration time columns. That is, the tracking device 1200 is a back plate whose date and time is stored in the registration time column that is later than the date and time described in the reset registration time column, and the three-dimensional position is the same as the three-dimensional position stored in the reset item position column. If there is a rear plate in a three-dimensional position, it is desirable to issue a sign for confirmation, such as an alarm.

Next, an example of the function of the tracking device 1200 will be described. FIG. 17 is a flowchart explaining an example of processing by the tracking device 1200 when tracking a thick plate. Referring to FIG. 17, an example of the function of the tracking device 1200 will be described.

In step S1701 of FIG. 17 , the first image acquisition unit 1201 acquires the first captured image. The first captured image is one of the first imaging devices 430a to 430y, the first imaging devices provided by the unmanned aerial vehicles 440a to 440c, and the first imaging device possessed by the operator (worker). 1 This is a captured image captured by an imaging device. The first captured image may be transmitted directly from the first imaging device to the tracking device 1200 or may be transmitted from the first imaging device to the tracking device 1200 through an information terminal device. The processing according to the flowchart of FIG. 17 includes any of the first imaging devices 430a to 430y, the first imaging devices provided by the unmanned aerial vehicles 440a to 440c, and the first imaging device carried by the operator (worker). This is executed each time a captured image captured by one imaging device is received by the tracking device 1200. In this embodiment, the ID of the first imaging device that captured the first captured image and the imaging conditions when the first captured image was captured are transmitted together with the first captured image.

The first image acquisition unit 1201 acquires the first captured image, the ID of the first imaging device that captured the first captured image, and the imaging conditions when the first captured image was captured. Among the information set in the imaging device at the time of imaging, the imaging conditions include information necessary to specify the position of the display area by a stencil or the display area by a label, as will be described later.

Additionally, if the position of the first imaging device when capturing the first captured image is not fixed, location information of the first imaging device when capturing the first captured image is transmitted together with the first captured image. . For example, when the unmanned aerial vehicles (440a to 440c) can determine the three-dimensional coordinates (x-y-z coordinates) of the unmanned aerial vehicle using GPS (Global Positioning System), the first image captured by the unmanned aerial vehicle is As the location information of the device, the three-dimensional coordinates may be transmitted. A first imaging device 430n installed on the transport ship 400, a first imaging device 430g to 430m installed on the unloading device 411, and a first imaging device 430q installed on the transport vehicle 700. ), similarly to the unmanned aerial vehicles 440a to 440c, the position information of the first imaging device may be transmitted together with the first captured image. This is the same for the first imaging device carried by the operator (worker).

Additionally, when the first imaging device cannot capture an area that should be included in the first captured image, prior to processing in step S1701, the imaging conditions (imaging direction, angle of view, zoom magnification, and aperture (F) of the first imaging device are set. value), etc.) are adjusted.

Next, in step S1702, the recognition unit 1202 selects the constituent elements (letters, numbers, symbols, marks) of the display item in the display using the stencil (label) from the first captured image acquired in step S1701. recognize The process of recognizing the contents of the components of the display item can be realized by, for example, a technology using known OCR (Optical character recognition), so detailed description thereof is omitted here. Additionally, the processing in step S1702 may be realized using technology using AI (Artificial Intelligence) such as NN (Neural Network) or technology combining AI and OCR (so-called AI OCR).

Additionally, the recognition unit 1202 specifies whether the component of the display item is displayed by a stencil or displayed by a label. For example, the recognition unit 1202 performs edge detection processing based on the first captured image acquired in step S1701, detects the edge of the thick plate, and specifies the area of the thick plate based on the detection result. Then, the recognition unit 1202 determines whether the specific area is the plate surface portion of the thick plate or the side (plate thickness) portion of the thick plate. This determination is realized, for example, based on the aspect ratio of the specific area. As a result of this determination, if the specific area is a plate surface portion of a thick plate, the recognition unit 1202 recognizes that the constituent elements of the display item are those of a display using a stencil. On the other hand, when the specific area is the side (plate thickness) portion of the thick plate, the recognition unit 1202 recognizes that the components of the display item are those displayed by the label.

Additionally, the recognition unit 1202 performs edge detection processing based on the first captured image, and when the area of the rectangular label 202 is detected, the recognition unit 1202 determines that the component of the display item is a display by the label. If not, the components of the display item may be recognized as those displayed by a stencil.

In addition, in this embodiment, to simplify the explanation, for the rear plate at the top, the first captured image is assumed to include the display area 201 by the stencil rather than the display area by the label 202. .

Additionally, the recognition unit 1202 derives the position of the display item. An example of a method by which the first recognition unit 1202 derives the location of a display item will be described. First, the recognition unit 1202 performs edge detection processing based on the first captured image acquired in step S1701 and detects the edge of the thick plate. Then, the recognition unit 1202 detects the origin of the position within the article (a predetermined position on the thick plate) in the first captured image. For example, the recognition unit 1202 determines the positional relationship in the captured image based on a positional relationship preset as the positional relationship between the display area by the stencil (label) and the origin of the position within the article among the eight vertices of the rear plate of the rectangular parallelepiped. A vertex that becomes the origin of the position within the product (a predetermined position on the thick plate) is derived.

As the positional relationship between the display area by the stencil (label) and the origin of the position within the article, for example, the following information may be set in advance in the tracking device 1200. In other words, when the component of the display item of the display using the stencil (label) is viewed so that the top of the component is located on the upper side, the display item by the stencil (label) is on the same side as the display by the stencil (label). Information that the vertex located at the lower right side of the display is the origin of the position within the article may be set in advance in the tracking device 1200. The recognition unit 1202 derives a position based on the origin of the position within the article in the captured image as the position of the component elements of the display item in the display using the stencil (label). When the processing in step S1702 ends, the processing in step S1703 is executed.

In step S1703, the acquisition unit 1203 obtains the information stored in the ID column of the actual item display column in the management table 1300 stored in the storage unit 1207.

Next, in step S1704, the matching unit 1204 compares the information recognized in step S1702 with the information obtained in step S1703. If the information recognized in step S1702 is recognized as information for a display using a stencil, the matching unit 1204 stores it in the ID column of the actual product display column, the stencil column, and the information obtained in step S1703. The information provided is compared with the information recognized in step S1702. On the other hand, if the information recognized in step S1702 is recognized as information for display by a label, the matching unit 1204 selects the actual product display column, the label column, and the ID column among the information obtained in step S1703. The information stored in and the information recognized in step S1702 are compared.

Next, in step S1705, the specifying unit 1205 specifies one thick plate from the thick plates managed in the management table 1300 based on the collation result in step S1703. Specifically, the specifying unit 1205 specifies information that matches the information recognized in step S1702 among the information obtained in step S1703.

Here, the match refers to the fact that in the information recognized in step S1702, there is a part that completely matches the content and arrangement order of the components stored in the stencil column of the actual product display column or the ID column of the label column. . In the example shown in FIGS. 3A and 3B, the contents of the components of IDs 305 and 312 displayed by stencils or labels are 5, 6, 3, 4, 9, 7, -, 0, 1, The order of arrangement is from left to right: 5, 6, 3, 4, 9, 7, -, 0, 1. Therefore, the specification unit 1205 uses the information in which 5, 6, 3, 4, 9, 7, -, 0, and 1 are arranged in this order from the left as the ID managed in the management table 1300. Specify from the column. In the information recognized in step S1702, not only when some of these components are missing, but also when components other than these components are arranged, the information recognized in step S1702 is displayed in the actual product display column. It is assumed that the information does not completely match the content and arrangement order of the components stored in the ID column of the stencil column or label column.

Next, in step S1706, the position derivation unit 1206 determines whether the position of the first imaging device that captured the first captured image acquired in step S1701 is known, based on the information acquired in step S1701. . In this embodiment, the first imaging devices 430a to 430f installed on the garter 412 and the bridge structure 413, and the first imaging device installed on the structure installed at the waiting position of the transport vehicle 700 (430o to 430p), first imaging devices (430r to 430w) installed in a structure installed at a location in the factory, and first imaging devices (430x to 430y) installed in a structure installed in the production line of the factory. The three-dimensional position of is assumed to be known. For each of these first imaging devices, the ID of the first imaging device and the location information (3-dimensional coordinates (x-y-z coordinates)) of the first imaging device are related to each other and registered in advance in the tracking device 1200. Let's do it. If position information (3-dimensional coordinates) associated with the same ID as the ID of the first imaging device acquired in step S1701 is registered in advance in the tracking device 1200, the position derivation unit 1206 It is determined that the position of the first imaging device that captured the first captured image is known, and if not, it is determined that the position of the first imaging device that captured the first captured image acquired in step S1701 is not known.

If, as a result of the determination in step S1706, the position of the first imaging device that captured the first captured image acquired in step S1701 is not known, the processing in step S1707 is executed. In step S1707, the position derivation unit 1206 determines whether the position information of the first imaging device that captured the first captured image has been acquired in step S1701. As a result of this determination, if the positional information of the first imaging device that captured the first captured image is not acquired in step S1701, the process proceeds to step S1712, which will be described later.

As a result of the determination in step S1706, a case where the position of the first imaging device that captured the first captured image acquired in step S1701 is known, and as a result of the determination in step S1707, location information of the first imaging device that captured the first captured image. When acquired in step S1701, the processing in step S1708 is executed. In step S1708, the position derivation unit 1206 determines whether the position in the height direction (z-axis direction) of the display area by the stencil (label) can be specified. In addition, a specific example of a case where the position in the height direction (z-axis direction) of the display area by a stencil (label) can be specified will be described later. If, as a result of the determination in step S1708, the position in the height direction (z-axis direction) of the display area by the stencil (label) cannot be specified, the process proceeds to step S1712, which will be described later.

On the other hand, if, as a result of the determination in step S1708, the position in the height direction (z-axis direction) of the display area by the stencil (label) can be specified, the process in step S1709 is executed. In step S1709, the position derivation unit 1206 derives the three-dimensional coordinates (x-y-z coordinates) of the display area by the stencil (label) as the three-dimensional position of the thick plate specified in step S1705.

Next, in step S1710, the storage unit 1207 stores the information stored in the product position column of the non-actual product display column of the management table 1300 as information about the heavy plate specified in step S1705, in step S1709. It is updated with information indicating the derived three-dimensional position of the thick plate. In this way, in this embodiment, a case is exemplified where the information stored in the product position column of the non-actual product display column of the management table 1300 is the information display position. The information stored in the item position column of the non-actual product display column of the management table 1300 is the x-y-z coordinates of the display area by the stencil (label). Therefore, the information display position is a three-dimensional position. As described above, the information display position of the same back plate changes not only in the x and y coordinates but also in the z coordinate (position in the height direction) as the position of the back plate moves.

Next, in step S1711, the output unit 1208 outputs information indicating the three-dimensional position of the thick plate specified in step S1705. For example, the output unit 1208 displays information indicating the three-dimensional position of the thick plate specified in step S1705 on a computer monitor. Then, the processing according to the flowchart in FIG. 17 ends.

Here, an example of a method of deriving the three-dimensional position of the display area by stencil (label) in step S1709 will be explained by taking the display area by stencil 201 shown in FIG. 2 as an example. In the example shown in FIG. 2, the display area 201 by the stencil is an area near the center in the width direction (y-axis direction) of the thick plate among the areas of the plate surface of the thick plate, and is on one end side in the longitudinal direction of the thick plate (x It is located in the area of the negative side of the axis. In this embodiment, the three-dimensional position of the display area 201 by the stencil is an example of the information display position.

FIG. 18 is a diagram illustrating an example of a method for deriving the three-dimensional position of the display area 201 using a stencil.

In Fig. 18, point P represents the three-dimensional position of the first imaging device (xyz coordinates of point P are (x _p , y _p , z _p )). The three-dimensional position of the first imaging device is, for example, the center position of the optical system (lens) of the first imaging device. In the example shown in FIG. 18, the z-axis is assumed to be a direction perpendicular to the surface of the thick plate (the surface on which the display area 201 by the stencil is formed). Additionally, the y-axis is assumed to be a direction parallel to the direction in which components of the same display item are arranged in the stencil display area 201. Additionally, the x-axis is assumed to be a direction parallel to the direction in which other display items are arranged in the stencil display area 201. Additionally, the x-axis, y-axis, and z-axis are assumed to be perpendicular to each other. In addition, here, to simplify the explanation, the width direction of the thick plate is assumed to be parallel to the y-axis, and the longitudinal direction of the thick plate is parallel to the x-axis.

Point S represents the position of a vertex located on the negative direction side of the x-axis and located on the positive direction side of the y-axis among the four vertices of the display area 201 by the stencil (the xyz coordinate of point S is (x _s , y _s , z _s ). Point O is the foot of the perpendicular line of point P (to the xy plane) (the intersection of the perpendicular line of point P and the xy plane). In the example shown in FIG. 18, point O is assumed to be the origin. Point Q is the intersection of a straight line passing through point O and parallel to the x-axis and a straight line passing through point S and parallel to the y-axis.

If ∠OPQ is β(rad) and the length of the line segment OP (distance from point O to point P) is H, ∠OQP(=η) is π/2-β(rad), so the line segment OQ The length (distance from point O to point Q) L is expressed by the following equation (1).

L=Hsinβ/sin{(π/2-β)} ···(1)

Additionally, if ∠OPS is α(rad), the length l of the line segment OS (the length from point O to point S) is expressed by the following equation (2). Then, if ∠SPQ is set to δ (rad), according to equation (2), the length (distance from point S to point Q) w _y of the line segment SQ is expressed as equation (3) below.

l=H/tanα···(2)

w _y =lsinδ=Hsinδ/tanα···(3)

Therefore, the xyz coordinates S(x _s , y _s , z _s ) of point S are expressed by the following equation (4).

S(x _s , y _s , z _s )=(L, w _y , 0) ···(4)

In this embodiment, a case is exemplified where the position derivation unit 1206 derives the xyz coordinates (x _s , y _s , z _s ) of the point S as the position of the display area 201 by the stencil. Once the length H, ∠OPQ(=β), ∠OPS(=α), and ∠SPQ(=δ) of the line segment OP are determined, the xyz coordinates (x _s , y _s , z _s ) of the point S are determined.

The length H of the line segment OP is a value obtained by subtracting the z coordinate z _s of the display area 201 by the stencil from the z coordinate z _p of the point P. Therefore, if the z coordinate (position in the height direction (z-axis direction) of the stencil display area 201) of the stencil display area 201 is specified, the length H of the line segment OP can be derived. 18 illustrates the case where the z coordinate z _s of the display area 201 by the stencil is 0 (zero), so the length H of the line segment OP is equal to the z coordinate z _p of the point P.

The position of point S in the first captured image is derived, for example, as follows. The position derivation unit 1206 derives a circumscribed rectangle determined by the positions of the maximum value of the x-axis, the minimum value of the x-axis, the maximum value of the y-axis, and the minimum value of the y-axis among the components of the display item by the stencil recognized in step S1702. . In addition, the position derivation unit 1206 is configured to, for example, the lower left side (the negative direction side of the x-axis and The position of the vertex located on the positive direction side of the y-axis is derived as the position of point S in the first captured image. In addition, when the first captured image is displayed by a label, the position derivation unit 1206 is viewed so that the top of the component of the item displayed by the label is located on the upper side among the vertices of the rectangular label. , for example, the vertex located on the lower left side is derived as the position of point S in the first captured image.

The position of point Q in the first captured image is a position on a straight line that passes through the position of point S in the first captured image and is parallel to the y-axis, and is the position of the horizontal angle of view (angle of view in the y-axis direction) at the time of imaging. The position corresponds to the center. Additionally, the point Q may be specified, for example, as follows. That is, laser light is irradiated from point P using a laser pointer, etc., and a straight line passing through the position of point S and parallel to the y-axis is displayed on the surface of the thick plate where the display area 201 by the stencil is formed. . Additionally, laser light is irradiated from point P using a laser pointer or the like, and a straight line passing through point O and parallel to the x-axis is displayed on the side of the thick plate where the display area 201 by the stencil is formed. In this case, the intersection of two straight lines displayed by the laser light becomes point Q. Therefore, the position derivation unit 1206 may specify the point Q from the first captured image in a state in which the intersection of two straight lines displayed by laser light is displayed. In this way, when point P, point O and point Q are on the x-z plane (on the plane of y=0), the side of the thick plate on which the display area 201 by the stencil is formed is relative to the Even if it is tilted, the position of point Q can be specified.

∠OPS(=α), ∠OPQ(=β), and ∠SPQ=∠SOQ(=δ) are, for example, the positions of point S and point Q in the first captured image acquired in step S1701, and the 1 It is determined based on the position of the center of the optical system (lens) when the image is captured (the position of the center of the optical system used to determine the focal length). The direction in which the line segment PS extends is specified by a straight line connecting the point S in the first captured image and the center position of the optical system (lens) when the first captured image is captured. The direction in which the line segment PQ extends is specified by a straight line connecting the point Q in the first captured image and the center position of the optical system (lens) when the first captured image is captured. ∠OPS(=α), ∠OPQ(=β), and ∠SPQ=∠SOQ(=δ) are determined based on the direction in which the line segment PS extends, the direction in which the line segment PQ extends, and the z-axis direction.

The method of deriving ∠OPS(=α), ∠OPQ(=β), and ∠SPQ=∠SOQ(=δ) is not limited to these methods. For example, the position derivation unit 1206 may derive ∠OPS(=α), ∠OPQ(=β), and ∠SPQ=∠SOQ(=δ) as follows. First, the position derivation unit 1206 determines the length of the line segment PS and the point PQ based on the measured value of a range sensor (for example, a time-of-flight (TOF) type range sensor) provided in the first imaging device. Derive . Then, the position derivation unit 1206 determines the positions of point S and point Q, assuming that the surface on which the display area 201 by the stencil is formed is the x-y plane, and the line segment QS is a straight line extending in the y-axis direction. Be specific. Then, the position derivation unit 1206 derives ∠OPS(=α), ∠OPQ(=β), and ∠SPQ=∠SOQ(=δ) based on the positions of point P, point S, and point Q. .

In addition, for example, the position derivation unit 1206 may determine the angle of the laser light irradiated from the first imaging device toward the point S, the angle of the laser light irradiated from the first imaging device toward the point Q, and Based on the scanning angle when the laser light is scanned from point S to point Q, ∠OPS(=α), ∠OPQ(=β), and ∠SPQ=∠SOQ(=δ) may be derived.

In addition, the position derivation unit 1206 displays the first captured image in a state in which a grid-shaped laser light is irradiated on the area including the point O and the surface on which the display area 201 by the stencil is formed among the plates of the thick plate. ∠OPS (=α), ∠OPQ (=β), and ∠SPQ=∠SOQ (=δ) can be derived by counting the number of rectangles arranged in a grid. In this case, the length in the x-axis direction of one rectangle is, for example, the length in the x-axis direction that changes when ∠OPS(=α) and ∠OPQ(=β) change by 1 rad, and the length in the y-axis direction is , for example, when ∠SPQ=∠SOQ(=δ) changes by 1 rad, it is the length in the y-axis direction that changes. Based on the results of the coefficients of the number of rectangles from the rectangle at the position corresponding to point O to the rectangle corresponding to points S and Q, ∠OPS(=α) and ∠OPQ(=β) are derived. Additionally, based on the result of the coefficient of the number of rectangles from the rectangle at the position corresponding to the point S to the rectangle at the position corresponding to the point Q, ∠SPQ(=δ) is derived. Counting the number of rectangles is performed based on the result of image processing on the first captured image.

Additionally, the positions of the display areas 1111, 1121, 1131, and 1141 are derived similarly to the method described with reference to FIG. 18. In addition, since the display area 201 by the stencil and the display area by the label 202 are different only in the surface on which they are placed on the back plate, see Figure 18 for the position of the display area by the label 202. It can be derived by imitating the method described above. Therefore, detailed description thereof is omitted here.

Next, in step S1708, an example of a case where it is determined that the position in the height direction (z-axis direction) of the display area by the stencil (label) can be specified will be described.

As a first example, first, the position derivation unit 1206 derives the distance in the height direction (z-axis direction) from the first imaging device to the display area by the stencil using a laser rangefinder. Then, the position derivation unit 1206 calculates the position obtained by subtracting the distance from the position in the height direction (z-axis direction) of the first imaging device to the height direction (z-axis direction) of the display area 201 by the stencil. Derived as the location of .

As a second example, the position derivation unit 1206 stores information indicating the position of the thick plate stored in the article position column of the management table 1300, and information stored in the registration time column of the management table 1300. , using information indicating the date and time when the back plate was placed in the current position, the position in the height direction (z-axis direction) of the display area by the stencil (label) is derived. In this way, an example of a method for deriving the position in the height direction (z-axis direction) of the display area using a stencil (label) will be described. In addition, here, the three-dimensional position of the thick plate stored in the item position column of the management table 1300 is referred to as the item position as necessary. In addition, here, to simplify the explanation, it is assumed that among the plurality of stacked thick plates, the lowest plate is placed on the ground.

First, the position derivation unit 1206 determines at least a part of the image pickup range (x-y coordinates) of the first image pickup device that captured the first image acquired in step S1701 from a thick plate other than the thick plate specified in step S1705. The thick plate is specified based on the article position stored in the management table 1300.

When deriving the position in the height direction (z-axis direction) of the display area 201 by stencil, the rear plate specified in step S1705 is at the uppermost level. Therefore, the position derivation unit 1206 determines that the rear plate at least partially included in the imaging range (x-y coordinates) of the first imaging device that captured the first captured image acquired in step S1701 is below the rear plate specified in step S1705. I judge that there is. In this case, the value obtained by adding all of the sizes (thicknesses) stored in the management table 1300 as information on the heavy plate below the heavy plate specified in step S1705 and the thickness of the heavy plate specified in step S1705 is the stencil-based This is the position in the height direction (z-axis direction) of the display area 201.

On the other hand, when deriving the position in the height direction (z-axis direction) of the display area by the label, the position derivation unit 1206 manages it as information on the thick plate specified in step S1705 among the thick plates specified as described above. It is determined that the plate stored in the registration time column that is later than the date and time stored in the registration time column of the table 1300 is the plate below the plate specified in step S1705. In this case, all the added values of the size (thickness) stored in the management table 1300 as information on the heavy plate below the heavy plate specified in step S1705 and the information on the heavy plate specified in step S1705 are stored in the management table 1300. The value obtained by adding the z-coordinate value of the stored position within the article becomes the position in the height direction (z-axis direction) of the display area of the label 202.

As a third example, when information capable of specifying the height direction (z-axis direction) is displayed in the first captured image, the position derivation unit 1206 displays a stencil (label) based on the information. The location of the area in the height direction (z-axis direction) can be derived. In this way, an example of a method for deriving the position in the height direction (z-axis direction) of the display area using a stencil (label) will be described. For example, when a structure with a scale indicating the position in the height direction (z-axis direction) is included in the first captured image, the position derivation unit 1206 recognizes the scale of the structure from the first captured image. . For example, the position derivation unit 1206 performs edge detection processing on the first captured image to extract the edge of the object, and extracts the structure from the extracted object. For example, information representing the outline of an object and information relating the name of the object to each other are registered in advance in the tracking device 1200. In this case, the position derivation unit 1206 may extract the object corresponding to the structure by comparing the information with the edge of the object extracted as described above. Additionally, the position derivation unit 1206 may use AI (Artificial Intelligence) such as NN (Neural Network) to extract the structure from the object extracted as described above.

Then, the position derivation unit 1206 determines the stencil (label) based on the position of the display area by the stencil (label) in the first captured image and the position of the scale displayed on the structure in the first captured image. Derive the position in the height direction (z-axis direction) of the display area by (label). Additionally, the numbers attached to the scale in the first captured image are recognized by a method similar to the method of recognizing the components of a display item using a stencil (label).

As a fourth example, the position derivation unit 1206 uses the second captured image acquired by the second image acquisition unit 1209 to determine the position of the display area 201 by the stencil in the height direction (z-axis direction). You can derive . In this case, in the flowchart of FIG. 17, a step of acquiring the second captured image by the second image acquisition unit 1209 is added before the timing at which the processing of step S1708 is started. The second captured image is a captured image captured by a second imaging device different from the first imaging device (an imaging device that captures an area including a display area by a stencil (label)) that captured the first captured image. The second captured image includes both the first imaging device that captured the first captured image acquired in step S1701 and the thick plate specified in step S1705. In this way, an example of a method for deriving the position in the height direction (z-axis direction) of the display area 201 using a stencil will be described.

FIG. 19 is a diagram illustrating an example of a method for deriving the position in the height direction (z-axis direction) of the display area 201 using a stencil.

In the example shown in FIG. 19 , similarly to FIG. 18 , point P represents the position of the first imaging device (xyz coordinates of point P are (x _p , y _p , z _p )). Additionally, the z-axis is a direction perpendicular to the surface of the thick plate (the surface on which the display area 201 by the stencil is formed). Additionally, the y-axis is a direction parallel to the direction in which components of the same display item are arranged in the stencil display area 201. Additionally, the x-axis is a direction parallel to the direction in which other display items are arranged in the stencil display area 201. Additionally, the x-axis, y-axis, and z-axis are assumed to be perpendicular to each other. Here too, to simplify the explanation, the width direction of the thick plate is assumed to be parallel to the y-axis, and the longitudinal direction of the thick plate is parallel to the x-axis.

In addition, point O' is the intersection of the plane containing the perpendicular line of point P (with respect to the xy plane) and the surface of the thick plate (the surface on which the display area 201 by the stencil is formed) (point O' is in FIG. corresponds to point O in 18). Point O is the foot of the perpendicular line of point P (to the xy plane) (the intersection of the perpendicular line of point P and the xy plane). In the example shown in FIG. 19, point O is assumed to be the origin. Point R represents the three-dimensional position of the second imaging device (xyz coordinates of point R are (x _r , y _r , z _r )). The three-dimensional position of the second imaging device is, for example, the center position of the optical system (lens) of the second imaging device.

It is assumed that the three-dimensional position of the second imaging device can be specified using the tracking device 1200. For example, when the 3D position of the second imaging device is fixed, the ID of the second imaging device and the 3D coordinates (x-y-z coordinates) of the second imaging device are related to each other and registered in advance in the tracking device 1200. do. In addition, when the three-dimensional position of the second imaging device is not fixed, for example, the second imaging device uses GPS to determine the three-dimensional coordinates (x-y-z coordinates) of the second imaging device and the second imaging device. Send the ID of In this case, the second image acquisition unit 1209 acquires this information transmitted from the second imaging device.

In the example shown in FIG. 19, to simplify the explanation, it is assumed that the lowest plate among the plurality of stacked plates is placed on the ground. Point D is the vertex closest to point R among the vertices of the surface on which the display area 201 by the stencil is formed. Point B is the vertex closest to point R among the vertices of the bottom of the bottom of the plurality of stacked thick plates.

In Figure 19, ∠BRD(=ϕ _B ) and ∠PRO(=ϕ _P ) are, for example, the positions of point P, point B, and point D in the second captured image, and the second captured image. It is determined based on the position of the center of the optical system (lens) at the time of image capture (the position of the center of the optical system used to determine the focal distance). The direction in which the line segment RP extends is specified by a straight line connecting the point P in the second captured image and the center position of the optical system (lens) when the second captured image is captured. The direction in which the line segment RB extends is specified by a straight line connecting the point B in the second captured image and the center position of the optical system (lens) when the second captured image is captured. The direction in which the line segment RD extends is specified by a straight line connecting the point D in the second captured image and the center position of the optical system (lens) when the second captured image is captured. ∠BRD(=ϕ _B ) is determined based on the direction in which the line segment RB extends and the direction in which the line segment RD extends. ∠PRO(=ϕ _P ) is determined based on the direction in which the line segment RP extends and the y-axis direction.

In addition, for example, the position derivation unit 1206 determines the length of the line segment RP based on the measured value of a range sensor (for example, a time-of-flight (TOF) type range sensor) provided in the second imaging device. Derive the length of line segment RD (distance from point R to point P) and line segment RD (distance from point R to point D). Then, the position derivation unit 1206 calculates the first image based on the position (z _r ) in the height direction (z-axis direction) of the second imaging device, the length of the line segment RP, and ∠PRO (=ϕ _P ). Derive the position (z _p ) in the height direction (z-axis direction) of the device (z _p = z _r + length of line segment RP × sinϕ _P ). _{Similarly ,} the position derivation unit 1206 calculates the The position of the display area 201 in the height direction (z-axis direction) is derived.

Additionally, FIG. 19 illustrates a case where the position of the second imaging device (point R) in the height direction (z-axis direction) is 0 (zero). In addition, when the second captured image is used as described above, the position derivation unit 1206 determines not only the position in the height direction (z-axis direction) of the display area 201 by the stencil, but also the height direction (z-axis direction) of the first imaging device. The position (z _p ) in the z-axis direction can be derived. In addition, the position derivation unit 1206 can derive not only the position in the height direction (z-axis direction), but also the position in the x-axis direction (x _p ) and the position in the y-axis direction (y _p ) of the first imaging device. there is. For example, x _p and y _p are derived based on the direction in which the line segment RP extends and y _r + the length of the line segment RP × cosϕ _P. In other words, a vector connecting the points R and P is derived based on the direction in which the line segment RP extends and y _r + the length of the line segment RP × cosϕ _P. The length of the x-axis component and the y-axis component of the vector are _xp and _yp .

In addition, here, the case where point P (the first imaging device) is included in the second captured image captured by the second imaging device has been described as an example. However, if the relative positional relationship (∠PRO (=ϕ _P ), etc.) between the first imaging device (point P) and the second imaging device (point R) can be derived, it is not necessarily necessary to do this. In this case, for example, a captured image captured by at least one imaging device different from the first imaging device and the second imaging device is used. A specific example of this case will be described. First, a third imaging device is placed in addition to the first and second imaging devices. The third imaging device includes point P (first imaging device) and point R (second imaging device) in its imaging range, and does not include point D in its imaging range. Additionally, the point indicating the position of the third imaging device is assumed to be R', not shown. Additionally, one or both of the positions of the point R indicating the position of the second imaging device and the position of the point R' indicating the position of the third imaging device are predetermined. In this case, the position derivation unit 1206 derives the elevation angle or depression angle when point R and point P are viewed from point R', based on the third captured image captured by the third imaging device. Then, the position derivation unit 1206 derives ∠PRO(=ϕ _P ) based on the derived elevation angle or depression angle. Derivation of the angle, direction, and length is realized by a method similar to the method described with reference to FIG. 19. Additionally, when the imaging range of the third imaging device includes point P (first imaging device) and does not include point R (second imaging device), for example, point R' (third imaging device) and the fourth captured image captured by the fourth imaging device whose imaging range includes the point R (the second imaging device) may be further used. On the other hand, for example, if the imaging range of the third imaging device includes point R (second imaging device) and does not include point P (first imaging device), for example, point R' (third imaging device) The fifth captured image captured by the fifth imaging device whose imaging range includes the point P (the first imaging device) and the point P (the first imaging device) may be further used.

Additionally, the position derivation unit 1206 may specify the position of point P (the first imaging device) using the sixth captured image captured by the sixth imaging device that includes the first imaging device in the imaging range. . In this case, it is assumed that the sixth imaging device is included in at least one imaging range of the second imaging device, the third imaging device, and the fourth imaging device. Additionally, the point indicating the position of the sixth imaging device is assumed to be P', not shown. Additionally, one or both of the positions of the point P indicating the position of the first imaging device and the position of the point P' indicating the position of the sixth imaging device are predetermined. In this case, for example, in the above description, calculations are made using the sixth imaging device and point P' instead of the first imaging device and point P. And, for example, based on the results of this calculation, the sixth captured image, and the positions of one or both of the point P and point P', the first imaging device (point P) and the second imaging device (point R) The relative positional relationship (∠PRO(=ϕ _P ), etc.) is derived.

In the above manner, even when the second imaging device cannot be arranged so that the imaging range of the second imaging device includes both the first imaging device and the rear plate, the height direction (z-axis) of the display area 201 by the stencil is direction) can be derived. Additionally, the third to sixth captured images are acquired by, for example, the second image acquisition unit 1209. However, separately from the second image acquisition unit 1209, third to sixth image acquisition units that individually acquire the third to sixth captured images may be used. In this case, in the flowchart of FIG. 17, the timing is before the timing at which the processing of step S1708 is started, and the timing is different from the timing at which the step of acquiring the second captured image by the second image acquisition unit 1209 is executed. A step for individually acquiring the third to sixth captured images is added. Additionally, if the position in the height direction (z-axis direction) of the display area by the stencil (label) can be specified without acquiring the second captured image or the like, the second image acquisition unit 1209 becomes unnecessary.

As a fifth example, the position derivation unit 1206 conveys the position of the thick plates 1002a and 1002b placed on the conveyance table 1001 in the height direction (z-axis direction) of the display area 201 by the stencil. It may be derived based on the position of the table 1001 and the size (thickness) stored in the management table 1300 as information on the thick plates 1000a and 1002b. In this case, the position of the transfer table 1001 is registered in advance with the tracking device 1200.

For example, in at least one of the above examples, when the position in the height direction (z-axis direction) of the display area by the stencil (label) can be derived, the display by the stencil (label) is performed in step S1708. It is determined that the position in the height direction (z-axis direction) of the area can be specified.

Additionally, when the thick plate is in the carrying tool, in step S1709, the position derivation unit 1206 acquires the three-dimensional position of the thick plate in the carrying tool. In this embodiment, the transport tools are a transport ship 400 and a transport vehicle 700. Therefore, in this embodiment, the case where the transport tools are the transport ship 400 and the transport vehicle 700 is exemplified.

For example, when a heavy plate is loaded on the transport ship 400, the position derivation unit 1206 acquires the ID of the heavy plate and information indicating the three-dimensional position of the heavy plate in the transportation ship 400. Then, the position derivation unit 1206 specifies a row in which information indicating the same ID as the ID of the acquired thick plate is stored from the ID column of the management table 1300. Then, the position derivation unit 1206 stores information indicating the acquired three-dimensional position of the thick plate in the transport ship 400 in the item position column of a specific row. For example, even if the operator (worker) of the transport ship 400 inputs the ID of the heavy plate and information indicating the three-dimensional position of the heavy plate in the transport ship 400 into an information processing terminal (tablet terminal, etc.) possessed by the operator (worker) of the transport ship 400, do. In this case, by transmitting the ID of the heavy plate and information indicating the three-dimensional position of the heavy plate in the transport ship 400 to the tracking device 1200 from the information processing terminal, the position derivation unit 1206 determines the heavy plate in question. Information indicating the ID and the three-dimensional position of the heavy plate in the transport ship 400 may be acquired. Information indicating the three-dimensional position of the thick plate is, for example, the x-y-z coordinates of the display area by the stencil (label).

Additionally, when a heavy plate is loaded on the transport vehicle 700, the position derivation unit 1206 acquires the ID of the heavy plate and information indicating the three-dimensional position of the heavy plate in the transportation vehicle 700. Then, the position derivation unit 1206 specifies a row in which information indicating the same ID as the ID of the acquired thick plate is stored from the ID column of the management table 1300. Then, the position derivation unit 1206 stores information indicating the acquired three-dimensional position of the thick plate in the transport vehicle 700 in the item position column of a specific row. For example, information indicating the ID of the heavy plate and the three-dimensional position of the heavy plate in the transportation vehicle 700 is stored in an information processing terminal (tablet terminal, etc.) held by the operator (worker) of the transportation vehicle 700. You may enter it. In this case, by transmitting the ID of the heavy plate and information indicating the three-dimensional position of the heavy plate in the transport vehicle 700 from the information processing terminal to the tracking device 1200, the position derivation unit 1206: Information indicating the ID of the thick plate and the three-dimensional position of the thick plate in the transport vehicle 700 may be acquired. Information indicating the three-dimensional position of the thick plate is, for example, the x-y-z coordinates of the display area by the stencil (label).

By acquiring the three-dimensional position of the thick plate in the conveyance tool as described above, the conveyance tool transporting each thick plate can be specified. Additionally, the three-dimensional position of the thick plate while it is being transported by the transport tool can be accurately specified.

Returning to the explanation of Fig. 17, if it is determined in step S1708 that the position in the height direction (z-axis direction) of the display area by the stencil (label) cannot be specified, the process of step S1712 is executed. In step S1712, the output unit 1208 outputs information indicating that the three-dimensional position of the thick plate cannot be specified and the ID of the first imaging device that captured the image of the thick plate. For example, the output unit 1208 causes the computer monitor to display information indicating that the three-dimensional position of the thick plate cannot be specified and the ID of the first imaging device that captured the image of the thick plate. When these information are output, the operator (worker) goes to the location where the corresponding first imaging device is located and specifies the three-dimensional position of the corresponding thick plate. Additionally, the operator (worker) may specify the three-dimensional position of the corresponding thick plate from the preview image of the corresponding first imaging device.

Next, in step S1713, the storage unit 1207 inputs information indicating the three-dimensional position of the heavy plate specified by the operator (worker), and displays the non-actual product in the management table 1300 as information about the heavy plate. The information stored in the item location column is updated with the information entered. In addition, as a form of inputting information indicating the three-dimensional position of the thick plate, input through manipulation of the user interface of the tracking device 1200, reception of information transmitted from an external device, and reading of information stored in a portable storage medium. , at least one can be employed. Then, the processing according to the flowchart in FIG. 17 ends.

<Integration>

As described above, in this embodiment, the tracking device 1200 is the information displayed on the thick plate, and the information managed by the manufacturer of the thick plate is used to determine the area including the display area displayed by the stencil (label). 1 Acquire the captured image. The tracking device 1200 derives the three-dimensional position of the display area by the stencil (label) based on the acquired first captured image and the position of the first imaging device that captured the first captured image, and determines the three-dimensional position of the display area by the stencil (label). Information indicating the three-dimensional position is stored in a storage medium in association with the ID of the thick plate. Therefore, for example, when the heavy plate is conveyed using various conveyance devices (transport ship 400, transport vehicle 700, conveyance table 1001, etc.), or when the heavy plate is conveyed by workers, or when the plate is transported by workers, the place address is Even when the plates are placed in an unspecified location or when the plates are stacked, tracking is possible. Therefore, the tracking precision of the thick plate can be improved.

Additionally, in this embodiment, the tracking device 1200 recognizes the information (ID305, 312) displayed in the display area by the stencil (label) based on the first captured image. Then, the tracking device 1200 compares the recognized information with the information (ID) stored in the management table 1300, and specifies the rear plate based on the collation result. Therefore, the thick plate itself, which is the target of tracking, can be reliably specified based on the first captured image. Therefore, compared to the case where the three-dimensional position of the thick plate is specified based on the operation of the conveying equipment, for example, the tracking accuracy of the thick plate can be improved.

Additionally, in this embodiment, the tracking device 1200 acquires a second captured image captured by a second image pickup device different from the first image pickup device that captured the first captured image. The second captured image includes the first captured image and the thick plate (targeted for tracking). The tracking device 1200 derives the position of the display area 201 by the stencil in the height direction (z-axis direction) based on the acquired second captured image and the position of the second imaging device. Then, the tracking device 1200 is positioned at the position in the height direction (z-axis direction) of the display area 201 by the stencil, the first captured image, and the position of the first imaging device that captured the first captured image. Based on this, the three-dimensional position of the display area 201 by the stencil is derived as the three-dimensional position of the thick plate. Therefore, even when the position in the height direction of the thick plate cannot be derived even using the first captured image or the sensor provided in the first imaging device, the thick plate can be tracked.

Additionally, in this embodiment, the display using a stencil and the display using a label are made into information that can be identified by humans. Therefore, management of the actual product by the operator (worker) becomes easy. Accordingly, the convenience of managing actual items is improved.

Additionally, in this embodiment, the display using the stencil and the display using the label include information determined individually for each plate (information unique to the plate). Therefore, based on the first captured image, the thick plate can be uniquely identified. Therefore, identification of the thick plate can be performed easily and quickly.

Additionally, in this embodiment, the tracking device 1200 derives three-dimensional coordinates (x-y-z coordinates) as the position of the display area by the stencil (label). Therefore, for example, even when a plurality of thick plates are stacked, the thick plates can be tracked.

<Variation example>

<<First modified example>>

The method of deriving the position of the display area by the stencil (label) included in the first captured image in step S1709 is not limited to the method explained with reference to FIG. 18. For example, the position derivation unit 1206 derives the origin position when determining the position within the article in the management table 1300 from the first captured image. Then, the position derivation unit 1206 determines the position of the origin by the stencil (label) included in the first captured image, based on the derived position of the origin and the information stored in the position within the article column of the management table 1300. Derive the location of the display area. In this way, an example of a method of deriving the position of the display area by stencil (label) included in the first captured image will be described by taking the display area by stencil 201 as an example.

FIG. 20 is a diagram illustrating an example of a method for deriving the position of the display area 201 using a stencil. In FIG. 20 , as in FIG. 18 , a case in which the xyz coordinates S (x _s , y _s , z _s ) of point S are derived as the position of the display area 201 by the stencil is exemplified. As explained with reference to FIG. 14, the position of the stencil in the article takes the vertex 203 shown in FIG. 2 as the origin, and from the vertex 203, the direction along the width direction of the thick plate is the positive direction of the y-axis. It is expressed as an xy coordinate with the direction along the longitudinal direction of the thick plate being the positive x-axis. Therefore, the position of point E in Figure 20 becomes the origin of the stencil's position within the article. Point F is a point arranged in the width direction (y-axis direction) with respect to point E among the vertices of the thick plate (point E and point F form one side along the width direction of the thick plate). Point T is the foot of the perpendicular line of point P to the line segment EF (the intersection of the perpendicular line of point P and the line segment EF). Point P and point O are the same as those shown in FIG. 18.

In Fig. 20, the length of the line segment PT is h _p . Additionally, let the length of the line segment PF be h ₁ , ∠FPT be θ ₁ (rad), the length of the line segment PE be h ₂ , ∠EPT be θ ₂ (rad), and ∠EPF be θ (rad). )(θ=θ ₁ +θ ₂ ). Additionally, the length of the line segment FT is set to w ₀₁ , and the length of the line segment ET is set to w ₀₂ .

Then, the x coordinate x _e of point E becomes the x coordinate x _p of point P plus the length l _p of the line segment OT (x _e =x _p +l _p ). The y coordinate y _e of point E becomes the value obtained by subtracting the length w ₀₂ of the line segment ET from the y coordinate y _p of point P (y _e = y _p -w ₀₂ ). Accordingly, if the length l _p of the line segment OT and the length w ₀₂ of the line segment ET are derived, the xy coordinates (x _e , y _e ) of the position of point E (the origin of the position of the stencil in the article) are derived.

The length l _p of the line segment OT is expressed by the following equation (5).

l _p =√(h _p ² -H ² )=√(h ₁ ² cos ² θ ₁ -H ² )=√(h ₂ ² cos ² θ ₂ -H ² )···(5)

The length w ₀₂ of the line segment ET is expressed by the following equation (6).

w ₀₂ =h ₂ sinθ ₂ ···(6)

Therefore, for example, if the length h ₂ and ∠EPT(=θ ₂ ) of the line segment PE are derived, the length l _p of the line segment OT and the length w ₀₂ of the line segment ET are derived. Additionally, the length H of the line segment OP is derived as described with reference to FIG. 18.

The position of point T in the first captured image is a position on the line segment EF in the first captured image, and is a position corresponding to the center of the horizontal angle of view (angle of view in the y-axis direction) at the time of imaging. ∠EPT(=θ ₂ ) is, for example, the positions of point E and point T in the first captured image acquired in step S1701, and the center position of the optical system (lens) when the first captured image was captured ( It is determined based on the location of the center of the optical system used to determine the focal length. The direction in which the line segment PE extends is specified by a straight line connecting the point E in the first captured image and the center position of the optical system (lens) when the first captured image is captured. The direction in which the line segment PT extends is specified by a straight line connecting the point T in the first captured image and the center position of the optical system (lens) when the first captured image is captured. ∠EPT(=θ ₂ ) is determined based on the direction in which the line segment PE extends and the direction in which the line segment PT extends.

The length h ₂ of the line segment PE is derived, for example, using a range sensor (for example, a time-of-flight (TOF) type range sensor) provided in the first imaging device. In addition, the length h ₂ of the line segment PE may be derived based on the length H of the line segment OP and ∠OPE.

As described above, the x coordinate x _e and y coordinate y _e of point E are derived. Then, the position derivation unit 1206 determines the xy coordinates (Ls10, Ws10) is read. Then, the position derivation unit 1206 derives the value obtained by adding Ls10 to the x coordinate x _e of point E as the x coordinate x _s of point S (x _s =x _e +Ls10). Additionally, the position derivation unit 1206 derives the value obtained by adding Ws10 to the y coordinate x _y of point E as the y coordinate y _s of point S (y _s = y _e + Ws10).

<<Second Modification Example>>

As in this embodiment, the demand name (302), standard (303), size (304), ID (305), demand price code (306), order number (307), standard (311), ID (312), It is preferable to use the size 313, delivery date 314, and demand code 315 as human-identifiable information because it makes it easier for the operator to manage the actual product. However, the demand name (302), specification (303), size (304), ID (305), demand price code (306), order number (307), specification (311), ID (312), size (313) ), delivery date 314, and demand price code 315 may be converted (encoded) into information that cannot be identified by humans, such as a barcode or two-dimensional code.

<<Third modification>>

In this embodiment, as shown in FIGS. 3A and 3B, the case where the ID 305 displayed by the stencil and the ID 312 displayed by the label are the same is exemplified. However, the expression and description contents of these IDs may be different. For example, there is a format without a "hyphen" in the ID. In this case, information indicating that these IDs (ID by stencil, ID by label) are for the same thick plate is stored in the management table 1300. For display items other than ID, the expression and description content of the same display item may be different between display using a stencil and display using a label. For example, when the standard is displayed on both the stencil and the label, for example, the label may display an abbreviated name for the standard. In addition, although the number of digits of the consumer code may be different between display using a stencil and display using a label, the consumer code is managed by the manufacturer. Also, regarding size, there are cases where all digits of the dimension are not indicated. Even in this case, information indicating that the different expressions and descriptions have the same meaning is stored in the management table 1300.

<<Fourth Modification>>

In this embodiment, the case where the xyz coordinates of the display area by the stencil (label) are derived as the three-dimensional position of the display area by the stencil (label) is exemplified. However, the three-dimensional position of the display area by the stencil (label) is not limited to xyz coordinates. For example, the three-dimensional position of the display area by the stencil (label) may be a relative three-dimensional position based on the position of the first imaging device. In this case, there is no need to use the coordinates (x _p , y _p , z _p ) of point P, which represents the three-dimensional position of the imaging device. In addition, when the origin of the xyz coordinates is the 3D position of the first imaging device, the 3D position of the display area by the stencil (label) is based on the 3D position of the first imaging device, even if expressed in xyz coordinates. It represents the relative three-dimensional position.

<<Fifth Modification>>

In this embodiment, the case where the identification information acquisition unit 1210 has a recognition unit 1202, an acquisition unit 1203, a matching unit 1204, and a specification unit 1205 is exemplified. However, when the identification information acquisition unit 1210 acquires the identification information of the thick plate associated with the three-dimensional position derived by the position derivation unit 1206, the recognition unit 1202, the acquisition unit 1203, It is not necessary to have a matching section 1204 and a specification section 1205. The identification information acquisition unit 1210 may have only the recognition unit 1202, for example. Additionally, the identification information acquisition unit 1210 may have only the acquisition unit 1203, for example. Additionally, the identification information acquisition unit 1210 may have only the recognition unit 1202 and the acquisition unit 1203, for example. For example, the identification information acquisition unit 1210 may acquire the ID recognized by the recognition unit 1202 as identification information for the thick plate. In this case, the identification information acquisition unit 1210 does not need to have an acquisition unit 1203, a matching unit 1204, and a specification unit 1205. In addition, for example, when the object of tracking is limited to one item in advance, the identification information acquisition unit 1210 may acquire the ID obtained by the acquisition unit 1203 as identification information for the thick plate. In this case, the identification information acquisition unit 1210 does not need to have the recognition unit 1202, the matching unit 1204, and the specifying unit 1205. In addition, for example, the identification information acquisition unit 1210 uses the recognition unit 1202 when specifying a thick plate only from the information recognized by the recognition unit 1202, and the items to be tracked are grouped into one in advance. In limited cases, the water intake unit 1203 may be used. In this case, the identification information acquisition unit 1210 does not need to have the matching unit 1204 and the specifying unit 1205 (it may only have the recognition unit 1202 and the acquisition unit 1203).

<<Variation 6>>

In this embodiment, the case where the management table 1300 is stored in the tracking device 1200 is exemplified. However, the management table 1300 may be outside the tracking device 1200.

<<Variation 7>>

In this embodiment, the case where the product is a thick plate is exemplified. However, the article is not limited to thick plates. For example, the article may be a product other than a thick plate. The article may be a product on which information managed by the manufacturer is displayed. The manufactured product may be, for example, a slab, billet, sheet coil or steel pipe. Additionally, it may be an article other than a manufactured product. The article may be a container, for example.

<<Other variations>>

Additionally, the embodiments of the present invention described above can be realized by a computer executing a program. Additionally, a computer-readable recording medium recording the above program and a computer program product such as the above program can also be applied as an embodiment of the present invention. As a recording medium, for example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, magnetic tape, non-volatile memory card, ROM, etc. can be used. Additionally, the embodiments of the present invention may be realized by a PLC (Programmable Logic Controller) or by dedicated hardware such as an ASIC (Application Specific Integrated Circuit).

In addition, the embodiments of the present invention described above are merely examples of embodiments of carrying out the present invention, and the technical scope of the present invention should not be construed as limited by these. That is, the present invention can be implemented in various forms without departing from its technical idea or main features.

<Hardware>

An example of the hardware of the tracking device 1200 will be described. In Figure 21, the tracking device 1200 includes a CPU 2101, a main memory 2102, an auxiliary memory 2103, a communication circuit 2104, a signal processing circuit 2105, an image processing circuit 2106, It has an I/F circuit 2107, a user interface 2108, a display 2109, and a bus 2110.

The CPU 2101 centrally controls the entire tracking device 1200. The CPU 2101 uses the main memory 2102 as a work area and executes the program stored in the auxiliary memory 2103. The main memory device 2102 temporarily stores data. The auxiliary memory device 2103 stores various data in addition to programs executed by the CPU 2101.

The communication circuit 2104 is a circuit for communicating with the outside of the tracking device 1200. The communication circuit 2104 may perform wireless or wired communication with the outside of the tracking device 1200.

The signal processing circuit 2105 performs various signal processing on signals received from the communication circuit 2104 or signals input according to control by the CPU 2101.

The image processing circuit 2106 performs various image processing on the input signal under control by the CPU 2101. The signal on which this image processing has been performed is output to the display 2109, for example.

The user interface 2108 is a part where the operator gives instructions to the tracking device 1200. The user interface 2108 has, for example, buttons, switches, and dials. Additionally, the user interface 2108 may have a graphical user interface using the display 2109.

The display 2109 displays an image based on a signal output from the image processing circuit 2106. The I/F circuit 2107 exchanges data between devices connected to the I/F circuit 2107. FIG. 21 shows a user interface 2108 and a display 2109 as devices connected to the I/F circuit 2107. However, devices connected to the I/F circuit 2107 are not limited to these. For example, a portable storage medium may be connected to the I/F circuit 2107. Additionally, at least a portion of the user interface 2108 and the display 2109 may be outside the tracking device 1200.

The output unit 1208 uses, for example, at least one of the communication circuit 2104, the signal processing circuit 2105, the image processing circuit 2106, the I/F circuit 2107, and the display 2109. It comes true.

Additionally, the CPU 2101, the main memory 2102, the auxiliary memory 2103, the signal processing circuit 2105, the image processing circuit 2106, and the I/F circuit 2107 are connected to the bus 2110. . Communication between these components is performed via bus 2110. Additionally, the hardware of the tracking device 1200 is not limited to that shown in FIG. 21 as long as it can realize the functions of the tracking device 1200 described above.

The present invention can be used for tracking goods, for example.

Claims (8)

It is a tracking device that tracks goods,
one or more image acquisition means for acquiring a captured image of an area containing display information displayed on the article;
Identification information acquisition means for acquiring identification information of the article;
Position derivation means for deriving an information display position, which is a three-dimensional position where the display information is displayed, based on the captured image acquired by the image acquisition means;
characterized by comprising storage means for storing the information display position on the article derived by the position derivation means in a storage medium in relation to the identification information of the article acquired by the identification information acquisition means. Tracking device. The method according to claim 1, wherein the display information displayed on the article includes identification information of the article,
The identification information acquisition means is,
Recognition means for recognizing identification information of the article based on the captured image acquired by the image acquisition means;
A tracking device comprising at least one acquisition means for obtaining identification information of the article stored in the storage means. The method according to claim 2, wherein the information display position stored in the storage medium in relation to the identification information of the article is information about a three-dimensional position where the display information is displayed, and is based on a predetermined position of the article. Contains information about the three-dimensional position,
A tracking device, wherein the captured image acquired by the image acquisition means includes an area containing a predetermined position of the article. The method of claim 2 or 3, wherein the identification information acquisition means is:
The recognition means,
The means of obtaining the above,
a collation means for comparing the identification information of the article recognized by the recognition means with the identification information of the article obtained by the acquisition means;
A tracking device comprising: specific means for specifying identification information of the article included in the captured image acquired by the image acquisition means, based on a result of the collation by the collation means. The method according to any one of claims 1 to 3, wherein the position deriving means determines the surface of the article on which the display information is displayed based on two or more captured images acquired by the two or more image acquisition means. A tracking device characterized in that it derives a position in the height direction. It is a tracking method for tracking goods,
One or more image acquisition steps of acquiring a captured image of an area containing display information displayed on the article;
An identification information acquisition process of acquiring identification information of the article;
A position derivation step of deriving an information display position, which is a three-dimensional position where the display information is displayed, based on the captured image acquired by the image acquisition step;
and a storage step of storing the information display position on the article derived by the position derivation step in a storage medium in relation to the identification information of the article obtained by the identification information acquisition step. Tracking method. It is a computer program stored on a recording medium for executing the processing of tracking goods on a computer,
One or more image acquisition steps of acquiring a captured image of an area containing display information displayed on the article;
An identification information acquisition process of acquiring identification information of the article;
A position derivation step of deriving an information display position, which is a three-dimensional position where the display information is displayed, based on the captured image acquired by the image acquisition step;
causing the computer to execute a storage step of storing the information display position on the article derived by the position derivation step in a storage medium in relation to the identification information of the article obtained by the identification information acquisition step. A computer program stored on a recording medium characterized by: delete

Images