KR20230082662A - Tracking device, tracking method, data structure and program of tracking data - Google Patents

Abstract

The tracking device 1200 derives the 3D position of the display area using the stencil (label) based on the first captured image and the position of the first imaging device that captured the first captured image, and determines the 3D position The information indicating is stored in the storage medium in association with the identification information of the thick plate.

Description

Tracking device, tracking method, data structure and program of tracking data

The present invention relates to a tracking device, a tracking method, a data structure and a program for tracking data, and is particularly suitable for use in tracking an article. This application claims priority based on Japanese Patent Application No. 2020-198065 for which it applied to Japan on November 30, 2020, and uses all their contents here.

Products such as steel products, including thick plates, move to various places. So, tracking the product is done. Tracking is to track and specify the position of an object.

In the technique described in Patent Literature 1, the steel sheet is tracked by registering the X-Y coordinates of an overhead crane when the steel sheet is lowered together with the steel sheet number of the steel sheet concerned.

In addition, in the technique described in Patent Document 2, based on the movement of the crane in the horizontal and vertical directions and operations such as load detection, a stationary place calculation program calculates the stationary place of steel materials represented by three-dimensional coordinates, , the stationary location information indicating the calculated stationary location is matched with the steel material identification information of the steel materials concerned, and stored.

Japanese Patent No. 5109451 Japanese Patent No. 3734419

However, in the technology described in Patent Literature 1, the information used for steel plate tracking is the X-Y coordinates of an overhead crane. For this reason, tracking can be performed only in the range which can be conveyed by an overhead crane. In addition, in the technology described in Patent Document 1, since the Z coordinate cannot be specified from the X-Y coordinates of the overhead crane, for example, when a plurality of steel plates overlap, the plurality of steel plates cannot be accurately specified.

In addition, in the technique described in Patent Literature 2, the three-dimensional coordinates of steel materials are calculated. However, similarly to Patent Literature 1, tracking can be performed only within the range that can be transported by crane. In addition, the 3-dimensional coordinates of the steel materials calculated by the technique described in patent document 2 are the 3-dimensional coordinates of the stationary place calculated by the stationary place calculation program. Therefore, with the technique described in patent document 2, steel materials can be tracked only in the stationary place previously determined by the stationary place calculation program.

In actual manufacturing sites, products are conveyed by various conveying devices. In addition, a worker conveys a product without using a conveying device. In addition, there are cases in which products are placed in stationary places where the address of the place is not determined. In addition, the method of placing the product is also diverse. In the techniques described in Patent Literatures 1 and 2, it is not easy to track such products. These subjects exist similarly also in articles other than manufactured goods.

The present invention has been made in view of the above problems, and aims to improve the tracking accuracy of articles.

A tracking device of the present invention is a tracking device for tracking an article, comprising: one or two or more image acquisition means for acquiring a captured image of an area including display information displayed on the article; and acquiring identification information of the article. identification information acquisition means, and position derivation means for deriving an information display position, which is a three-dimensional position at which the display information is displayed, based on the captured image acquired by the image acquisition means, and deriving by the position derivation means and storage means for storing, in a storage medium, the information display position of the article obtained in association with the identification information of the article acquired by the identification information acquisition means.

A tracking method of the present invention is a tracking method for tracking an article, and includes one or more image acquisition steps of acquiring a captured image of an area including display information displayed on the article, and acquiring identification information of the article. An identification information acquisition step, a position derivation step of deriving an information display position that is a three-dimensional position at which the display information is displayed based on the captured image acquired by the image acquisition step, and derivation by the position derivation step and a storage step of storing, in a storage medium, the information display position of the article obtained in association with the identification information of the article obtained in the identification information acquisition step.

The data structure of the tracking data of the present invention has display information displayed on an article and non-display information not displayed on the article, and the non-display information is an information display position that is a three-dimensional position where the display information is displayed. , wherein the tracking device is used to derive the information display position on the article based on a captured image of an area including display information displayed on the article.

The program of the present invention is a program for causing a computer to execute processing for tracking an article, comprising: one or two or more image acquisition steps of acquiring a captured image of an area including display information displayed on the article; an identification information acquisition step of obtaining identification information; a position derivation step of deriving an information display position, which is a three-dimensional position at which the display information is displayed, based on the captured image acquired by the image acquisition step; and the position characterized by causing a computer to execute a storage step of storing the information display position of the article derived by the derivation step in a storage medium in association with the identification information of the article acquired by the identification information acquisition step. .

1 is a diagram for explaining an example of a product flow in a factory.
2 is a diagram showing an example of a display area for a rear plate.
Fig. 3A is a diagram showing an example of display contents using a stencil.
Fig. 3B is a diagram showing an example of contents displayed by labels.
Fig. 4 is a diagram showing an example of how the heavy plate is loaded and unloaded with respect to the transport ship.
Fig. 5 is a view of a state in which the back plate is suspended from the loading and unloading device, as viewed from the garter extension direction (y-axis direction).
Fig. 6A is a diagram showing a first example of a transport form of a thick plate.
Fig. 6B is a diagram showing a second example of a heavy plate conveyance mode.
Fig. 6C is a diagram showing a third example of a heavy plate conveyance mode.
Fig. 6D is a diagram showing a fourth example of a heavy plate conveyance mode.
Fig. 6E is a diagram showing a fifth example of a heavy plate conveyance mode.
Fig. 7 is a diagram showing an example of how a heavy plate is loaded and unloaded with respect to a transport vehicle.
8 is a diagram showing an example of a state in which a heavy plate is loaded on a transport vehicle.
9A is a diagram showing a first example of a state of a thick plate placed in a place in a factory.
9B is a diagram showing a second example of a state of a thick plate placed in a place in a factory.
Fig. 9C is a diagram showing a third example of the state of a thick plate placed in a place in a factory.
10 is a diagram showing an example of how thick plates are processed in a production facility.
Fig. 11 is a diagram showing an example of the original thick plate and the thick plate after cutting.
Fig. 12 is a diagram showing an example of a functional configuration of a tracking device.
13 is a diagram showing an example of a management table.
Fig. 14 is a diagram showing an example of a position of a stencil in an article and a position of components of each display item in a display using a stencil.
Fig. 15 is a diagram showing an example of the position of a label in an article and the position of components of each display item in a label display.
16A is a diagram showing a first example of information stored in a management table.
16B is a diagram showing a second example of information stored in the management table.
Fig. 17 is a flowchart for explaining an example of tracking device processing when tracking a thick plate.
18 is a diagram for explaining an example of a method of deriving a three-dimensional position of a display area using a stencil.
19 is a diagram for explaining an example of a method for deriving a position in the height direction (z-axis direction) of a display area using a stencil.
20 is a diagram for explaining a modified example of a method for deriving a three-dimensional position of a display area using a stencil.
Fig. 21 is a diagram showing an example of the hardware configuration of the tracking device.

EMBODIMENT OF THE INVENTION Hereinafter, one Embodiment of this invention is described, referring drawings.

In addition, those with the same comparison target, such as length, position, size, spacing, etc., are different within a range that does not deviate from the scope of the invention (for example, different within the range of tolerances determined at the time of design), except when they are strictly identical. that) is also included.

<Plant overview>

In this embodiment, a case where the heavy plate is tracked in a place including a factory that performs various treatments such as heat treatment and cutting for various kinds of thick plates (for example, thick steel plates) manufactured at a plurality of steelworks is exemplified. . 1 is a diagram for explaining an example of a product flow in a factory to which the present embodiment is applied.

In Fig. 1, thick plates manufactured in a steel mill are loaded onto a transport ship. After that, the transport ship lands on the quay near the factory. The heavy plate is unloaded from a transport ship and loaded onto a transport vehicle (such as a truck). When the heavy plate transported by the transport vehicle arrives at the factory, it is placed in a place within the factory. The thick plate placed in the storage area is placed in a warehouse after processing such as heat treatment or cutting (gas cutting) is performed. After that, the thick plate is shipped from the factory to the customer.

<Imagination>

When the position of the thick plate conveyed as described above is tracked based on the operation of the conveying machine as in the techniques described in Patent Literatures 1 and 2, restrictions arise in the range that can be tracked. Then, the present inventors paid attention to the fact that information managed by the manufacturer of the thick plate is displayed on the back plate for actual product management (management of the position of each back plate). Tracking the display position of such information is equivalent to tracking the back plate. Then, if the information managed by the manufacturer of such a thick plate can be specified, the thick plate can be identified. This embodiment is made based on such thought.

<Indication applied to heavy plate>

2 is a diagram showing an example of a display area for a rear plate. In addition, the x-axis, y-axis, and z-axis shown in FIG. 2 show the relationship of directions between FIG. 2 and FIGS. 3A, 3B, 14 to 15, and 18 to 20 described later, and FIG. 4 and It does not necessarily correspond to the directions of the x-y-z coordinates shown in FIG. In addition, x-y-z coordinates are coordinates displayed in the world coordinate system, for example.

In FIG. 2 , a display area 201 is a display area using a stencil. In the example shown in FIG. 2 , the display area 201 using the stencil exists on the plate surface of the back plate. In addition, the broken line shown in FIG. 2 is a virtual line and is not actually displayed. In the example shown in FIG. 2 , a label (side label) 202 is affixed to the side surface (plate thickness portion) of the back plate.

Fig. 3A is a diagram showing an example of display contents using a stencil. Incidentally, as described above, the x-y-z coordinates shown in FIG. 3A indicate a directional relationship with FIG. 2 and the like. Also, in the x-y-z coordinates shown in Fig. 3A, a symbol with a black circle (●) in a white circle (○) indicates a direction from the inner side of the paper to the front side. The notation of these x-y-z coordinates is the same in other drawings.

In the example shown in FIG. 3A , the display by the stencil includes a mark 301, a customer name 302, a specification 303, a size 304, an ID 305, a customer code 306, and an order number. (307) is included.

A mark 301 is a mark indicating the manufacturer of the thick plate. The customer's name 302 is information indicating the customer's price (purchaser) of the heavy plate. The standard 303 is information indicating the standard of the thick plate. The size 304 is information indicating the size (for example, thickness x width x length) of the back plate. ID 305 is information for uniquely identifying the thick plate, and in the example shown in Fig. 3A, it is the plate number (identification number of the thick plate). Therefore, the same ID is not attached to different thick plates. The demand code 306 is information specified to be attached to the heavy plate by the manufacturer of the heavy plate from the customer. The order number 307 is a part of a number for identifying a heavy plate order from a customer.

The mark 301, customer name 302, specification 303, size 304, ID 305, customer code 306 and order number 307 can be commonly displayed for thick plates. It is information. In addition, information other than ID 305 need not be included in the display by the stencil. In addition, information other than the above information may be included in the display by the stencil.

Further, for example, the number of digits of the demand code 306 differs depending on the demand, or the notation method of the standard 303 differs depending on the steel mill that manufactures the steel plate. In addition, for example, the order number 307 may or may not be attached depending on the demand. In addition, the size of the display item may also differ depending on the steel mill that manufactures the thick plate. In addition, the position of the display area by the stencil may also differ depending on the steel mill that manufactures the thick plate. In this way, the content, location, and size of the mark by the stencil differ depending on the steel mill that manufactures the thick plate and the demand.

The mark 301, customer name 302, specification 303, size 304, ID 305, customer code 306, and order number 307 are information managed by the heavy plate manufacturer. . In addition, the mark 301, customer name 302, specification 303, size 304, ID 305, customer code 306, and order number 307 are information that can be identified by a person. Information that can be identified by humans means that the meaning of the information can be understood when a person sees the information. Information that requires machine encoding and decoding to produce and read information, such as barcodes and two-dimensional codes, is not human-identifiable information. Specifically, the mark 301, customer name 302, specification 303, size 304, ID 305, customer code 306, and order number 307 are decoded by a machine upon reading. is unnecessary information, and includes at least one of letters, numbers, and symbols, and does not include information requiring decoding by a machine for reading.

Fig. 3B is a diagram showing an example of contents displayed by labels. Incidentally, as described above, the x-y-z coordinates shown in FIG. 3B indicate a directional relationship with FIG. 2 and the like. Further, in the x-y-z coordinates shown in Fig. 3B, a symbol appended with a cross in a white circle indicates a direction from the front side of the paper to the back side. The notation of these x-y-z coordinates is the same in other drawings.

In the example shown in Fig. 3B, the display by the label includes specifications 311, ID 312, size 313, delivery date 314, and demand code 315. The delivery date 314 is information indicating the delivery date of the thick plate. Specification 311, ID 312, size 313, and demand code 315 are the same as standard 303, size 304, ID 305, and demand code 306, respectively. Information other than the ID 312 need not be included in the label display. In addition, information other than the above information may be included in the display by the label. The content, position, and size of the marking by the label, as in the marking by the stencil, also differ depending on the steel mill that manufactures the thick plate and the demand.

However, the meaning of the same display items for the same thick plate is the same for display using a stencil and display using a label. Therefore, when the stencil display shown in FIG. 3A and the label display shown in FIG. 3B are performed on the same back plate, the ID 305 displayed by the stencil and the ID 312 displayed by the label will mean the same thing. In addition, the standard 303 displayed by the stencil and the standard 311 displayed on the label mean the same content. The size 313 and the demand code 315 also mean the same contents as the size 304 and the demand code 306, respectively. As described above, the standard 303, ID 305, size 304, and demand code 306 are information managed by the thick plate manufacturer. Accordingly, the standard 311, ID 312, size 313, and demand code 315 are also information managed by the manufacturer of the thick plate. In addition, the delivery date 314 of the thick plate is also information managed by the manufacturer of the thick plate. In addition, the specification 311, ID 312, size 313, delivery date 314, and demand code 315 are information that can be identified by a person.

Note that, in addition to the above display, display by, for example, engraving may be applied to the back plate.

<Conveyance mode of heavy plate>

<<Transportation by Transport Ship>>

Fig. 4 is a diagram showing an example of how the heavy plate is loaded and unloaded with respect to the transport ship. In FIG. 4 , with respect to the transport ship 400, a case in which heavy plates are loaded and unloaded using a gantry crane 410 is illustrated. FIG. 5 is a view of a state in which the back plate 420 is suspended from the loading device 411 as viewed from the extension direction of the garter 412 (y-axis direction). The x-y-z coordinates shown in FIGS. 4 and 5 represent the relationship in the directions of FIGS. 4 and 5 .

The gantry crane 410 has a loading device (trolley or the like) 411, a garter (traversing digit) 412, and a leg structure 413. It moves along the extension installation direction (y-axis direction) of (412). In FIG. 4 , a state in which the thick plate 420 is lifted up by the loading and unloading device 411 and moved is shown. When the back plate 420 is suspended from the loading device 411 and viewed from the extension direction of the garter 412 (y-axis direction), it is as shown in FIG. 5 .

In addition, in FIG. 4 and FIG. 5, the state in which the back plate 420 of 1 sheet was suspended by the loading apparatus 411 is illustrated. However, the transport mode of the heavy plate by the loading device 411 is not limited to such a transport mode. 6A to 6E are diagrams showing first to fifth examples of a heavy plate conveyance mode. 6A to 6E, the loading device 411 is shown more simplified.

6A illustrates a state in which two thick plates are simultaneously transported by the loading device 411 in a state in which they are arranged in a horizontal plane (x-y plane) direction. 6B illustrates a state in which one thick plate is conveyed on the top side (the negative direction side of the x-axis) of the loading apparatus 411. 6C illustrates a state in which one thick plate is conveyed on the bottom side (positive direction of the x-axis) of the loading device 411. 6D illustrates a state in which one thick plate is being conveyed in the center of the loading device 411 . 6E illustrates a state in which a plurality of (two) thick plates are simultaneously conveyed by the loading device 411 in a stacked state. In this way, in this embodiment, there is a case where a plurality of thick plates are simultaneously conveyed in a stacked state.

As described above, the transport form of the heavy plate by the loading device 411 is not limited. In addition, the hoisting method of the heavy board by the loading device 411 may be a method using magnetic attraction force by a magnet, a method using a hacker, or other methods.

Since the gantry crane 410 itself can be described with a known technique, detailed description is omitted here.

The thick plate manufactured in the steelworks is transported to the quay wall and is placed in a predetermined place that can be transported by the gantry crane 410. As shown in Fig. 4, a plurality of back plates may be placed in a predetermined place in a stacked state. In some cases, the back plate is placed in a predetermined place one by one. In addition, in FIG. 4 etc., the case where the thickness (board thickness) of a plurality of thick plates is the same is illustrated for convenience of notation. However, there are cases where the thicknesses of the plurality of thick plates are different. That is, there is a case where at least one of the thickness, width, and length of the thick plate placed in a predetermined place is different.

4 illustrates a case where a plurality of thick plates are overlapped so that the plate surface of the thick plates is parallel to the surface on which the thick plates are placed. However, the method of stacking the thick plates is not limited to this method. For example, in some cases, a plurality of thick plates are overlapped so that the plate surface of the thick plates is perpendicular or at an angle close to vertical with the surface on which the thick plates are placed. This is the same also in positions other than the predetermined place|placement of a quay wall.

In this embodiment, first imaging devices 430a to 430f are attached to the garter 412 and the leg structure 413 . The number and positions of the first imaging devices 430a to 430f are set so that as many display areas of the back plate as possible are imaged, such as the display area 201 by the stencil of the back plate and the display area by the label 202 of the back plate. Also in the loading and unloading device 411, first imaging devices 430g to 430m are installed. The number and positions of the first imaging devices 430a to 430m are indicated by a stencil of the thick plate 420 during transport by the gantry crane 410 in at least one of the first imaging devices 430a to 430m. At least one of the display areas by the area 201 and the label 202 is set to be imaged as much as possible.

The first imaging devices 430a to 430m can adjust imaging conditions (imaging direction, angle of view, zoom magnification, aperture (F value), etc.). Adjustment of the imaging conditions may be performed by an operator (worker) operating the first imaging device, or may be performed automatically based on an instruction from an external device. In addition, for example, among the first imaging devices 430g to 430m attached to the garter 412, the first imaging devices 430i to 430m installed in a position close to the thick plate 420 during conveyance may be a fiber scope camera. do. If a fiberscope camera is used, for example, the display area 201 by the stencil and the display area by the label 202 of the thick plate 420 during transportation can be close-up.

In addition, the captured image captured by the first imaging device is not limited to a still image, but may be a moving image. When the captured image captured by the first imaging device is a moving image, an image extracted from the moving image captured by the first imaging device (i.e., a frame image of the moving image) is used in a tracking device 1200 described later. In this case, it is preferable to extract a still image in focus as much as possible from the moving image captured by the first imaging device. The fact that the captured image captured by the imaging device is not limited to a still image is the same for the second to seventh imaging devices described later.

The heavy plate transported from the steelworks is placed in a predetermined place of the gantry crane 410. An area including the display area 201 using a stencil of the back plate placed in a predetermined place of the gantry crane 410 is captured by the first imaging devices 430c to 430f. The operator (worker) operates an information processing terminal in the operating room of the gantry crane 410 or an information processing terminal (such as a tablet terminal) possessed by the operator (worker) of the gantry crane 410, A preview image display instruction is given to the imaging devices 430c to 430f. The first imaging devices 430c to 430f transmit the preview image to the information processing terminal based on the preview image display instruction. Then, the preview image is displayed by the information processing terminal. An operator (worker) operates the information processing terminal to give an imaging instruction to the first imaging device that is imaging a preview image in which a display area by a stencil to be captured is displayed. Upon receiving an imaging instruction, the first imaging devices 430c to 430f capture a captured image of an area including the display area 201 using a stencil. In addition, it is assumed that the preview image is a live image displayed for confirmation by an operator (worker), and is displayed based on a preview image display instruction. It is assumed that the captured image is an image captured based on an imaging instruction when a preview image is displayed. In addition, although the case where a preview image is displayed is exemplified here, it is not always necessary to display a preview image.

When a plurality of thick plates are stacked and transported at the same time, the region including the display area 201 by the stencil cannot be imaged for the underlying thick plates. In this case, instead of the captured image of the area including the display area 201 by the stencil, a captured image of the area including the display area by the label 202 is captured.

Depending on the position where the back plate is placed, the display area 201 using the stencil and the display area using the label 202 may not be included in the imaging range of the first imaging devices 430c to 430f. In this case, the operator (worker) operates the unmanned aerial vehicles (drones) 440a to 440b equipped with the first imaging device to capture the area (or label) of the desired thick plate including the display area 201 using the stencil. The captured image of the area including the display area by (202) may be captured using the first imaging device. In addition, the area including the display area 201 using the stencil (or the area including the display area using the label 202) of the desired back plate is captured by the first imaging device possessed by the operator (worker). You may take a captured image. In addition, the first imaging device possessed by the operator (worker) may be, for example, an information terminal device (tablet terminal, smart phone, etc.) equipped with the first imaging device.

In addition, the display of the preview image and the imaging of the captured image by the first imaging devices 430c to 430f described above are carried out by the first imaging devices 430a to 430b and 430g to 430w other than the first imaging devices 430c to 430f. is also executed in the same way. In addition, when a plurality of thick plates are stacked and conveyed simultaneously, capturing an image of an area including a display area by the label 202 is performed by a first imaging device other than the first imaging devices 430c to 430f. (430a to 430b, 430g to 430w) are similarly executed. In addition, an imaging method in the case where the display area 201 using the stencil and the display area using the label 202 are not included in the imaging range of the first imaging devices 430c to 430f is The same applies to the first imaging devices 430a to 430b and 430g to 430w other than 430f). Therefore, in the following, detailed descriptions of the first imaging devices 430a to 430b and 430g to 430w other than the first imaging devices 430c to 430f are omitted.

After the heavy plate is placed in a predetermined place of the gantry crane 410, it is loaded onto the transport ship 400 by the gantry crane 410. There is a case where a plurality of heavy plates are loaded onto the transport ship 400 at the same time in a stacked state. In some cases, thick plates are loaded onto the transport ship 400 one by one. In addition, at least one of the thickness, width, and length of the thick plate loaded on the transport ship 400 may be different.

A captured image of the area including the display area 201 using the thick plate and stencil being transported by the gantry crane 410 is captured by the first imaging devices 430g to 430m installed in the loading and unloading device 411 .

When the thick plate is loaded onto the transport ship 400, a captured image of the region including the display area 201 using the stencil of the thick plate is displayed by the first imaging devices 430a to 430b attached to the garter 412 or An image is captured by the first imaging device 430n installed in the transport ship 400 . Similarly to the first imaging devices 430a to 430m, the first imaging device 430n installed in the transport ship 400 also adjusts the imaging conditions (imaging direction, angle of view, zoom magnification, aperture (F value), etc.) it is possible In addition, in FIG. 4 , the number of first imaging devices installed in the transport ship 400 is set to one for convenience of notation. However, a plurality of first imaging devices are installed in the transport ship 400 so that the area where the thick plate is placed is included in the imaging range.

The transport ship 400 loaded with thick plates in the above manner is focused on a quay wall near the factory. As described above, the heavy plate is unloaded from the transport ship 400. As described above, in this embodiment, the heavy plate is unloaded using the gantry crane 410. Also for the thick plate being transported by the gantry crane 410, a captured image is captured in the same manner as in the case of loading onto the transport ship 400. In addition, when the load is unloaded by the gantry crane 410 and placed in a predetermined location of the gantry crane 410, the steel plate is transported from the steelworks and placed in a predetermined location of the gantry crane 410. An image is captured.

<<Transportation by transport vehicle>>

The heavy plate placed in the predetermined place of the gantry crane 410 is loaded onto a transport vehicle. Fig. 7 is a diagram showing an example of how a heavy plate is loaded and unloaded with respect to a transport vehicle. The heavy plate placed in the predetermined place of the gantry crane 410 is loaded onto the transport vehicle 700 using a crane not shown or the like. As shown in FIG. 7 , there is a case where a plurality of thick plates are loaded onto the transport vehicle 700 at the same time in a stacked state. However, there are cases where the thick plates are loaded onto the transport vehicle 700 one by one. In addition, at least one of the thickness, width, and length of the thick plate loaded on the transport vehicle 700 may be different. Further, as shown in FIGS. 6A to 6E , the heavy plate may be transported and loaded onto the transport vehicle 700 .

FIG. 8 is a diagram showing an example of a state in which a heavy plate is loaded onto the transport vehicle 700 . First imaging devices 430o to 430p are installed in a structure installed in a waiting position of the transport vehicle 700 . The number and position of the first imaging devices 430o to 430p are the display area 201 by the stencil and the display area by the label 202 of the thick plate loaded on the transport vehicle 700, displaying as many thick plates as possible. An area is set to be imaged. A first imaging device 430q is also installed in the transport vehicle 700 . Similarly to the first imaging devices 430a to 430m, the first imaging devices 430o to 430q can also adjust imaging conditions (imaging direction, angle of view, zoom magnification, aperture (F value), etc.).

An area including the display area 201 using the stencil of the thick plate loaded on the transport vehicle 700 is captured by the first imaging devices 430o to 430q.

<<Delivery within the factory>>

When the transport vehicle 700 loaded with the heavy plate as described above arrives at the factory, the heavy plate is unloaded from the transport vehicle 700 . The heavy plate loaded onto the transport vehicle 700 is unloaded from the transport vehicle 700 using a crane or the like not shown. As shown in FIG. 7 , there is a case where a plurality of thick plates are unloaded from the transport vehicle 700 at the same time in a stacked state. However, there are cases where the heavy plates are unloaded from the transport vehicle 700 one by one. In addition, at least one of the thickness, width, and length of the thick plate unloaded from the transport vehicle 700 may be different. Alternatively, as shown in FIGS. 6A to 6E , the heavy plate may be transported and unloaded from the transport vehicle 700 . In addition, when conveying a heavy plate from the transport ship 400 to a predetermined place of the gantry crane 410, the conveying device (gantry crane 410) used when conveying the heavy plate from the transport vehicle 700 to a place in the factory The transport equipment used (such as a crane not shown) is another transport equipment. In this way, a thick plate transported to a certain place by a certain conveying machine may be conveyed to a different place that cannot be conveyed by a conveying machine different from the certain conveying machine. This is the same also in the case of conveying a thick plate between other storage places, such as between each storage place in a factory.

9A to 9C are diagrams showing first to third examples of the appearance of thick plates placed in a storage area in a factory. First imaging devices 430r to 430w are installed in a structure installed in a storage area in a factory. As shown in FIGS. 9A to 9C , the thick plate is placed in a place in the factory in various ways. Therefore, the number and position of the first imaging devices 430r to 430w are set so that as many display areas of the back plate as possible are imaged, such as the display area 201 by the stencil and the display area by the label 202 of the back plate placed in the place. is set Similarly to the first imaging devices 430a to 430m, the first imaging devices 430r to 430w can also adjust imaging conditions (imaging direction, angle of view, zoom magnification, aperture (F value), etc.).

An area including a display area 201 using a stencil of a thick plate placed in a storage area in a factory is captured by the first imaging devices 430r to 430w.

When the back plate is placed in a place for loading, the area including the display area 201 (or the display area using the label 202) using the stencil of the back plate is the first imaging device (430r to 430w) as described above. ) is imaged by

After that, the thick plate is conveyed to the production facility. 10 is a diagram showing an example of how thick plates are processed in a production facility. In FIG. 10 , a production facility 1000 is, for example, a facility for heat treatment and a facility for gas cutting. In FIG. 10, the case where thick board 1002a, 1002b is conveyed using the conveyance roll 1001 is illustrated. In addition, conveyance of the thick plate between the place and the conveyance roll 1001 is realized by using a conveyance device or the like not shown.

First imaging devices 430x to 430y are installed in a structure installed in a production line of a factory. The number and positions of the first imaging devices 430x to 430y are set so that the display area 201 of the rear plates 1002a and 1002b is imaged by the stencil. Similarly to the first imaging devices 430a to 430m, the first imaging devices 430x to 430y can also adjust the imaging conditions (imaging direction, angle of view, zoom magnification, aperture (F value), etc.). In addition, it is assumed that thick plates 1002a and 1002b conveyed on the conveyance roll 1001 do not overlap. In addition, the conveyance route of thick board 1002a, 1002b on the conveyance roll 1001 is assumed beforehand. For this reason, in this embodiment, it is assumed that there is no case where the display area 201 by the stencil of the back plate 1002a, 1002b is not imaged by the first imaging devices 430x to 430y. However, you may image the display area 201 by the stencil of the back plate 1002a, 1002b using the unmanned aerial vehicle provided with the 1st imaging device. In addition, in FIG. 10, the broken line shows the conveyance route of thick board 1002a, 1002b on the conveyance roll 1001, and is not real.

The thick plate 1002a conveyed at the place in the factory is loaded on the conveyance roll 1001 and conveyed by the conveyance roll 1001 . Before entering the production facility 1000, the area including the display area 201 using the stencil of the thick plate 1002a during conveyance on the conveyance roll 1001 is captured by the first imaging device 430x. In addition, after processing is completed in the production facility 1000 and exited from the production facility 1000, the area including the display area 201 by the stencil of the thick plate 1002b during conveyance on the conveyance roll 1001 is , is captured by the first imaging device 430y.

After that, the thick plate is placed in a storage place as illustrated in Figs. 9A to 9C in the warehouse. As described above, the area including the display area 201 using the stencil (or display area using the label 202) of the thick plate placed in the storage area in the factory is imaged by the first imaging devices 430r to 430w. do. At the time of shipment, a thick board is moved from a warehouse to a place for shipment. An area including the display area 201 using the stencil (or display area using the label 202) of the thick plate placed in the place for shipment is captured by the first imaging devices 430r to 430w.

Even in each storage place in a factory, there are cases where a plurality of thick plates are placed in a stacked state. In some cases, thick plates are placed at each place in the factory one by one. In addition, there are cases in which at least one of the thickness, width, and length of the thick plate placed at each place in the factory is different.

Fig. 11 is a diagram showing an example of the original thick plate and the thick plate after cutting. 11 illustrates a case in which the thick plate 1100 is cut and the thick plates 1110, 1120, 1130, and 1140 are manufactured. After the thick plates 1110, 1120, 1130, and 1140 are manufactured in this way, display is performed on each of the thick plates 1110, 1120, 1130, and 1140. In FIG. 11 , display areas 1111 , 1121 , 1131 , and 1141 for the rear plates 1110 , 1120 , 1130 , and 1140 are shown. In the display areas 1111 , 1121 , 1131 , and 1141 , information including IDs of the rear plates 1110 , 1120 , 1130 , and 1140 is displayed. In the display regions 1111, 1121, 1131, and 1141, display is performed using, for example, a stencil. In addition to display with a stencil, at least one of display with a mark and display with a label may be performed. In addition, the display in the display areas 1111, 1121, 1131, and 1141 may be handwritten display.

In the display areas 1111, 1121, 1131, and 1141, the ID of the original thick plate 1100 may be displayed. In the display regions 1111, 1121, 1131, and 1141, display is performed using, for example, a stencil. However, display in the display areas 1111, 1121, 1131, and 1141 may be handwritten display. This is the same for display in the display area 201 and display using the label 202 . Below, it is assumed that the display areas 1111, 1121, 1131, and 1141 are display areas using stencils.

When the thick plates 1110, 1120, 1130, and 1140 are manufactured, for example, display areas 1111, 1121, 1131, 1141) may be captured. In addition, the first imaging device possessed by the operator (worker) may be, for example, an information terminal device (tablet terminal, smart phone, etc.) equipped with the first imaging device. In addition, an operator (worker) operates an unmanned aerial vehicle equipped with the first imaging device to obtain a captured image of a region including the display regions 1111, 1121, 1131, and 1141 using a stencil of a desired rear plate. You may image by the 1st captured image.

As described above, in this embodiment, at least when placed on the transport ship 400, when unloaded from the transport ship 400 and loaded onto the transport vehicle 700, and when the gantry crane 410 and the plant are placed An example of a case in which captured images of areas including display areas 201, 1111, 1121, 1131, and 1141 by stencils, and areas including display areas by labels 202 of each thick plate are obtained when placed in a position do.

In this embodiment, the position of the back plate is automatically tracked by the tracking device using the captured image captured as described above. In addition, when the thick plates are conveyed one by one, since the display area 201 can be imaged using the stencil, a captured image of the area including the display area using the label 202 becomes unnecessary. In the following description, when the display by stencil and the display by label 202 are collectively referred to, display by stencil (label) is used as necessary. In addition, when the display areas 201, 1111, 1121, 1131, 1141 and the display areas using the label 202 are collectively referred to as stencil display areas, they are indicated as display areas using stencils (labels) as necessary.

<Tracking device>

12 is a diagram showing an example of a functional configuration of the tracking device 1200. The hardware of the tracking device 1200 is realized by using, for example, an information processing device provided with a CPU, ROM, RAM, HDD, and various interfaces, or dedicated hardware. 13 is a diagram showing an example of the management table 1300 managed by the tracking device 1200.

As shown in FIG. 12 , in the present embodiment, the tracking device 1200 includes a first image acquisition unit 1201, a recognition unit 1202, an acquisition unit 1203, a matching unit 1204, It has a specification unit 1205, a position derivation unit 1206, a storage unit 1207, an output unit 1208, and a second image acquisition unit 1209.

The first image acquiring unit 1201 acquires a first captured image of an area including a display area by a stencil (label). The recognition unit 1202 recognizes, from the first captured image acquired by the first image acquisition unit 1201, components, positions, and the like of display items in display using a stencil (label). Acquisition unit 1203 obtains information stored in the column of ID of the column of actual product display in management table 1300 described later. The matching unit 1204 checks the information recognized by the recognizing unit 1202 and the information acquired by the obtaining unit 1203. The specifying unit 1205 specifies one thick plate from among the thick plates managed in the management table 1300 based on the result of collation in the verifying unit 1204 . In this embodiment, the identification information acquisition unit 1210 that acquires the identification information of the thick plate has a recognition unit 1202, an acquisition unit 1203, a verification unit 1204, and a specification unit 1205. The position derivation unit 1206 derives the x-y-z coordinates (3-dimensional coordinates) of the display area by the stencil (label) as the 3-dimensional position of the back plate specified in the specification unit 1205. The storage unit 1207 stores information indicating the three-dimensional position of the back plate in the management table 1300. The output unit 1208 outputs information indicating the three-dimensional position of the back plate. The second image acquisition unit 1209 acquires, in the position derivation unit 1206, a second captured image for specifying a position in the height direction (z-axis direction) of the display area by a stencil (label). A detailed example of the tracking device 1200 will be described later with reference to a flowchart in FIG. 17 .

In Fig. 13, information used by the tracking device 1200 to track the back plate is stored in the management table 1300. The information stored in the management table 1300 includes information managed by the thick plate manufacturer. However, part of the information stored in the management table 1300 may be information that can be utilized also in distribution or the like. Here, the information managed by the heavy plate manufacturer is information set or acquired by the heavy plate manufacturer. The information managed by the heavy plate manufacturer includes information that allows an operator who is an employee of the heavy plate manufacturer to understand the meaning of the information. More specifically, the information managed by the heavy plate manufacturer is information created by, for example, the heavy plate manufacturer and stored in the management table 1300 . In this embodiment, the information stored in the management table 1300 is an example of tracking data, and by storing each information in the management table 1300, an example of the data structure of the tracking data is realized.

In Fig. 13, information in the same column becomes identification information for the same thick plate (a column under the same No. becomes a column in which identification information for the same thick plate is stored, respectively).

In Fig. 13, information related to each display item displayed by a stencil (label) is stored in the field of actual item display. Information other than information about each display item of display by a stencil (label) is stored in the column of the non-actual item display. In the present embodiment, the information stored in the column of the actual item display is an example of display information included in the tracking data, and the information stored in the non-display column is an example of non-display information included in the tracking data.

An example of information stored in the column of the actual product display of the management table 1300 will be described.

In the column of the actual product display, information related to the stencil is stored in the column of the stencil.

In the ID column of the stencil column, information indicating the contents and positions of the components of ID305 included in the display by the stencil is stored. In addition, a constituent element refers to individual information constituting a display item, such as letters, numbers, symbols, and marks.

In the column of the name of the customer in the column of the stencil, information indicating the contents and positions of the constituent elements of the name of the customer 302 included in the display by the stencil is stored.

In the size column of the stencil column, information indicating the contents and positions of components of size 304 included in the display by the stencil is stored.

In the specification column of the stencil column, information indicating the contents and positions of components of the standard 303 included in the display by the stencil is stored.

In the demand code column of the stencil column, information indicating the contents and positions of components of the demand code 306 included in the display by the stencil is stored.

In the order number column of the stencil column, information indicating the contents and positions of components of the order number 307 included in the display by the stencil is stored.

In addition to the display items in the above stencil display, the contents of display items that may be displayed as display items in the stencil display are also stored in the management table 1300 . "..." under the additional information column of the stencil column in Fig. 13 indicates this. Also, the display items themselves may be added and/or deleted.

In the field of display of the actual item, information related to the label is stored in the column of label.

In the ID column of the label column, information indicating the contents and positions of components of the ID 312 included in the label display is stored.

In the size column of the label column, information indicating the contents and positions of components of size 313 included in the label display is stored.

In the specification column of the label column, information indicating the contents and positions of components of the standard 311 included in label display is stored.

In the delivery date column of the label column, information indicating the contents and positions of components of the delivery date 314 included in the display by the label is stored.

In the demand code field of the label column, information indicating the contents and positions of components of the demand code 315 included in the display by the label is stored.

In addition to the display items in the above label display, the contents of display items that may be displayed as display items in the label display are also stored in the management table 1300 . "..." under the column of the demand code in the column of the label in Fig. 13 indicates this. Also, the display items themselves may be added and/or deleted.

Next, an example of information stored in the column of non-actuated item display of the management table 1300 will be described.

In the column of position in the article, information indicating the position of the display area when a predetermined position on the back plate is used as the origin (reference) is stored.

In the column of the stencil of the column of position in the article, information indicating the position of the display area by the stencil when a predetermined position in the back plate is used as the origin (standard) is stored.

In the column of label in the column of position in article, information indicating the position of the display area by the label when a predetermined position in the back plate is used as the origin (standard) is stored.

In the article position column, information indicating the three-dimensional position of the back plate is stored. The information stored in the article position column is information indicating the three-dimensional position of the back plate specified by the tracking device 1200 . Therefore, in the 3D position of the same thick plate specified by the tracking device 1200, not only the x and y coordinates but also the z coordinate may change as the tracking progresses (movement of the thick plate). Therefore, the three-dimensional position of the information display position of the same back plate specified by the tracking device 1200 is not only the x coordinate and the y coordinate (position in the horizontal plane direction) but also the z coordinate along with the progress of tracking (movement of the back plate). (position in the height direction) may also change.

In the column of the registration time, information indicating the date and time (timing determined from the date and time) when the rear plate is placed at the current position is stored.

In the parent ID column, information indicating the ID of the original back plate is stored. There are cases in which thick plates are cut in factories. The original thick plate refers to the thick plate before cutting. Therefore, when information indicating the ID of the original back plate is stored in the parent ID column, the information stored in the column of the same column as the parent ID column becomes information on the back plate after cutting. When the thick plates 1110, 1120, 1130, and 1140 are manufactured as shown in FIG. 11, in the management table 1300, in the column for displaying the actual product for the thick plates 1110, 1120, 1130, and 1140, Information indicating the contents and positions of constituent elements of the display items displayed in the display areas 1111, 1121, 1131, and 1141 is stored. Further, in the parent ID column for the thick plates 1110, 1120, 1130, and 1140, information indicating the ID of the thick plate 1100 is stored. Thus, in this embodiment, about the information of the back plate after cutting, the case where it manages as a new column No in the management table 1300 shown in FIG. 13 is illustrated. In the management table 1300, information about the thick plate after cutting is stored in the fields of display on actual item and display of non-existent item.

Here, for example, there is a case in which a plurality of other thick plates are manufactured by cutting the thick plate 1110 . In this case, information indicating the ID of the thick plate 1110 is stored in the parent ID column for the plurality of other thick plates. In the parent ID column for the back plate 1110, information indicating the ID of the back plate 1100 is stored. In this way, by tracing the information stored in the column of parent ID, it is possible to specify the change of each back plate. In this way, for a certain thick plate, the sequence of thick plates obtained by tracing the information stored in the parent ID field is referred to as the out-of-plate series of the thick plate. For example, the out-of-plate series of the thick plate 1110 becomes the thick plate 1100, 1110. In addition, in the column of parent ID of each back plate, not only information indicating the ID of the next back plate but also information indicating the ID of all back plates belonging to that section series is stored so that, for example, the out-of-school series can be known do. In this case, any ID defined by the manufacturer is used.

In the column of the carrying tool ID, the ID of the heavy plate carrying tool is stored. In this embodiment, the case where the transport tool is the transport ship 400 and the transport vehicle 700 is exemplified.

When the thick plate is loaded on the transport ship 400, the tracking device 1200 may store information indicating the ID of the transport ship 400 as follows, for example.

First, the tracking device 1200 acquires the ID of the thick plate and the ID of the carrier 400 (so-called ship number). Then, the tracking device 1200 specifies, from the ID column of the management table 1300, a column in which information indicating the same ID as the ID of the acquired back plate is stored. Then, the tracking device 1200 stores information indicating the ID of the transport ship 400 acquired in the column of the carrying tool ID of the specific column.

For example, the display area 201 using a stencil or the display area using a label 202 and information indicating the ID of the transport ship 400 are displayed by the first imaging devices 430a, 430b, and 430n. A captured image including an area may be captured. In this case, the tracking device 1200 may recognize and acquire the ID of the thick plate and the ID of the transport ship 400 from the captured image.

In addition, an image captured by the first imaging devices 430a, 430b, and 430n including the display area 201 using the stencil or the display area using the label 202 and information indicating the ID of the transport ship 400 are provided. A captured image including the displayed area may be captured. For example, after the captured image including the display area 201 by the stencil or the display area by the label 202, the captured image including the area where the information indicating the ID of the transportation ship 400 is displayed is It may be taken as being imaged. In this case, the tracking device 1200 displays information indicating the ID recognized from the captured image including the display area 201 by the stencil or the display area by the label 202 and the ID of the transport ship 400 IDs recognized from the captured image including the area being formed may be acquired as the ID of the thick plate and the ID of the transport ship 400.

Further, for example, the thick plate ID and the ID of the transport ship 400 may be input to an information processing terminal (tablet terminal or the like) possessed by an operator (worker) of the transport ship 400 . In this case, the ID of the thick plate and the ID of the transport ship 400 are transmitted from the information processing terminal to the tracking device 1200, so that the tracking device 1200 transmits the ID of the heavy plate and the ID of the transport ship 400. may acquire

In addition, when the heavy plate is loaded onto the transport vehicle 700, the tracking device 1200 may store information indicating the ID of the transport vehicle 700 as follows, for example.

First, the tracking device 1200 acquires the ID of the heavy plate and the ID of the transportation vehicle 700 (for example, license plate information). Then, the tracking device 1200 specifies, from the ID column of the management table 1300, a column in which information indicating the same ID as the ID of the acquired back plate is stored. Then, the tracking device 1200 stores the information indicating the acquired ID of the transport vehicle 700 in the column of the transportation tool ID of the specific row.

For example, the display area 201 using a stencil or the display area using a label 202 by the first imaging devices 430o to 430q, and an area where information indicating the ID of the transportation vehicle 700 is displayed You may take a captured image including . In this case, the tracking device 1200 may recognize and acquire the ID of the rear plate and the ID of the transportation vehicle 700 from the captured image.

In addition, by the first imaging devices 430o to 430q, a captured image including the display area 201 using a stencil or the display area using a label 202 and information indicating the ID of the transportation vehicle 700 are displayed. You may take a captured image including the area in which it has been. For example, after the captured image including the display area 201 by the stencil or the display area by the label 202, the captured image includes the area where the information indicating the ID of the transportation vehicle 700 is displayed. This may be imaged. In this case, the tracking device 1200 provides information indicating the ID recognized from the captured image including the display area 201 by the stencil or the display area by the label 202 and the ID of the transportation vehicle 700 The ID recognized from the captured image including the displayed area may be acquired as the ID of the rear plate and the ID of the transport vehicle 700 .

Further, for example, the ID of the back plate and the ID of the transport vehicle 700 may be input to an information processing terminal (such as a tablet terminal) possessed by an operator (worker) of the transport vehicle 700 . In this case, by transmitting the ID of the thick plate and the ID of the transport vehicle 700 from the information processing terminal to the tracking device 1200, the tracking device 1200 transmits the ID of the thick plate and the ID of the transport vehicle 700. You may acquire an ID.

The data structure of the management table 1300 is a data structure in which information stored in each column of the management table 1300 (for example, product position and registration time) is updated whenever a back plate is placed in a new position. may be In addition, the data structure of the management table 1300 may be a data structure to which information is added. In the case of the latter, it has the advantage of being able to track the history of tracking.

Next, an example of the content stored in the item display column of the management table 1300 and the content stored in the item position column in the non-item display column will be explained in more detail.

Fig. 14 is a diagram showing an example of a position of a stencil in an article and a position of components of each display item in a display using a stencil. In FIG. 14, a case where display by a stencil is made in the display position shown in FIG. 2 and the content shown in FIG. 3A is exemplified.

In Fig. 14, among the eight vertices of the thick plate of the rectangular parallelepiped, the vertex 203 shown in Fig. 2 is the origin, and the direction from the vertex 203 along the width direction of the thick plate is the positive direction of the y-axis and the longitudinal direction of the thick plate. In the x-y coordinates where the direction along is the positive direction of the x-axis, the position of the stencil in the article and the position of the component of each display item by the stencil are shown.

In the present embodiment, four vertices of a circumscribed rectangle determined by the positions corresponding to the maximum value of the x-axis, the minimum value of the x-axis, the maximum value of the y-axis, and the minimum value of the y-axis among the positions of the components of each display item in the display using the stencil Among them, the x-y coordinates of the two vertices on the diagonal are taken as the positions of the stencil in the article. In Fig. 14, the x-y coordinates (Ls10, Ws10) and (Ls11, Ws11) of the two vertices 1401a and 1401b of the circumscribed rectangle represent positions of the stencil in the article. Note that, in order to simplify the explanation here, the circumscribed rectangle is assumed to be a rectangle parallel to the coordinate axes (x-axis and y-axis in the example shown in Fig. 14) expressing the position in the article.

Further, the x-y coordinates of the positions of the centers of gravity (or centroids) of the constituent elements (letters, numbers, symbols, and marks) of each display item in the display using the stencil are taken as the positions of the constituent elements. In FIG. 14, the y-axis coordinates of 5, 6, 3, 4, 9, 7, -, 0, 1, which are components of the ID 305 included in the display by the stencil, are Ws1, Ws2, Ws3, Ws4, Ws5, Ws6, Ws7, Ws8, and Ws9 indicate that all of the x-axis coordinates are Ls1.

As described above, the position of each component of each display item of the display by the stencil is displayed by x-y coordinates.

Fig. 15 is a diagram showing an example of the position of a label in an article and the position of components of each display item in a label display. In FIG. 15, a case where display by a label is performed in the display position shown in FIG. 2 and the display content shown in FIG. 3B is exemplified.

In Fig. 15, among the eight vertices of the thick plate of the rectangular parallelepiped, the vertex 204 shown in Fig. 2 is the origin, and the direction from the vertex 204 along the width direction of the thick plate is the positive direction of the y-axis, the plate thickness of the thick plate In the y-z coordinate where the direction along the direction is the positive direction of the z-axis, the position of the label in the article and the position of the component of each display item by the label are shown.

In this embodiment, it is assumed that the label has a rectangular shape. Among the four vertices of the label, the y-z coordinates of two vertices on the diagonal are taken as the position of the label in the article. Further, the y-z coordinate of the position of the center of gravity (or centroid) of the component of each display item in the label display is taken as the position of the component. In Fig. 15, the y-z coordinates of the two vertexes 1501a and 1501b of the label represent positions of the label in the article.

As described above, in the management table 1300 shown in FIG. 13, in the ID column of the stencil column of the actual product display column, the contents and positions of the components of the ID 305 included in the display by the stencil Information representing is stored. Therefore, as shown in FIG. 16A, in the example shown in FIG. 14, in the ID column of the stencil column of the actual product display column, as the content of the component of the ID 305 included in the display by the stencil, "5", "6", "3", "4", "9", "7", "-", "0", "1" are stored. Further, in the ID field of the stencil column of the actual product display column, (Ls1, Ws1), (Ls2, Ws1), (Ls3, Ws1), (Ls4, Ws1), (Ls5, Ws1), (Ls6, Ws1), (Ls7, Ws1), (Ls8, Ws1), and (Ls9, Ws1) are memorized.

Information indicating the contents and positions of components of the display item is also stored in the column of other display items in the column of the actual product display, similarly to the column of ID in the column of the stencil. In addition, the information indicating the position of the component of each display item in the column of the label is y-z coordinates, not x-y coordinates.

Further, as shown in FIG. 16B, in the example shown in FIG. 14, in the stencil column of the column of the position in the article of the column of the non-actual item display, as information indicating the position of the stencil in the article, (Ls10, Ws10) , (Ls11, Ws11) are memorized.

Information indicating the position of the label in the article is also stored in the column of the label of the column of the position in the article, similarly to the column of the stencil of the column of the position in the article. In addition, as described above, the information representing the position of the label in the article becomes y-z coordinates, not x-y coordinates.

When creating the management table 1300, for example, an operator may operate the user interface of the tracking device 1200 and input information to be stored in each column of the management table 1300 into the tracking device 1200. . In addition, the tracking device 1200 may receive information to be stored in each column of the management table 1300 from an external device such as a computer that manages factory operations.

The information stored in the item display column of the management table 1300, and the information stored in the product position in the item display column and the parent ID column in the non-item display column may be determined after the display for the rear plate is performed. In some cases, it is decided before the indication for is done. Therefore, when the information to be stored in the corresponding column is determined before display of the back plate is performed, the tracking device 1200 acquires the information to be stored in the corresponding column as described above, for example, and manages It is stored in each column of the table 1300.

On the other hand, there are cases in which the information stored in the item position and registration time column of the non-actual item display column cannot be determined before the display on the back plate is performed. Therefore, the tracking device 1200 acquires, for example, the information to be stored in the column at the timing at which the information to be stored in the column is obtained, and stores the information in each column of the management table 1300. As described above, the date and time when the information stored in the item location column is changed is stored in the registration time field. In this case, the date and time when the rear plate is placed at the current position is registered in the field of the registration time.

In addition, for the item in the column of non-actual item display, information is stored in the column of the relevant item only when the information of the item is acquired, and when the information of the item is not acquired, the column of the relevant item information is not remembered. In addition, when the information stored in the article position column is not updated even after a predetermined time elapses from the date and time stored in the registration time column, the tracking device 1200 indicates that the identification (tracking) of the position of the back plate has been cut off. It is preferable to make a judgment and reset the information stored in the fields of the item location and registration time. Also, in the same position, the later plate with the earlier date and time stored in the registration time field does not move before the later plate with the later date and time stored in the registration time field (that is, when there is an overlap of boards, There is no case where the underlying plate suddenly disappears). For this reason, when the tracking device 1200 resets the information stored in the fields of the item position and registration time, it is preferable to do the following. That is, the tracking device 1200 is a back plate in which a date and time later than the date and time recorded in the field of registration time to be reset is stored in the field of registration time, and the same as the three-dimensional position stored in the field of item position to be reset. When there is a back plate in a three-dimensional position, it is preferable to issue a sign to confirm with an alarm or the like.

Next, an example of the function of the tracking device 1200 will be described. Fig. 17 is a flowchart for explaining an example of the processing of the tracking device 1200 when tracking a thick plate. An example of the function of the tracking device 1200 will be described with reference to FIG. 17 .

In step S1701 of FIG. 17 , the first image acquisition unit 1201 acquires a first captured image. The first captured image is any one of the first imaging devices 430a to 430y, the first imaging devices provided to the unmanned aerial vehicles 440a to 440c, and the first imaging devices owned by operators (workers). 1 It is a captured image captured by the imaging device. The first captured image may be directly transmitted from the first imaging device to the tracking device 1200, or may be transmitted from the first imaging device to the tracking device 1200 via an information terminal device. The processing according to the flowchart of FIG. 17 is carried out using any one of the first imaging devices 430a to 430y, the first imaging devices provided to the unmanned aerial vehicles 440a to 440c, and the first imaging devices possessed by operators (workers). It is executed whenever a captured image captured by one imaging device is received by the tracking device 1200 . In this embodiment, it is assumed that the ID of the first imaging device that captured the first captured image and the imaging conditions when the first captured image was captured are transmitted together with the first captured image.

The first image acquisition unit 1201 acquires the ID of the first imaging device that captured the first captured image and imaging conditions when the first captured image was captured, together with the first captured image. The imaging conditions include information required to specify the position of a display area using a stencil or a display area using a label, as will be described later, among information set in the imaging device at the time of imaging.

In addition, if the position of the first imaging device at the time the first captured image is captured is not fixed, positional information of the first imaging device at the time the first captured image is captured is transmitted together with the first captured image. . For example, when the unmanned aerial vehicles 440a to 440c can position the three-dimensional coordinates (x-y-z coordinates) of the unmanned aerial vehicle using a Global Positioning System (GPS), the first image pickup provided by the unmanned aerial vehicle As the location information of the device, the three-dimensional coordinates may be transmitted. The first imaging device 430n installed in the transport ship 400, the first imaging devices 430g to 430m installed in the loading and unloading device 411, and the first imaging device 430q installed in the transport vehicle 700 ), the location information of the first imaging device may be transmitted together with the first captured image in the same manner as for the unmanned aerial vehicles 440a to 440c. This is the same also for the first imaging device possessed by the operator (worker).

In addition, when the first imaging device cannot image the region to be included in the first captured image, the imaging conditions of the first imaging device (imaging direction, angle of view, zoom magnification, and aperture (F value), etc.) are adjusted.

Next, in step S1702, the recognizing unit 1202 identifies components (letters, numbers, symbols, and marks) of display items in display using a stencil (label) from the first captured image obtained in step S1701. Recognize. Since the process of recognizing the contents of the constituent elements of the display item can be realized by, for example, a known technique using OCR (Optical Character Recognition), a detailed description thereof is omitted here. Further, the processing in step S1702 may be realized using a technique using AI (Artificial Intelligence) such as NN (Neural Network) or a technique combining AI and OCR (so-called AI OCR).

In addition, the recognition unit 1202 specifies which of the components of the display item is display using a stencil or display using a label. For example, the recognizing unit 1202 performs edge detection processing based on the first captured image acquired in step S1701, detects the edge of the back plate, and specifies the region of the back plate based on the detection result. Then, the recognition unit 1202 determines which of the plate surface portion of the thick plate and the side (plate thickness) portion of the thick plate is the specified region. This determination is realized based on the aspect ratio of a specific region, for example. As a result of this determination, when the specified area is the plate surface portion of the back plate, the recognition unit 1202 recognizes that the constituent elements of the display item are displayed using a stencil. On the other hand, when the specific area is the side surface (plate thickness) portion of the thick plate, the recognition unit 1202 recognizes that the component of the display item is a label display.

Further, the recognizing unit 1202 performs edge detection processing based on the first captured image, and when the area of the rectangular label 202 is detected, the constituent elements of the display item are those of the display by the label. If not, it may be recognized that the constituent elements of the display item are those of display using a stencil.

In this embodiment, in order to simplify the description, it is assumed that the first captured image of the uppermost rear plate includes a display area 201 using a stencil instead of a display area using a label 202. .

Also, the recognition unit 1202 derives the position of the display item. An example of a method in which the first recognizer 1202 derives the position of a display item will be described. First, the recognizing unit 1202 performs edge detection processing based on the first captured image acquired in step S1701 to detect the edge of the thick plate. Then, the recognition unit 1202 detects the origin of the position in the article in the first captured image (predetermined position in the back plate). The recognizing unit 1202 determines, for example, in the captured image, based on a positional relationship preset as a positional relationship between a display area by a stencil (label) and the origin of a position in an article among eight vertices of a back plate of a rectangular parallelepiped. A vertex serving as the origin of the position in the article (predetermined position in the back plate) is derived.

As the positional relationship between the display area by the stencil (label) and the origin of the position within the article, for example, the following information may be set in advance in the tracking device 1200. That is, when the component of the display item of the display by the stencil (label) is viewed so that the top of the component is located on the upper side, on the same side as the display by the stencil (label), by the stencil (label) Information indicating that the vertex located on the lower right side of the display is the origin of the position within the article may be set in advance in the tracking device 1200 . The recognizing unit 1202 derives, as positions of constituent elements of a display item in display using a stencil (label), a position based on the origin of a position in an article in a captured image. When the process of step S1702 ends, the process of step S1703 is executed.

In step S1703, the acquisition unit 1203 obtains the information stored in the ID column of the actual item display column in the management table 1300 stored in the storage unit 1207.

Next, in step S1704, the verification unit 1204 checks the information recognized in step S1702 and the information acquired in step S1703. If the information recognized in step S1702 is recognized as information for display by stencil, the matching unit 1204 stores it in the ID column of the stencil column of the actual product display column among the information acquired in step S1703. The obtained information is collated with the information recognized in step S1702. On the other hand, when it is recognized that the information recognized in step S1702 is label information, the verification unit 1204, among the information acquired in step S1703, identifies the ID column of the label column of the actual item display column. The information stored in and the information recognized in step S1702 are collated.

Next, in step S1705, the specifying unit 1205 specifies one thick plate from among the thick plates managed in the management table 1300 based on the collation result in step S1703. Specifically, the specifying unit 1205 specifies, from among the information obtained in step S1703, information that matches the information recognized in step S1702.

Here, matching refers to the fact that there is a part in which the information recognized in step S1702 completely matches the components stored in the stencil column of the actual product display column or the ID column of the label column, in content and arrangement order. . In the examples shown in FIGS. 3A and 3B , the contents of the components of the IDs 305 and 312 displayed by stencils or labels are 5, 6, 3, 4, 9, 7, -, 0, 1, The order of arrangement is 5, 6, 3, 4, 9, 7, -, 0, 1 from the left. Therefore, the specification unit 1205 converts the information in which 5, 6, 3, 4, 9, 7, -, 0, 1 are arranged in this order from the left to the IDs managed in the management table 1300. specified from the column. In the information recognized in step S1702, not only when some of these components are missing, but also in the case where components other than these components are arranged, the information recognized in step S1702 is It is assumed that the content and arrangement order do not completely coincide with the components stored in the ID column of the stencil column or the label column.

Next, in step S1706, the position deriving unit 1206 determines whether or not the position of the first imaging device that captured the first captured image acquired in step S1701 is known, based on the information acquired in step S1701. . In this embodiment, the first imaging devices 430a to 430f installed on the garter 412 and the leg structure 413, and the first imaging devices installed on the structure installed in the standby position of the transportation vehicle 700 (430o to 430p), first imaging devices (430r to 430w) installed in a structure installed in a place in a factory, and first imaging devices (430x to 430y) installed in a structure installed in a production line in a factory The three-dimensional position of is a known one. For each of these first imaging devices, the ID of the first imaging device and the location information (three-dimensional coordinates (x-y-z coordinates)) of the first imaging device are correlated with each other and registered in advance in the tracking device 1200 do it as The position derivation unit 1206, when the positional information (three-dimensional coordinates) associated with the same ID as the ID of the first imaging device acquired in step S1701 is registered in the tracking device 1200 in advance, the position information acquired in step S1701 It is determined that the position of the first imaging device that captured the first captured image is known, and if not, it is determined that the position of the first imaging device that captured the first captured image obtained in step S1701 is not known.

As a result of the determination in step S1706, when the position of the first imaging device that captured the first captured image obtained in step S1701 is not known, the processing in step S1707 is executed. In step S1707, the position derivation unit 1206 determines whether or not positional information of the first imaging device that captured the first captured image was acquired in step S1701. As a result of this determination, if the positional information of the first imaging device that captured the first captured image is not acquired in step S1701, the process proceeds to step S1712 described later.

When the position of the first imaging device that captured the first captured image obtained in step S1701 is known as a result of the determination in step S1706, and as a result of the determination in step S1707, positional information of the first imaging device that captured the first captured image is acquired in step S1701, the process of step S1708 is executed. In step S1708, the position derivation unit 1206 determines whether or not the position in the height direction (z-axis direction) of the display area by the stencil (label) can be specified. A specific example of a case where the position in the height direction (z-axis direction) of the display area by the stencil (label) can be specified will be described later. As a result of the determination in step S1708, if the position in the height direction (z-axis direction) of the display area by the stencil (label) cannot be specified, the process proceeds to step S1712 described later.

On the other hand, as a result of the determination in step S1708, when the position in the height direction (z-axis direction) of the display area by the stencil (label) can be specified, the processing in step S1709 is executed. In step S1709, the position derivation unit 1206 derives the three-dimensional coordinates (x-y-z coordinates) of the display area by the stencil (label) as the three-dimensional position of the back plate specified in step S1705.

Next, in step S1710, the storage unit 1207 converts the information stored in the article position column of the non-actuated item display column of the management table 1300 as the information on the thick plate specified in step S1705 into step S1709. It is updated with information indicating the 3D position of the derived heavy plate. In this way, in this embodiment, the case where the information stored in the item position column of the non-actual item display column of the management table 1300 is the information display position is exemplified. The information stored in the item position column of the non-actual item display column of the management table 1300 is the x-y-z coordinates of the display area by the stencil (label). Therefore, the information display position is a three-dimensional position. As described above, in the information display position of the same thick plate, not only the x coordinate and the y coordinate but also the z coordinate (position in the height direction) changes with the movement of the position of the thick plate.

Next, in step S1711, the output unit 1208 outputs information indicating the three-dimensional position of the thick plate specified in step S1705. For example, the output unit 1208 displays the information indicating the three-dimensional position of the rear plate specified in step S1705 on the computer monitor. Then, the processing according to the flowchart in Fig. 17 ends.

Here, an example of a method for deriving the three-dimensional position of the display area using the stencil (label) in step S1709 will be described taking the display area 201 using the stencil shown in FIG. 2 as an example. In the example shown in FIG. 2 , the display area 201 using the stencil is an area near the center in the width direction (y-axis direction) of the thick plate among the plate surface regions of the thick plate, and one end side (x on the negative side of the axis). In this embodiment, the three-dimensional position of the display area 201 by the stencil is an example of the information display position.

18 is a diagram for explaining an example of a method of deriving the three-dimensional position of the display area 201 using a stencil.

In FIG. 18 , point P represents the three-dimensional position of the first imaging device (xyz coordinates of point P are (x _p , y _p , z _p )). The three-dimensional position of the first imaging device is, for example, the center position of the optical system (lens) of the first imaging device. In the example shown in Fig. 18, the z-axis is assumed to be a direction perpendicular to the surface of the thick plate (the surface on which the display area 201 by the stencil is formed). Note that the y-axis is assumed to be a direction parallel to the direction in which components of the same display item are arranged in the display area 201 using the stencil. Note that the x-axis is assumed to be a direction parallel to the direction in which other display items are arranged in the display area 201 using the stencil. Also, the x-axis, y-axis, and z-axis are assumed to be perpendicular to each other. In addition, here, for simplicity of explanation, it is assumed that the width direction of the thick plate is parallel to the y-axis, and the longitudinal direction of the thick plate is parallel to the x-axis.

The point S represents the position of a vertex located on the negative x-axis side and also located on the positive y-axis side among the four vertices of the display area 201 by the stencil (the xyz coordinates of the point S are (x _s , y _s , z _s ). Point O is the foot of the perpendicular to point P (with respect to the xy plane) (the intersection of the perpendicular to point P with the xy plane). In the example shown in Fig. 18, it is assumed that the point O is the origin. Point Q is the intersection of a straight line passing through point O and parallel to the x-axis, and a straight line passing through point S and parallel to the y-axis.

If ∠OPQ is β(rad) and the length of line segment OP (distance from point O to point P) is H, ∠OQP(=η) is π/2-β(rad), so line segment OQ The length (distance from point O to point Q) L of is expressed by the following formula (1).

L=Hsinβ/sin{(π/2-β)} ... (1)

In addition, when ∠OPS is α (rad), the length l of the line segment OS (the length from point O to point S) is expressed by the following equation (2). Then, when ∠SPQ is δ (rad), the length of the line segment SQ (distance from point S to point Q) w _y is expressed by the following equation (3) by equation (2).

l = H/tanα ... (2)

w _y =lsinδ=Hsinδ/tanα ... (3)

Therefore, the xyz coordinate S (x _s , y _s , z _s ) of the point S is expressed by the following formula (4).

S(x _s , y _s , z _s )=(L, w _y , 0) ... (4)

In this embodiment, a case where the position derivation unit 1206 derives the xyz coordinates (x _s , y _s , z _s ) of the point S as the position of the display area 201 by the stencil is exemplified. When the length H of line segment OP, ∠OPQ(=β), ∠OPS(=α), and ∠SPQ(=δ) are determined, the xyz coordinates (x _s , y _s , z _s ) of point S are determined.

The length H of the line segment OP is a value obtained by subtracting the z coordinate z _s of the display area 201 using the stencil from the z coordinate z _p of the point P. Therefore, if the z-coordinate of the display area 201 by the stencil (position in the height direction (z-axis direction) of the display area 201 by the stencil) is specified, the length H of the line segment OP can be derived. 18 illustrates the case where the z-coordinate z _s of the display area 201 using the stencil is 0 (zero), the length H of the line segment OP is equal to the z-coordinate z _p of the point P.

The position of the point S in the first captured image is derived as follows, for example. The position derivation unit 1206 derives a circumscribed rectangle determined by the positions corresponding to the maximum value of the x-axis, the minimum value of the x-axis, the maximum value of the y-axis, and the minimum value of the y-axis among the components of the display item using the stencil recognized in step S1702. . Then, the position derivation unit 1206 determines, for example, the lower left side (negative direction side of the x-axis and The position of the vertex located on the positive side of the y-axis) is derived as the position of the point S in the first captured image. In addition, in the case where the label is displayed in the first captured image, the position deriving unit 1206 views the top of the component of the display item by the stencil as being located on the upper side among the vertices of the rectangular label. , for example, a vertex located on the lower left side is derived as the position of the point S in the first captured image.

The position of the point Q in the first captured image is a position on a straight line passing through the position of point S in the first captured image and parallel to the y-axis, and the horizontal angle of view (angle of view in the y-axis direction) at the time of imaging is position corresponding to the center. In addition, you may perform identification of the point Q as follows, for example. That is, a laser beam is irradiated from point P with a laser pointer or the like, and a straight line passing through the position of point S and parallel to the y-axis is displayed on the surface of the back plate on which the display area 201 by the stencil is formed. . In addition, a laser beam is irradiated from point P with a laser pointer or the like, and a straight line passing through point O and parallel to the x-axis is displayed on the surface of the thick plate on which the display area 201 by the stencil is formed. In this case, the intersection point of the two straight lines displayed by the laser beam becomes the point Q. Therefore, the position derivation part 1206 may specify the point Q from the 1st captured image imaged in the state where the intersection of the two straight lines displayed by the laser beam is displayed. In this way, when point P, point O, and point Q are on the x-z plane (on the plane of y = 0), the surface on which the display area 201 is formed by the stencil among the plate surfaces of the back plate is relative to the x-y plane Even if it is inclined, the position of point Q can be specified.

∠OPS(=α), ∠OPQ(=β), and ∠SPQ=∠SOQ(=δ) are, for example, the positions of points S and Q in the first captured image acquired in step S1701, and the second 1 It is determined based on the position of the center of the optical system (lens) when the captured image is captured (the position of the center of the optical system used to determine the focal length). The direction in which the line segment PS extends is specified by a straight line connecting the point S in the first captured image and the center position of the optical system (lens) when the first captured image is captured. The direction in which the line segment PQ extends is specified by a straight line connecting the point Q in the first captured image and the center position of the optical system (lens) when the first captured image is captured. ∠OPS(=α), ∠OPQ(=β), and ∠SPQ=∠SOQ(=δ) are determined based on the direction in which line segment PS extends, the direction in which line segment PQ extends, and the z-axis direction.

Methods for deriving ∠OPS(=α), ∠OPQ(=β) and ∠SPQ=∠SOQ(=δ) are not limited to these methods. For example, the position derivation unit 1206 may derive ∠OPS(=α), ∠OPQ(=β), and ∠SPQ=∠SOQ(=δ) as follows. First, the position deriving unit 1206 determines the lengths of the line segment PS and the point PQ based on the measured values of the range sensor (eg, a time-of-flight (TOF) type range sensor) provided in the first imaging device. derive Then, the position deriving unit 1206 assumes that the plane on which the display area 201 by the stencil is formed is an x-y plane and the line segment QS is a straight line extending in the y-axis direction, and determines the positions of the points S and Q. be specific Then, the position derivation unit 1206 derives ∠OPS(=α), ∠OPQ(=β), and ∠SPQ=∠SOQ(=δ) based on the positions of the points P, S, and Q. .

Further, for example, the position deriving unit 1206 determines the angle of the laser beam emitted from the first imaging device toward the point S, the angle of the laser beam emitted from the first imaging device toward the point Q, and the first imaging device. ∠OPS(=α), ∠OPQ(=β), and ∠SPQ=∠SOQ(=δ) may be derived based on the scan angle when the laser beam is scanned from point S to point Q.

Further, the position deriving unit 1206 determines the first captured image captured in a state in which the grid-shaped laser beam is irradiated to the area including the point O and the surface of the thick plate on which the display area 201 using the stencil is formed. ∠OPS(=α), ∠OPQ(=β), and ∠SPQ=∠SOQ(=δ) may be derived by counting the number of rectangles arranged in a lattice form. In this case, the length in the x-axis direction of one rectangle is, for example, the length in the x-axis direction that changes when ∠OPS (=α) and ∠OPQ (=β) change by 1 rad, and the length in the y-axis direction is , For example, it is the length in the y-axis direction that changes when ∠SPQ=∠SOQ (=δ) changes by 1 rad. ∠OPS(=α), ∠OPQ(=β) are derived based on the result of the coefficient of the number of rectangles from the rectangle at the position corresponding to the point O to the rectangles corresponding to the points S and Q. Further, based on the result of the coefficient of the number of rectangles from the rectangle at the position corresponding to the point S to the rectangle at the position corresponding to the point Q, ∠SPQ(=δ) is derived. The counting of the number of rectangles is performed based on the result of image processing on the first captured image.

Also, the positions of the display regions 1111, 1121, 1131, and 1141 are derived similarly to the method described with reference to FIG. 18 . In addition, since the display area 201 using the stencil and the display area using the label 202 are different only in terms of arrangement on the back plate, the position of the display area using the label 202 is also referred to in FIG. 18. It can be derived by imitating the method described while doing so. Therefore, its detailed description is omitted here.

Next, in step S1708, an example of a case where it is determined that the position in the height direction (z-axis direction) of the display area by the stencil (label) can be specified will be described.

As a first example, first, the position deriving unit 1206 derives the distance in the height direction (z-axis direction) from the first imaging device to the display area using the stencil using a laser range finder. Then, the position deriving unit 1206 obtains a position obtained by subtracting the distance from the position of the first imaging device in the height direction (z-axis direction) of the display area 201 using the stencil in the height direction (z-axis direction). is derived as the position of

As a second example, the position derivation unit 1206 provides information indicating the position of the back plate stored in the product position column of the management table 1300 and information stored in the registration time column of the management table 1300. , The position in the height direction (z-axis direction) of the display area by the stencil (label) is derived using information indicating the date and time when the back plate is placed in the current position. An example of a method for deriving the position in the height direction (z-axis direction) of the display area by the stencil (label) in this way will be described. In addition, here, the three-dimensional position of the back plate stored in the article position column of the management table 1300 is referred to as an article position as necessary. In addition, here, in order to simplify explanation, it is assumed that the lowermost thick plate among the plurality of stacked thick plates is placed on the ground.

First, the position deriving unit 1206 detects that at least a part is included in the imaging range (x-y coordinates) of the first imaging device that has captured the first captured image acquired in step S1701 from a back plate other than the back plate specified in step S1705. The back plate is specified based on the article position stored in the management table 1300 .

In the case of deriving the position in the height direction (z-axis direction) of the display area 201 by the stencil, the back plate specified in step S1705 is at the top. Accordingly, the position deriving unit 1206 calculates that the back plate at least partially included in the imaging range (x-y coordinates) of the first imaging device that captured the first captured image acquired in step S1701 is below the thick plate specified in step S1705. judge that there is In this case, the value obtained by adding all of the sizes (thicknesses) stored in the management table 1300 as the information of the thick plate under the thick plate specified in step S1705 and the thickness of the thick plate specified in step S1705 is This is the position in the height direction (z-axis direction) of the display area 201 .

On the other hand, in the case of deriving the position in the height direction (z-axis direction) of the display area by the label, the position deriving unit 1206 manages as information of the thick plate specified in step S1705 among the thick plates specified as described above. It is determined that the thick plate whose date and time later than the date and time stored in the registration time column of table 1300 is stored in the registration time column is the thick plate located below the thick plate specified in step S1705. In this case, all the added values of the size (thickness) stored in the management table 1300 as the information of the heavy plate under the heavy plate specified in step S1705 and the information of the heavy plate specified in step S1705 are stored in the management table 1300 A value obtained by adding the values of the z-coordinates of the stored positions in the article becomes the position in the height direction (z-axis direction) of the display area by the label 202 .

As a third example, when information capable of specifying the height direction (z-axis direction) is displayed in the first captured image, the position deriving unit 1206 displays a stencil (label) based on the information. The position of the region in the height direction (z-axis direction) can be derived. An example of a method for deriving the position in the height direction (z-axis direction) of the display area by the stencil (label) in this way will be described. For example, the position deriving unit 1206 recognizes the scale of the structure from the first captured image when a structure marked with a scale indicating the position in the height direction (z-axis direction) is included in the first captured image. . For example, the position derivation unit 1206 extracts the edge of an object by executing edge detection processing on the first captured image, and extracts the structure concerned from the extracted object. For example, information representing the outline of an object and information related to the name of the object are registered in the tracking device 1200 in advance. In this case, the position derivation part 1206 may extract the object corresponding to the said structure by comparing the edge of the object extracted as mentioned above, and the said information. In addition, the position derivation part 1206 may extract the said structure from the object extracted as mentioned above using AI (Artificial Intelligence), such as NN (Neural Network).

Then, the position deriving unit 1206 determines the stencil based on the position of the display area by the stencil (label) in the first captured image and the position of the scale displayed on the structure in the first captured image. The position in the height direction (z-axis direction) of the display area by (label) is derived. In addition, the numbers attached to the graduations in the first captured image are recognized by a method similar to the method of recognizing components of display items using stencils (labels).

As a fourth example, the position derivation unit 1206 uses the second captured image acquired by the second image acquisition unit 1209 to determine the height direction (z-axis direction) position of the display area 201 using the stencil. can derive In this case, in the flowchart of Fig. 17, a step of acquiring the second captured image by the second image acquiring unit 1209 is added prior to the timing at which the process of step S1708 starts. The second captured image is a captured image captured by a second imaging device different from the first imaging device (an imaging device that captures an area including a display area with stencils (labels)) that captured the first captured image. The second captured image includes both the first imaging device that captured the first captured image acquired in step S1701 and the back plate specified in step S1705. An example of a method for deriving the position in the height direction (z-axis direction) of the display area 201 using the stencil in this way will be described.

19 is a diagram for explaining an example of a method for deriving the position of the display area 201 in the height direction (z-axis direction) using a stencil.

Also in the example shown in FIG. 19 , as in FIG. 18 , point P indicates the position of the first imaging device (xyz coordinates of point P are (x _p , y _p , z _p )). In addition, the z-axis is a direction perpendicular to the surface of the back plate (the surface on which the display area 201 by the stencil is formed). In addition, the y-axis is a direction parallel to the direction in which components of the same display item are arranged in the display area 201 using the stencil. The x-axis is a direction parallel to the direction in which other display items are arranged in the display area 201 using the stencil. Also, the x-axis, y-axis, and z-axis are assumed to be perpendicular to each other. Again, for simplicity of explanation, it is assumed that the width direction of the thick plate is parallel to the y-axis, and the longitudinal direction of the thick plate is parallel to the x-axis.

In addition, the point O' is the intersection of the perpendicular line of the point P (with respect to the xy plane) and the plane including the surface of the back plate (the surface on which the display area 201 by the stencil is formed) (the point O' is shown in FIG. Corresponds to point O in 18). Point O is the foot of the perpendicular to point P (with respect to the xy plane) (the intersection of the perpendicular to point P with the xy plane). In the example shown in Fig. 19, it is assumed that the point O is the origin. Point R represents the three-dimensional position of the second imaging device (xyz coordinates of point R are (x _r , y _r , z _r )). The three-dimensional position of the second imaging device is, for example, the center position of the optical system (lens) of the second imaging device.

It is assumed that the 3-dimensional position of the second imaging device can be specified by the tracking device 1200 . For example, when the 3D position of the second imaging device is fixed, the ID of the second imaging device and the 3D coordinates (x-y-z coordinates) of the second imaging device are related to each other and registered in the tracking device 1200 in advance. do. In addition, when the 3D position of the second imaging device is not fixed, for example, the second imaging device determines the 3D coordinates (x-y-z coordinates) of the second imaging device positioned using GPS and the second imaging device send the ID of In this case, the second image acquiring unit 1209 acquires these pieces of information transmitted from the second imaging device.

In the example shown in Fig. 19, it is assumed that the lowermost thick plate among the plurality of stacked thick plates is placed on the ground for simplicity of explanation. The point D is a vertex located closest to the point R among the vertices of the surface on which the display area 201 by the stencil is formed. The point B is a vertex located closest to the point R among the vertices of the bottom surface of the lowermost thick plate among the stacked plurality of thick plates.

19 , ∠BRD(=φ _B ) and ∠PRO(=φ _P ) denote positions of points P, B, and D in a second captured image, and the second captured image, for example. It is determined based on the position of the center of the optical system (lens) at the time of imaging (the position of the center of the optical system used to determine the focal length). The direction in which the line segment RP extends is specified by a straight line connecting the point P in the second captured image and the center position of the optical system (lens) when the second captured image is captured. The direction in which the line segment RB extends is specified by a straight line connecting the point B in the second captured image and the center position of the optical system (lens) when the second captured image is captured. The direction in which the line segment RT extends is specified by a straight line connecting the point D in the second captured image and the center position of the optical system (lens) when the second captured image is captured. ∠BRD(=φ _B ) is determined based on the direction in which segment RB extends and the direction in which segment RD extends. ∠PRO(=φ _P ) is determined based on the direction in which the line segment RP extends and the y-axis direction.

Further, for example, the position deriving unit 1206 determines the length of the line segment RP based on the measured value of the range sensor (eg, a time-of-flight (TOF) type range sensor) provided in the second imaging device. (distance from point R to point P) and the length of segment RD (distance from point R to point D). Then, the position deriving unit 1206 determines the first image pickup based on the position (z _r ) of the second imaging device in the height direction (z-axis direction), the length of the line segment RP, and ∠PRO (=φ _P ). The position (z _p ) in the height direction (z-axis direction) of the device is derived (z _p =z _r + length of line segment RP×sinφ _P ). Similarly, the position deriving unit 1206 determines the position (z _r ) of the second imaging device in the height direction (z-axis direction), the length of the line segment RD, and ∠BRD (=φ _B ) based on the stencil. The position of the display area 201 in the height direction (z-axis direction) is derived.

19 illustrates the case where the position of the second imaging device (point R) in the height direction (z-axis direction) is 0 (zero). In addition, when the second captured image is used as described above, the position deriving unit 1206 determines not only the position of the display area 201 by the stencil in the height direction (z-axis direction) but also the height direction of the first imaging device ( The position (z _p ) of the z-axis direction) can be derived. In addition, the position deriving unit 1206 can derive not only the position in the height direction (z-axis direction), but also the position (x _p ) and the position (y _p ) of the y-axis direction of the first imaging device in the x-axis direction. there is. For example, x _p and y _p are derived based on the direction in which the line segment RP extends and y _r + the length of the line segment RP × cosφ _P . That is, a vector connecting the points R and P is derived based on the direction in which the line segment RP extends and y _r + the length of the line segment RP × cosφ _P . The length of the x-axis component and the length of the y-axis component of the vector are x _p , y _p .

In addition, here, the case where the point P (the first imaging device) is included in the second captured image captured by the second imaging device has been described as an example. However, it is not always necessary to do this as long as the relative positional relationship (eg, ∠PRO(=φ _P )) between the first imaging device (point P) and the second imaging device (point R) can be derived. In this case, a captured image captured by at least one imaging device different from the first imaging device and the second imaging device is used, for example. A specific example of such a case will be described. First, a third imaging device is disposed in addition to the first imaging device and the second imaging device. In the third imaging device, point P (first imaging device) and point R (second imaging device) are included in the imaging range, and point D is not included in the imaging range. In addition, a point indicating the position of the third imaging device is set to R' (not shown). In addition, it is assumed that one or both of the position of the point R indicating the position of the second imaging device and the position of the point R' indicating the position of the third imaging device are determined in advance. In this case, the position derivation unit 1206 derives the elevation angle or depression angle when viewing the point R and the point P from the point R' based on the third captured image captured by the third imaging device, and the position The derivation unit 1206 derives ∠PRO(=φ _P ) based on the derived elevation angle or depression angle. Derivation of the angle, direction, and length is realized by a method similar to the method described with reference to FIG. 19 . In addition, when point P (first imaging device) is included in the imaging range of the third imaging device and point R (second imaging device) is not included, for example, point R' (third imaging device) and the fourth captured image captured by the fourth imaging device including the point R (second imaging device) in the imaging range may be further used. On the other hand, for example, when point R (second imaging device) is included in the imaging range of the third imaging device and point P (first imaging device) is not included, for example, point R' (third imaging device) is included. The fifth captured image captured by the fifth imaging device including the imaging device) and the point P (the first imaging device) in the imaging range may be further used.

Further, the position deriving unit 1206 may specify the position of the point P (first imaging device) using a sixth captured image captured by a sixth imaging device including the first imaging device in an imaging range. . In this case, it is assumed that the sixth imaging device is included in the imaging range of at least one of the second imaging device, the third imaging device, and the fourth imaging device. In addition, a point indicating the position of the sixth imaging device is set to P' (not shown). In addition, it is assumed that one or both of the position of the point P indicating the position of the first imaging device and the position of the point P' indicating the position of the sixth imaging device are determined in advance. In this case, for example, calculation using the sixth imaging device and point P' instead of the first imaging device and point P in the above description is performed. Then, for example, based on the result of this calculation, the sixth captured image, and the positions of one or both of the points P and P', the first imaging device (point P) and the second imaging device (point R) The relative positional relationship of (∠PRO(=φ _P ), etc.) is derived.

As described above, even when the second imaging device cannot be arranged so that both the first imaging device and the rear plate are included in the imaging range of the second imaging device, the height direction (z-axis) of the display area 201 by the stencil direction) can be derived. In addition, the 3rd to 6th captured images are acquired by the 2nd image acquisition part 1209, for example. However, apart from the second image acquisition unit 1209, third to sixth image acquisition units that individually acquire the third to sixth captured images may be used. In this case, in the flowchart of FIG. 17, the timing prior to the timing at which the process of step S1708 starts, and the timing at which the step of acquiring the second captured image by the second image acquisition unit 1209 is executed, is a different timing. To this, a step of separately acquiring the third to sixth captured images is added. In addition, if the position in the height direction (z-axis direction) of the display area by the stencil (label) can be specified without acquiring the second captured image or the like, the second image acquisition unit 1209 becomes unnecessary.

As a fifth example, the position derivation unit 1206 conveys the positions of the thick plates 1002a and 1002b placed on the conveyance table 1001 in the height direction (z-axis direction) of the display area 201 using the stencil. It may be derived based on the position of the table 1001 and the size (thickness) stored in the management table 1300 as information on the back plates 1000a and 1002b. In this case, the position of the transport table 1001 is registered in the tracking device 1200 in advance.

For example, in at least one of the above examples, when the position in the height direction (z-axis direction) of the display area can be derived using the stencil (label), in step S1708, display using the stencil (label) It is determined that the position in the height direction (z-axis direction) of the region can be specified.

Further, when the heavy plate is in the carrying tool, in step S1709, the position deriving unit 1206 acquires the three-dimensional position of the heavy plate in the carrying tool. In this embodiment, the transportation tools are the transport ship 400 and the transport vehicle 700 . So, in this embodiment, the case where the transport tools are the transport ship 400 and the transport vehicle 700 is exemplified.

For example, when a heavy plate is loaded onto the transport ship 400, the position deriving unit 1206 acquires the ID of the heavy plate and information indicating the three-dimensional position of the heavy plate in the transport ship 400. And the position derivation part 1206 specifies the column in which the information which shows the same ID as the ID of the acquired thick plate is memorize|stored from the column of ID of the management table 1300. Then, the position deriving unit 1206 stores the obtained information indicating the three-dimensional position of the thick plate in the transport ship 400 in the article position column of the specific row. For example, even if the operator (worker) of the transport ship 400 inputs the ID of the thick plate and the information indicating the three-dimensional position of the heavy plate in the transport ship 400 to an information processing terminal (such as a tablet terminal) possessed by the transport ship 400. do. In this case, the heavy plate ID and the information indicating the three-dimensional position of the thick plate in the transport ship 400 are transmitted from the information processing terminal to the tracking device 1200, so that the position derivation unit 1206 determines the thick plate It is also possible to obtain information indicating the ID of and the three-dimensional position of the thick plate in the transport ship 400. The information indicating the three-dimensional position of the back plate is, for example, x-y-z coordinates of a display area by a stencil (label).

Further, when the heavy plate is loaded onto the transport vehicle 700, the position deriving unit 1206 acquires the ID of the heavy plate and information indicating the three-dimensional position of the heavy plate in the transport vehicle 700. Then, the position derivation unit 1206 specifies, from the ID column of the management table 1300, a column in which information indicating the same ID as the ID of the acquired back plate is stored. Then, the position derivation unit 1206 stores the obtained information indicating the three-dimensional position of the rear plate in the transport vehicle 700 in the article position column of the specific row. For example, an information processing terminal (such as a tablet terminal) possessed by an operator (worker) of the transport vehicle 700 transmits information indicating the ID of the heavy plate and the three-dimensional position of the heavy plate in the transport vehicle 700. You may enter In this case, by transmitting the ID of the heavy plate and the information indicating the three-dimensional position of the heavy plate in the transport vehicle 700 from the information processing terminal to the tracking device 1200, the position derivation unit 1206, The ID of the thick plate and information indicating the three-dimensional position of the thick plate in the transportation vehicle 700 may be acquired. The information indicating the three-dimensional position of the back plate is, for example, x-y-z coordinates of a display area by a stencil (label).

As described above, by acquiring the three-dimensional positions of the heavy plates in the carrying tool, it is possible to specify the carrying tool for carrying each heavy plate. In addition, the three-dimensional position of the heavy plate can be accurately specified when the heavy plate is being transported by the conveying tool.

Returning to the description of Fig. 17, in step S1708, when it is determined that the position in the height direction (z-axis direction) of the display area by the stencil (label) cannot be specified, the process of step S1712 is executed. In step S1712, the output unit 1208 outputs information indicating that the three-dimensional position of the back plate cannot be specified and the ID of the first imaging device that captured the image of the back plate. For example, the output unit 1208 displays information indicating that the three-dimensional position of the back plate cannot be specified and the ID of the first imaging device that captured the back plate on the computer monitor. When these pieces of information are output, the operator (worker) goes to the position where the corresponding first imaging device exists and specifies the three-dimensional position of the corresponding rear plate. Further, the operator (worker) may specify the three-dimensional position of the corresponding rear plate from the preview image of the corresponding first imaging device.

Next, in step S1713, the storage unit 1207 inputs information indicating the three-dimensional position of the heavy plate specified by the operator (worker), and displays non-actuated items in the management table 1300 as information on the heavy plate. The information stored in the item position column in the column of is updated with the input information. In addition, as an input form of information indicating the three-dimensional position of the back plate, input by user interface operation of the tracking device 1200, reception of information transmitted from an external device, and reading of information stored in a portable storage medium , at least one can be employed. Then, processing according to the flowchart of Fig. 17 ends.

<integration>

As described above, in the present embodiment, the tracking device 1200 is information displayed on the back plate, and the information managed by the manufacturer of the back plate is the control of the area including the display area displayed by the stencil (label). 1 Acquire a captured image. The tracking device 1200 derives a three-dimensional position of a display area using a stencil (label) based on the acquired first captured image and the position of the first imaging device that captured the first captured image, and The information indicating the three-dimensional position is stored in the storage medium in association with the ID of the back plate. Therefore, for example, when a heavy plate is conveyed using various conveying devices (transport ship 400, transport vehicle 700, conveying table 1001, etc.), or when a worker conveys a heavy plate, Even when the back plate is placed in an unspecified place or when the back plate is stacked, the back plate can be tracked. Accordingly, the tracking accuracy of the thick plate can be improved.

Further, in the present embodiment, the tracking device 1200 recognizes information (ID305, 312) displayed in the display area using a stencil (label) based on the first captured image. Then, the tracking device 1200 collates the recognized information with the information (ID) stored in the management table 1300, and identifies the thick plate based on the result of the collation. Accordingly, the thick plate itself as the tracking target can be reliably specified based on the first captured image. Therefore, compared to the case where the three-dimensional position of the thick plate is specified based on the operation of the conveying machine, for example, it is possible to improve the tracking accuracy of the thick plate.

Further, in this embodiment, the tracking device 1200 acquires a second captured image captured by a second imaging device different from the first imaging device that captured the first captured image. The second captured image includes the first captured image and the back plate (to be tracked). The tracking device 1200 derives the position in the height direction (z-axis direction) of the display area 201 using the stencil based on the acquired second captured image and the position of the second imaging device. Then, the tracking device 1200 is located in the height direction (z-axis direction) of the display area 201 by the stencil, the first captured image, and the position of the first imaging device that captured the first captured image. Based on this, the 3D position of the display area 201 by the stencil is derived as the 3D position of the back plate. Therefore, even if the first captured image or the sensor included in the first captured image is used, even when the height direction position of the back plate cannot be derived, the back plate can be tracked.

Further, in the present embodiment, a display using a stencil and a display using a label are regarded as information that can be identified by a person. Therefore, management of the actual item by the operator (worker) becomes easy. Therefore, the convenience of actual product management is improved.

Further, in the present embodiment, information determined individually for each back plate (information unique to the back plate) is included in the display using the stencil and the display using the label. Therefore, the back plate can be uniquely identified based on the first captured image. Therefore, identification of the thick plate can be performed easily and quickly.

In the present embodiment, the tracking device 1200 derives three-dimensional coordinates (x-y-z coordinates) as the position of the display area using the stencil (label). Therefore, even when a plurality of back plates are stacked, for example, the back plates can be tracked.

<Example of modification>

<<First modification>>

The method of deriving the position of the display area by the stencil (label) included in the first captured image in step S1709 is not limited to the method described with reference to FIG. 18 . For example, the position derivation unit 1206 derives the position of the origin at the time of determining the position within the article in the management table 1300 from the first captured image. Then, the position derivation unit 1206 based on the derived position of the origin and the information stored in the item position column of the management table 1300, based on the stencil (label) included in the first captured image. The location of the display area is derived. In this way, an example of a method of deriving the position of the display area by the stencil (label) included in the first captured image will be described taking the display area 201 by the stencil as an example.

20 is a diagram for explaining an example of a method of deriving the position of the display area 201 using a stencil. In FIG. 20 , as in FIG. 18 , a case where the xyz coordinates S(x _s , y _s , z _s ) of the point S are derived as the position of the display area 201 by the stencil is exemplified. As described with reference to FIG. 14 , the position of the stencil in the product is the vertex 203 shown in FIG. 2 as the origin, and the direction from the vertex 203 along the width direction of the back plate is the positive direction of the y-axis, It is indicated by xy coordinates with the direction along the longitudinal direction of the thick plate as the positive direction of the x-axis. Therefore, the position of point E in FIG. 20 becomes the origin of the position of the stencil in the article. Point F is a point aligned in the width direction (y-axis direction) with respect to point E among vertices of the thick plate (one side along the width direction of the thick plate is formed by point E and point F). The point T is the foot of the perpendicular of the point P to the line segment EF (the intersection of the perpendicular to the point P and the segment EF). Point P and point O are the same as those shown in FIG. 18 .

In Fig. 20, the length of the line segment PT is h _p . In addition, the length of the line segment PF is h ₁ , ∠FPT is θ ₁ (rad), the length of line segment PE is h ₂ , ∠EPT is θ ₂ (rad), ∠EPF is θ (rad ) (θ=θ ₁ +θ ₂ ). In addition, let the length of the line segment FT be w ₀₁ , and let the length of the line segment ET be w ₀₂ .

Then, the x-coordinate x _e of the point E is the value obtained by adding the length l _p of the line segment OT to the x-coordinate x _p of the point P (x _e =x _p +l _p ). The y-coordinate y _e of the point E is the value obtained by subtracting the length w _o2 of the line segment ET from the y-coordinate y _p of the point P (y _e =y _p -w _o2 ). Therefore, when the length l _p of the line segment OT and the length w _o2 of the line segment ET are derived, the xy coordinates (x _e , y _e ) of the position of point E (the origin of the position of the stencil in the article) are derived.

The length l _p of the line segment OT is expressed by the following formula (5).

l _p =√(h _p ² -H ² )=√(h ₁ ² cos ² θ ₁ -H ² )=√(h ₂ ² cos ² θ ₂ -H ² ) ...(5)

The length w _o2 of the line segment ET is expressed by the following formula (6).

w _o2 = h ₂ sinθ ₂ ... (6)

Therefore, for example, when the length h ₂ of line segment PE and ∠EPT(=θ ₂ ) are derived, the length l _p of line segment OT and the length w _o2 of line segment ET are derived. The length H of the line segment OP is derived as described with reference to FIG. 18 .

The position of the point T in the first captured image is a position on the line segment EF in the first captured image, and is a position corresponding to the center of the horizontal angle of view (angle of view in the y-axis direction) during imaging. ∠EPT(=θ ₂ ) is, for example, the positions of points E and T in the first captured image acquired in step S1701 and the center position of the optical system (lens) when the first captured image is captured ( It is determined based on the position of the center of the optical system used to determine the focal length). The direction in which the line segment PE extends is specified by a straight line connecting the point E in the first captured image and the center position of the optical system (lens) when the first captured image is captured. The direction in which the line segment PT extends is specified by a straight line connecting the point T in the first captured image and the center position of the optical system (lens) when the first captured image is captured. ∠EPT(=θ ₂ ) is determined based on the direction in which the line segment PE extends and the direction in which the line segment PT extends.

The length h ₂ of the line segment PE is derived using, for example, a range sensor (eg, a time-of-flight (TOF) type range sensor) included in the first imaging device. In addition, the length h ₂ of the line segment PE may be derived based on the length H of the line segment OP and ∠OPE.

As described above, the x coordinate x _e and the y coordinate y _e of the point E are derived. Then, the position derivation unit 1206 determines the xy coordinates (Ls10, Ls10, Ws10) is read. Then, the position derivation unit 1206 derives a value obtained by adding LS10 to the x coordinate x _e of the point E as the x coordinate x _s of the point S (x _s =x _e +LS10). Further, the position derivation unit 1206 derives a value obtained by adding WS10 to the y coordinate x _y of the point E as the y coordinate y _s of the point S (y _s =y _e +WS10).

<<Second modified example>>

As in the present embodiment, customer name 302, specification 303, size 304, ID 305, customer code 306, order number 307, specification 311, ID 312, When the size 313, the delivery date 314, and the demand code 315 are information that can be identified by a person, it is preferable because the operator (worker) can easily manage the actual product. However, the customer name (302), size (303), size (304), ID (305), customer code (306), order number (307), size (311), ID (312), size (313) ), the delivery date 314, and the demand code 315 may be converted (encoded) into information that is not humanly identifiable, such as a bar code or a two-dimensional code.

<<The third modified example>>

In this embodiment, as shown in Figs. 3A and 3B, the case where the ID 305 displayed by the stencil and the ID 312 displayed by the label are the same has been exemplified. However, there are cases where the expression and description of these IDs are different. For example, there are forms without "hyphens" in the ID. In this case, information indicating that these IDs (ID by stencil, ID by label) are for the same back plate is stored in the management table 1300. Also for display items other than ID, the expression and description of the same display item may differ between display using a stencil and display using a label. For example, when a standard is displayed on both a stencil and a label, for example, an abbreviation of a standard may be displayed on the label. In addition, although the number of digits of the consumer code may differ between the display by the stencil and the display by the label, the consumer code is managed by the manufacturer. Also, regarding the size, there are cases where all digits of the dimension are not indicated. Even in this case, information indicating that these other expressions and descriptions have equivalent meanings is stored in the management table 1300.

<<The 4th modified example>>

In the present embodiment, the case where the xyz coordinates of the display area by the stencil (label) is derived as the three-dimensional position of the display area by the stencil (label) is exemplified. However, the three-dimensional position of the display area by the stencil (label) is not limited to xyz coordinates. For example, the 3D position of the display area by the stencil (label) may be a relative 3D position based on the position of the first imaging device. In this case, it is not necessary to use the coordinates (x _p , y _p , z _p ) of the point P indicating the 3-dimensional position of the imaging device. In addition, when the origin of the xyz coordinates is the 3-dimensional position of the first imaging device, the 3-dimensional position of the display area by the stencil (label) is based on the 3-dimensional position of the 1st imaging device even if expressed in xyz coordinates. It is to indicate the relative 3-dimensional position to be.

<<Fifth modified example>>

In this embodiment, the case where the identification information acquisition unit 1210 has a recognition unit 1202, an acquisition unit 1203, a verification unit 1204, and a specification unit 1205 has been exemplified. However, if the identification information acquisition unit 1210 acquires the identification information of the thick plate associated with the three-dimensional position derived by the position derivation unit 1206, the recognition unit 1202, the acquisition unit 1203, It is not necessary to have the matching section 1204 and the specifying section 1205. The identification information acquisition unit 1210 may have only the recognition unit 1202, for example. In addition, the identification information acquisition unit 1210 may have only the acquisition unit 1203, for example. In addition, the identification information acquisition unit 1210 may have only the recognition unit 1202 and acquisition unit 1203, for example. For example, the identification information acquisition unit 1210 may acquire the ID recognized by the recognition unit 1202 as the identification information of the back plate. In this case, the identification information acquisition unit 1210 does not need to include the acquisition unit 1203, the verification unit 1204, and the specification unit 1205. In addition, for example, when the tracking target item is limited to one item in advance, the identification information acquisition unit 1210 may acquire the ID acquired by the acquisition unit 1203 as the identification information of the back plate. In this case, the identification information acquisition unit 1210 does not have to include the recognition unit 1202, the verification unit 1204, and the specification unit 1205. In addition, for example, the identification information acquisition unit 1210 uses the recognition unit 1202 when specifying a thick plate from only the information recognized by the recognition unit 1202, and the items to be tracked are one in advance. If limited, you may use the acquisition part 1203. In this case, the identification information acquisition unit 1210 does not have to have the verification unit 1204 and the specification unit 1205 (it may have only the recognition unit 1202 and the acquisition unit 1203).

<<Sixth modified example>>

In this embodiment, the case where the management table 1300 is stored in the tracking device 1200 has been exemplified. However, the management table 1300 may exist outside the tracking device 1200.

<<Seventh modification>>

In this embodiment, the case where the article is a thick plate is exemplified. However, the article is not limited to the thick plate. For example, the article may be a product other than a thick plate. The article may be a product on which information managed by the manufacturer is displayed. The product may be, for example, a slab, a billet, a sheet coil or a steel pipe. Moreover, it may be an article other than a manufactured product. The article may be, for example, a container.

<<other modifications>>

In addition, the embodiment of the present invention described above can be realized by a computer executing a program. In addition, a computer readable recording medium in which the above program is recorded and a computer program product such as the above program are also applicable as embodiments of the present invention. As the recording medium, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used. In addition, the embodiment of the present invention may be realized by a programmable logic controller (PLC) or dedicated hardware such as an application specific integrated circuit (ASIC).

In addition, all of the embodiments of the present invention described above are only examples of concrete implementation in implementing the present invention, and the technical scope of the present invention should not be construed as being limited by these. That is, the present invention can be implemented in various forms without departing from its technical idea or its main characteristics.

<hardware>

An example of the hardware of the tracking device 1200 will be described. 21, a tracking device 1200 includes a CPU 2101, a main memory device 2102, an auxiliary memory device 2103, a communication circuit 2104, a signal processing circuit 2105, an image processing circuit 2106, It has an I/F circuit 2107, a user interface 2108, a display 2109 and a bus 2110.

The CPU 2101 collectively controls the entire tracking device 1200. The CPU 2101 uses the main storage device 2102 as a work area to execute programs stored in the auxiliary storage device 2103. The main memory device 2102 temporarily stores data. The auxiliary storage device 2103 stores various types of data in addition to programs executed by the CPU 2101 .

The communication circuit 2104 is a circuit for communicating with the outside of the tracking device 1200. The communication circuit 2104 may perform wireless communication or wired communication with the outside of the tracking device 1200.

The signal processing circuit 2105 performs various signal processes on signals received by the communication circuit 2104 and input signals under control by the CPU 2101.

The image processing circuit 2106 performs various image processes on input signals under the control of the CPU 2101. A signal subjected to this image processing is output to the display 2109, for example.

The user interface 2108 is a part where the operator gives instructions to the tracking device 1200. The user interface 2108 has, for example, buttons, switches, and dials. In addition, the user interface 2108 may have a graphical user interface using the display 2109 .

The display 2109 displays an image based on a signal output from the image processing circuit 2106 . The I/F circuit 2107 exchanges data between devices connected to the I/F circuit 2107. In FIG. 21, as devices connected to the I/F circuit 2107, a user interface 2108 and a display 2109 are shown. However, devices connected to the I/F circuit 2107 are not limited to these. For example, a portable storage medium may be connected to the I/F circuit 2107. In addition, at least a part of the user interface 2108 and the display 2109 may exist outside the tracking device 1200 .

The output unit 1208 is obtained by using at least one of, for example, the communication circuit 2104, the signal processing circuit 2105, the image processing circuit 2106, the I/F circuit 2107, and the display 2109. come true

Also, the CPU 2101, main memory device 2102, auxiliary memory device 2103, signal processing circuit 2105, image processing circuit 2106, and I/F circuit 2107 are connected to the bus 2110. . Communication between these components is performed via the bus 2110. In addition, the hardware of the tracking device 1200 is not limited to that shown in FIG. 21 as long as it can realize the functions of the tracking device 1200 described above.

The present invention can be used for tracking articles, for example.

Claims (8)

It is a tracking device for tracking goods,
one or two or more image acquisition means for acquiring a captured image of a region including display information displayed on the article;
identification information acquisition means for acquiring identification information of the article;
position derivation means for deriving an information display position, which is a three-dimensional position at which the display information is displayed, based on the captured image acquired by the image acquisition means;
and storage means for storing the information display position of the article derived by the position derivation means in a storage medium in association with the identification information of the article acquired by the identification information acquisition means. tracking device. The method of claim 1, wherein the display information displayed on the article includes identification information of the article,
The identification information acquisition means,
recognizing means for recognizing identification information of the article based on the captured image acquired by the image acquiring means;
The tracking device characterized by having at least one means of acquisition means for acquiring the identification information of the article stored in the storage means. The information display position according to claim 2, wherein the information display position stored in the storage medium in association with the identification information of the item is information about a three-dimensional position where the display information is displayed, based on a predetermined position of the item. Including information about the three-dimensional position to be,
The tracking device according to claim 1, wherein the captured image acquired by the image acquisition means includes a region including a predetermined position of the article. The method of claim 2 or 3, wherein the identification information acquisition means,
the recognition means; and
the means of obtaining; and
Verification means for comparing the identification information of the article recognized by the recognition means with the identification information of the article obtained by the acquisition means;
and specifying means for specifying identification information of the article included in the captured image acquired by the image acquisition means, based on a result of collation by the verification means. 5. The method according to any one of claims 1 to 4, wherein the position derivation means determines, based on two or more captured images acquired by the two or more image acquisition means, the surface of the article on which the display information is displayed. A tracking device characterized by deriving a position in the height direction. It is a tracking method for tracking goods,
one or two or more image acquisition steps of acquiring a captured image of a region including display information displayed on the article;
An identification information acquisition step of acquiring identification information of the article;
a position derivation step of deriving an information display position, which is a three-dimensional position at which the display information is displayed, based on the captured image acquired by the image acquisition step;
and a storage step of storing the information display position of the article derived by the position derivation step in a storage medium in association with the identification information of the article acquired by the identification information acquisition step. tracking method. display information displayed on the article;
Has non-display information that is not displayed on the article;
The non-display information includes an information display position, which is a three-dimensional position where the display information is displayed;
A data structure of tracking data used by a tracking device to derive the information display position on the article based on a captured image of an area containing display information displayed on the article. It is a program for executing processing of tracking an article on a computer,
one or two or more image acquisition steps of acquiring a captured image of a region including display information displayed on the article;
An identification information acquisition step of acquiring identification information of the article;
a position derivation step of deriving an information display position, which is a three-dimensional position at which the display information is displayed, based on the captured image acquired by the image acquisition step;
causing a computer to execute a storage step of storing the information display position of the article derived by the position derivation step in a storage medium in association with the identification information of the article acquired by the identification information acquisition step; featured program.

Images