WO2023233506A1 - Article management device, article management method, data structure of data for article management, and program - Google Patents

Abstract

In the present invention, an article management device (1200) derives the three-dimensional position of the display area using a stencil (label) on the basis of a first captured image and the position of a first imaging device with which the first captured image has been captured, and stores information indicating the three-dimensional position in a storage medium in association with information for identifying the thick plate plate.

Description

Article management device, article management method, data structure of article management data, and program

The present invention relates to an article management device, an article management method, a data structure of article management data, and a program, and is particularly suitable for use in managing articles.

The position of objects such as steel products such as thick plates is managed.
In the technique described in Patent Document 1, the position of the steel plate is managed by registering the XY coordinates of the overhead crane when the steel plate is lowered together with the steel plate number of the steel plate.

In addition, in the technology described in Patent Document 2, based on operations such as horizontal and vertical movement of a hoist and load detection, a fixed location calculation program calculates a fixed location for steel materials indicated by three-dimensional coordinates, The fixed location information indicating the calculated fixed location is stored in association with the steel material identification information of the steel material.

Patent No. 5109451 Patent No. 3734419

However, in the technology described in Patent Document 1, the information used to manage the position of the steel plate is the XY coordinates of the overhead crane. For this reason, the position of articles can only be managed within the range that can be transported by an overhead crane. Furthermore, with the technology described in Patent Document 1, it is not possible to specify the Z coordinate from the XY coordinates of the overhead crane, so for example, when a plurality of steel plates are piled up, it is difficult to accurately specify the plurality of steel plates. Can not.

Furthermore, in the technique described in Patent Document 2, the three-dimensional coordinates of the steel material are calculated. However, similar to Patent Document 1, the position of the article can only be managed within the range that can be transported by the hoist. Further, the three-dimensional coordinates of the steel material calculated by the technique described in Patent Document 2 are the three-dimensional coordinates of the fixed location calculated by the fixed location calculation program. Therefore, with the technique described in Patent Document 2, the position of the steel material can be managed only at the fixed location that is predetermined by the fixed location calculation program.

At actual manufacturing sites, products may be transported by various transport devices. Further, there are cases where a worker transports the product without using a transport device. In addition, manufactured products may be placed in storage areas for which no storage area address has been determined. Furthermore, there are various ways to place products. With the techniques described in Patent Documents 1 and 2, it is not easy to manage the positions of such products. These problems also exist in articles other than manufactured products.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to enable appropriate management of the positions of articles.

The article management device of the present invention is an article management device that manages the position of articles, and includes one or more image acquisition means for acquiring a captured image of an area including display information displayed on the article; identification information acquisition means for acquiring identification information of an article; and position derivation means for deriving an information display position, which is a three-dimensional position where the display information is displayed, based on the captured image acquired by the image acquisition means. and storage means for storing the information display position on the article derived by the position derivation means in a storage medium in association with the identification information of the article acquired by the identification information acquisition means. shall be.

The article management method of the present invention is an article management method for managing the position of an article, and includes one or more image acquisition steps of acquiring a captured image of an area including display information displayed on the article; an identification information acquisition step of acquiring identification information of the article; and a position derivation step of deriving an information display position, which is a three-dimensional position where the display information is displayed, based on the captured image acquired in the image acquisition step. and a storage step of storing the information display position on the article derived in the position derivation step in a storage medium in association with the identification information of the article acquired in the identification information acquisition step. shall be.

The data structure of the article management data of the present invention includes display information that is displayed on the article and hidden information that is not displayed on the article, and the hidden information is a three-dimensional item on which the display information is displayed. The information display position is a position, and is used by an article management device to derive the information display position on the article based on a captured image of a region including display information displayed on the article.

The program of the present invention is a program for causing a computer to execute processing for managing the position of an article, and the program acquires one or more images of an area containing display information displayed on the article. an identification information acquisition step of acquiring identification information of the article, and an information display position that is a three-dimensional position where the display information is displayed based on the captured image acquired by the image acquisition step. and a storage step of storing the information display position on the article derived in the position derivation step in a storage medium in association with the identification information of the article acquired in the identification information acquisition step. It is characterized by being executed by a computer.

FIG. 1 is a diagram illustrating an example of the flow of products in a factory. FIG. 2 is a diagram showing an example of a display area for a thick plate. FIG. 3A is a diagram illustrating an example of display content using a stencil. FIG. 3B is a diagram illustrating an example of display contents by labels. FIG. 4 is a diagram showing an example of how planks are loaded and unloaded onto a transport ship. FIG. 5 is a view of the plank being lifted by the cargo handling device as viewed from the direction in which the gutter extends (y-axis direction). FIG. 6A is a diagram illustrating a first example of a thick plate conveyance mode. FIG. 6B is a diagram illustrating a second example of the transport mode of the thick plate. FIG. 6C is a diagram illustrating a third example of the transport mode of the thick plate. FIG. 6D is a diagram illustrating a fourth example of the transport mode of the thick plate. FIG. 6E is a diagram illustrating a fifth example of the transport mode of the thick plate. FIG. 7 is a diagram showing an example of how planks are loaded and unloaded onto a transport vehicle. FIG. 8 is a diagram showing an example of a state in which thick plates are loaded onto a transport vehicle. FIG. 9A is a diagram showing a first example of the appearance of thick plates placed in a storage area in a factory. FIG. 9B is a diagram showing a second example of the appearance of thick plates placed in a storage area in a factory. FIG. 9C is a diagram showing a third example of the appearance of thick plates placed in a storage area in a factory. FIG. 10 is a diagram showing an example of how a thick plate is processed in a production facility. FIG. 11 is a diagram showing an example of the original plank and the plank after cutting. FIG. 12 is a diagram illustrating an example of the functional configuration of the article management device. FIG. 13 is a diagram showing a first example of the management table. FIG. 14 is a diagram showing an example of the position of the stencil in the article and the position of the constituent elements of each display item in the display using the stencil. FIG. 15 is a diagram showing an example of the position of the label in the article and the position of the constituent elements of each display item in the display by the label. FIG. 16A is a diagram showing a first example of information stored in the management table. FIG. 16B is a diagram showing a second example of information stored in the management table. FIG. 17 is a flowchart illustrating a first example of processing by the article management device. FIG. 18 is a diagram illustrating an example of a method for deriving a three-dimensional position of a display area using a stencil. FIG. 19 is a diagram illustrating an example of a method for deriving the position in the height direction (z-axis direction) of a display area using a stencil. FIG. 20A is a diagram illustrating an example of a first captured image including a display area formed by a stencil. FIG. 20B is a diagram illustrating an example of the geometrical relationship between the inside of the first imaging device and the display area formed by the stencil. FIG. 21 is a diagram illustrating a modification of the method of deriving the three-dimensional position of the display area using a stencil. FIG. 22 is a diagram showing an example of a steel frame of a structure installed on the ground. FIG. 23 is a diagram showing an example of a pipe installed underground. FIG. 24 is a diagram showing a second example of the management table. FIG. 25 is a flowchart illustrating a second example of processing by the article management device. FIG. 26 is a diagram illustrating an example of the hardware configuration of the article management device.

Embodiments of the present invention will be described below with reference to the drawings.
Note that the comparison objects are the same in terms of length, position, size, spacing, etc., as well as cases in which they are strictly the same, and items that are different within the scope of the invention (for example, due to tolerances determined at the time of design). This shall also include items that are different within the scope.

(First embodiment)
First, a first embodiment will be described. In this embodiment, a case will be exemplified in which the position of an article is changed over time, and the position of an article is changed over time. Such location management is also referred to as tracking. Tracking refers to tracking and specifying the position of an object. In this embodiment, the position of various thick plates (for example, thick steel plates) manufactured at multiple steel mills is managed in a place including a factory that performs various treatments such as heat treatment and cutting. Illustrate a case. Therefore, first, an overview of the factory to which this embodiment is applied will be explained.

<Factory overview>
FIG. 1 is a diagram illustrating an example of the flow of products in a factory to which this embodiment is applied.

In Figure 1, plates manufactured at a steel mill are loaded onto a transport ship. The transport ship then docks at a quay near the factory. The planks are unloaded from the transport vessel and loaded onto a transport vehicle (such as a truck). When the plates transported by transport vehicles arrive at the factory, they are placed in a storage area within the factory. The planks placed in the storage area are subjected to processes such as heat treatment and cutting (gas cutting) before being placed in the warehouse. Thereafter, the plates are shipped from the factory to customers.

<Idea>
If the position of the thick plate transported as described above is managed based on the operation of the transport device as in the techniques described in Patent Documents 1 and 2, there will be restrictions on the range in which the position can be managed. Therefore, the present inventors focused on the fact that information managed by the manufacturer of the thick plate is displayed on the thick plate for the purpose of managing the actual product (management of the position of each thick plate). Managing the display position of such information is equivalent to managing the position of a thick plate. If the information managed by the manufacturer of such a thick plate can be specified, the thick plate can be identified. This embodiment is based on this idea.

<Displays given to thick plates>
FIG. 2 is a diagram showing an example of a display area for a thick plate. Note that the x-axis, y-axis, and z-axis shown in FIG. 2 indicate the directional relationship between FIG. 2 and FIGS. 3A, 3B, 14 to 15, 18 to 19, and 21, which will be described later. , which does not necessarily correspond to the xyz coordinate directions shown in FIGS. 4 and 5. Note that the xyz coordinates are, for example, coordinates expressed in the world coordinate system.
In FIG. 2, a display area 201 is a stencil display area. In the example shown in FIG. 2, the stencil display area 201 exists on the plate surface of the thick plate. Note that the broken lines shown in FIG. 2 are virtual lines and are not actually displayed. In the example shown in FIG. 2, the label (side label) 202 is attached to the side surface (thick part) of the thick plate.

FIG. 3A is a diagram illustrating an example of display content using a stencil. Note that, as described above, the xyz coordinates shown in FIG. 3A indicate the directional relationship with FIG. 2, etc. Furthermore, in the xyz coordinates shown in FIG. 3A, a symbol with a black circle (●) inside a white circle (○) indicates that it goes from the back side of the page to the front side. This representation of xyz coordinates is the same in other figures.
In the example shown in FIG. 3A, the stencil display includes a mark 301, a customer name 302, a standard 303, a size 304, an ID 305, a customer code 306, and an order number 307.

The mark 301 is a mark representing the manufacturer of the thick plate. The consumer name 302 is information indicating a consumer (purchaser) of the thick plate. Standard 303 is information indicating the standard of the thick plate. The size 304 is information indicating the size of the thick plate (for example, thickness x width x length). The ID 305 is information for uniquely identifying the thick plate, and in the example shown in FIG. 3A, it is the plate number (the identification number of the thick plate). Therefore, the same ID will not be assigned to different planks. The customer code 306 is information specified by the customer to be attached to the thick plate by the manufacturer of the thick plate. The order number 307 is part of a number for identifying a thick plate order from a customer.

The mark 301, customer name 302, standard 303, size 304, ID 305, customer code 306, and order number 307 are information that can be generally displayed for thick plates. Note that information other than the ID 305 does not need to be included in the stencil display. Furthermore, information other than the information described above may be included in the stencil display.

Note that, for example, the number of digits of the customer code 306 differs depending on the customer, and the notation method of the standard 303 differs depending on the steelworks that manufacture thick plates. Further, for example, the order number 307 may or may not be assigned by the consumer. Furthermore, the sizes of display items may also vary depending on the steelworks that manufacture the thick plates. Further, the position of the display area by the stencil may also differ depending on the steelworks that manufacture the thick plate. As described above, the content, position, and size of the stencil display vary depending on the steelworks that manufacture the thick plates, the customers, and the like.

The mark 301, customer name 302, standard 303, size 304, ID 305, customer code 306, and order number 307 are information managed by the thick plate manufacturer. Further, the mark 301, customer name 302, standard 303, size 304, ID 305, customer code 306, and order number 307 are information that can be identified by a person. Information that can be identified by a person means that a person can understand the meaning of the information when looking at the information. Information that requires machine encoding and decoding to create and read information, such as barcodes and two-dimensional codes, is not human-identifiable information. Specifically, the mark 301, customer name 302, standard 303, size 304, ID 305, customer code 306, and order number 307 are information that does not require machine decoding to read, and are at least letters, numbers, and symbols. Contains one and does not contain information that requires machine decoding to read.

FIG. 3B is a diagram illustrating an example of display contents by labels. Note that, as described above, the xyz coordinates shown in FIG. 3B indicate the directional relationship with FIG. 2, etc. Furthermore, in the xyz coordinates shown in FIG. 3B, a symbol with an x (cross mark) inside a white circle indicates a direction from the front side to the back side of the page. This representation of xyz coordinates is the same in other figures.
In the example shown in FIG. 3B, the label display includes a standard 311, an ID 312, a size 313, a delivery date 314, and a customer code 315. The delivery date 314 is information indicating the delivery date of the thick plate. Standard 311, ID 312, size 313, and customer code 315 are the same as standard 303, size 304, ID 305, and customer code 306, respectively. Note that information other than the ID 312 does not need to be included in the label display. Furthermore, information other than the information described above may be included in the label display. The content, position, and size of the label display also vary depending on the steel mill that manufactures the plate, the customer, etc., similar to the stencil display.

However, the meaning of the same display item for the same plate shall be the same for both stencil and label display. Therefore, when the display using the stencil shown in FIG. 3A and the display using the label shown in FIG. Become something to do. Further, the standard 303 displayed by the stencil and the standard 311 displayed on the label mean the same content. The size 313 and customer code 315 also have the same meaning as the size 304 and customer code 306, respectively. As described above, the standard 303, ID 305, size 304, and customer code 306 are information managed by the thick plate manufacturer. Therefore, the standard 311, ID 312, size 313, and customer code 315 are also information managed by the thick plate manufacturer. Further, the thick plate delivery date 314 is also information managed by the thick plate manufacturer. Further, the standard 311, ID 312, size 313, delivery date 314, and customer code 315 are information that can be identified by a person.
In addition to the above-mentioned markings, the thick plate may be marked with markings, for example. In addition, handwritten markings may be applied to the planks.

<Transportation mode of thick plate>
<<Transportation by transport ship>>
FIG. 4 is a diagram showing an example of how planks are loaded and unloaded onto a transport ship. FIG. 4 illustrates a case where thick plates are loaded and unloaded onto and from a transport ship 400 using a gantry crane 410. FIG. 5 is a diagram showing the plank 420 being lifted by the cargo handling device 411 as viewed from the direction in which the gutter 412 extends (y-axis direction). The xyz coordinates shown in FIGS. 4 and 5 indicate the orientation relationships in FIGS. 4 and 5.
The gantry crane 410 includes a cargo handling device (such as a trolley) 411, a girder (traversing girder) 412, and a leg structure 413. axial direction). FIG. 4 shows how the plank 420 is lifted and moved by the cargo handling device 411. FIG. 5 shows how the thick plate 420 is lifted by the cargo handling device 411 when viewed from the direction in which the gutter 412 extends (the y-axis direction).

Note that FIGS. 4 and 5 illustrate how one thick plate 420 is lifted by the cargo handling device 411. However, the manner in which thick plates are conveyed by the cargo handling device 411 is not limited to such a manner. FIGS. 6A to 6E are diagrams showing first to fifth examples of conveyance modes for thick plates. 6A to 6E, the cargo handling device 411 is shown in a more simplified manner.

FIG. 6A illustrates a state in which two thick plates are arranged side by side in the horizontal plane (xy plane) and are simultaneously conveyed by the cargo handling device 411. FIG. 6B illustrates a state in which one thick plate is being transported on the top side (the negative direction side of the x-axis) of the cargo handling device 411. FIG. 6C illustrates a state in which one thick plate is being transported on the bottom side (positive direction side of the x-axis) of the cargo handling device 411. FIG. 6D illustrates a state in which one thick plate is being transported at the center of the cargo handling device 411. FIG. 6E illustrates a state in which a plurality of (two) thick plates are stacked and transported simultaneously by the cargo handling device 411. In this manner, in this embodiment, a plurality of thick plates may be stacked and transported at the same time.

As described above, the manner in which thick plates are transported by the cargo handling device 411 is not limited. Further, the method for lifting the plank by the cargo handling device 411 may be a method using magnetic attraction of a magnet, a method using a hacker, or any other method.
Since the gantry crane 410 itself can be explained using a known technique, a detailed explanation will be omitted here.

The thick plates manufactured at the steel mill are transported to the quay and placed at a predetermined location where they can be transported by the gantry crane 410. As shown in FIG. 4, a plurality of planks may be placed in a predetermined location in a stacked state. In some cases, the planks are placed one by one in a predetermined place. Moreover, in FIG. 4 etc., the case where the thickness (plate thickness) of several sheets of thick plates is the same is illustrated for convenience of description. However, the thicknesses of the plurality of planks may differ. That is, at least one of the thickness, width, and length of the planks placed in a given location may differ.

In FIG. 4, a case is illustrated in which a plurality of thick plates are placed one on top of the other so that the plate surface of the thick plate is parallel to the surface on which the thick plate is placed. However, the method of stacking the planks is not limited to this method. For example, a plurality of planks may be placed one on top of the other so that the surface of the plank is perpendicular or nearly perpendicular to the surface on which the plank is placed. This also applies to locations other than the predetermined locations on the quay.

In this embodiment, first imaging devices 430a to 430f are attached to the garter 412 and the leg structure 413. The number and position of the first imaging devices 430a to 430f are set so that as many display areas of the plank as possible are imaged as the display area 201 of the plank stencil and the display area of the label 202 of the plank. Further, first imaging devices 430g to 430m are also attached to the cargo handling device 411. The number and position of the first imaging devices 430a to 430m are at least one of the first imaging devices 430a to 430m, and the number and position of the first imaging devices 430a to 430m are determined by the stencil display area 201 and label 202 of the plank 420 being transported by the gantry crane 410. At least one of the display areas is set to be imaged as much as possible.

The first imaging devices 430a to 430m are capable of adjusting imaging conditions (imaging direction, angle of view, zoom magnification, aperture (F number), etc.). The adjustment of the imaging conditions may be performed by an operator (worker) operating the first imaging device, or may be performed automatically based on instructions from an external device. Further, for example, among the first imaging devices 430g to 430m attached to the gutter 412, the first imaging devices 430i to 430m attached to a position close to the plank 420 being transported may be a fiberscope camera. . If a fiberscope camera is used, for example, it is possible to take close-up pictures of the display area 201 formed by the stencil and the display area formed by the label 202 of the thick plate 420 being transported.

Note that the captured image captured by the first imaging device is not limited to a still image, and may be a moving image. When the captured image captured by the first imaging device is a moving image, the image extracted from the moving image captured by the first imaging device (i.e., the image of the frame of the moving image) is the article management device described below. Used in 1200. In this case, it is preferable to extract a still image in which the subject imaged by the first imaging device is in focus as much as possible from the moving image imaged by the first imaging device. The fact that the captured image captured by the imaging device is not limited to a still image in this way also applies to the second to seventh imaging devices described later.

The thick plate transported from the steelworks is placed at a predetermined location on the gantry crane 410. An area of the thick plate placed at a predetermined location of the gantry crane 410, including the stencil display area 201, is imaged by the first imaging devices 430c to 430f. The operator (worker) operates the information processing terminal in the operation room of the gantry crane 410 or the information processing terminal (tablet terminal, etc.) held by the operator (worker) of the gantry crane 410 to operate the first imaging device. A preview image display instruction is given to 430c to 430f. The first imaging devices 430c to 430f transmit the preview image to the information processing terminal based on the preview image display instruction. Then, a preview image is displayed by the information processing terminal. An operator (worker) operates the information processing terminal to issue an imaging instruction to a first imaging device that is imaging a preview image in which a display area of a stencil to be photographed is displayed. Upon receiving the imaging instruction, the first imaging devices 430c to 430f capture an image of an area including the stencil display area 201. Note that the preview image is a live image displayed for confirmation by the operator (worker), and is displayed based on a preview image display instruction. It is assumed that the captured image is an image that is captured based on an imaging instruction while a preview image is being displayed. Further, although a case where a preview image is displayed is illustrated here, it is not always necessary to display the preview image.

When a plurality of planks are stacked and transported at the same time, it is not possible to image the area including the stencil display area 201 of the plank below. In this case, instead of the captured image of the area including the display area 201 by the stencil, a captured image of the area including the display area by the label 202 is captured.

Depending on the position where the plank is placed, the stencil display area 201 and the label 202 display area may not be included in the imaging range of the first imaging devices 430c to 430f. In this case, an operator (worker) operates an unmanned aerial vehicle (drone) 440a to 440b equipped with a first imaging device to display an area of a desired plank including the display area 201 by the stencil (or the display area by the label 202). The first imaging device may be used to capture an image of the region (including the region). Alternatively, an image of the area including the stencil display area 201 (or the label 202 display area) of a desired thick plate may be captured using the first imaging device held by the operator (worker). good. Note that the first imaging device held by the operator (worker) may be, for example, an information terminal device (tablet terminal, smartphone, etc.) including the first imaging device.

Note that the display of preview images and the capturing of captured images by the first imaging devices 430c to 430f described above are performed similarly in the first imaging devices 430a to 430b and 430g to 430w other than the first imaging devices 430c to 430f. executed. Further, when a plurality of planks are stacked and conveyed at the same time, capturing an image of an area including the display area of the label 202 is performed by a first image capturing device other than the first image capturing devices 430c to 430f. 430a to 430b and 430g to 430w are similarly executed. Furthermore, when the display area 201 by the stencil and the display area by the label 202 are not included in the imaging range of the first imaging devices 430c to 430f, the imaging method is as follows: The same applies to the imaging devices 430a to 430b and 430g to 430w. Therefore, detailed description of the first imaging devices 430a to 430b and 430g to 430w other than the first imaging devices 430c to 430f will be omitted below.

After the planks are placed at a predetermined location on the gantry crane 410, they are loaded onto the transport ship 400 by the gantry crane 410. A plurality of planks may be stacked and loaded onto the transport ship 400 at the same time. In some cases, the planks are loaded onto the transport ship 400 one by one. Furthermore, at least one of the thickness, width, and length of the planks loaded onto the transport ship 400 may be different.

A captured image of the area including the stencil display area 201 of the thick plate being transported by the gantry crane 410 is captured by first imaging devices 430g to 430m attached to the cargo handling device 411.

When the plank is loaded onto the transport ship 400, a captured image of the area of the plank including the stencil display area 201 is captured by the first imaging devices 430a to 430b attached to the gutter 412 or the transport ship 400. The image is captured by a first imaging device 430n attached to the. The first imaging device 430n attached to the transport ship 400 also adjusts the imaging conditions (imaging direction, angle of view, zoom magnification, aperture (F number), etc.) in the same way as the first imaging devices 430a to 430m. It is now possible to do so. Note that in FIG. 4, for convenience of notation, the number of first imaging devices attached to the transport ship 400 is one. However, a plurality of first imaging devices are attached to the transport ship 400 so that the area where the plank is placed is included in the imaging range.

The transport ship 400 loaded with thick plates as described above docks at a quay near the factory. As described above, the planks are unloaded from the transport ship 400. As described above, in this embodiment, the gantry crane 410 is used to unload the thick plates. Images of the thick plates being transported by the gantry crane 410 are also captured in the same manner as when loading the plates onto the transport ship 400. Further, when the plate is unloaded by the Gundry crane 410 and placed at a predetermined location of the Gundry crane 410, the plate is also treated in the same manner as when it is transported from a steelworks and placed at a predetermined location of the Gantry crane 410. A captured image is captured.

<<Transportation by transport vehicle>>
The plank placed at a predetermined location on Gundry crane 410 is loaded onto a transport vehicle. FIG. 7 is a diagram showing an example of how planks are loaded and unloaded onto a transport vehicle. The planks placed at a predetermined location of Gundry crane 410 are loaded onto transport vehicle 700 using a crane (not shown) or the like. As shown in FIG. 7, a plurality of planks may be stacked and loaded onto a transport vehicle 700 at the same time. However, the planks may be loaded onto the transport vehicle 700 one by one. Furthermore, at least one of the thickness, width, and length of the planks loaded onto the transport vehicle 700 may be different. Alternatively, the planks may be transported and loaded onto the transport vehicle 700 as shown in FIGS. 6A to 6E.

FIG. 8 is a diagram showing an example of a state in which thick plates are loaded onto the transport vehicle 700. First imaging devices 430o to 430p are attached to a structure installed at a standby position of transport vehicle 700. The number and position of the first imaging devices 430o to 430p are such that as many display areas of the thick plates as possible can be imaged as the display area 201 of the thick plates loaded on the transportation vehicle 700 by the stencils and the display area of the labels 202. is set so that Furthermore, a first imaging device 430q is also attached to the transport vehicle 700. The first imaging devices 430o to 430q can also adjust the imaging conditions (imaging direction, angle of view, zoom magnification, aperture (F number), etc.) in the same way as the first imaging devices 430a to 430m. ing.

An area of the thick plate loaded on the transport vehicle 700, including the stencil display area 201, is imaged by the first imaging devices 430o to 430q.

<<Transportation within the factory>>
When the transport vehicle 700 loaded with thick plates as described above arrives at the factory, the thick plates are unloaded from the transport vehicle 700. The thick plates loaded onto the transport vehicle 700 are unloaded from the transport vehicle 700 using a crane or the like (not shown). As shown in FIG. 7, a plurality of planks may be stacked and unloaded from a transportation vehicle 700 at the same time. However, the planks may be unloaded from the transportation vehicle 700 one by one. Furthermore, at least one of the thickness, width, and length of the planks unloaded from transport vehicle 700 may be different. Alternatively, the plank may be transported and unloaded from the transportation vehicle 700 as shown in FIGS. 6A to 6E. Note that there is a transport device (gantry crane 410) used to transport the thick plate from the transport ship 400 to a predetermined storage location of the gantry crane 410, and a transport device used to transport the thick plate from the transport vehicle 700 to the storage location in the factory. (a crane, etc. not shown) is a different conveyance device. In this way, a plate transported to a certain storage site by a certain transport device is transported to another storage site, which cannot be transported by the certain transport device, by a transport device different from the certain transport device. There may be cases where The same is true when transporting thick plates between other storage areas, such as between storage areas within a factory.

FIGS. 9A to 9C are diagrams showing first to third examples of the appearance of thick plates placed in a storage area in a factory. First imaging devices 430r to 430w are attached to a structure installed in a storage area in a factory. As shown in FIGS. 9A-9C, planks are placed in various locations in a factory. Therefore, the number and position of the first imaging devices 430r to 430w are determined so that as many display areas of the thick plates as possible are imaged as the display area 201 of the stencils and the display area of the labels 202 of the thick plates placed in the storage area. is set to The first imaging devices 430r to 430w can also adjust the imaging conditions (imaging direction, angle of view, zoom magnification, aperture (F number), etc.) in the same way as the first imaging devices 430a to 430m. ing.

The first imaging devices 430r to 430w image an area of a thick plate placed in a warehouse in a factory, including the stencil display area 201.

When the plank is placed in the receiving area, the area of the plank including the stencil display area 201 (or the label 202 display area) is imaged by the first imaging devices 430r to 430w as described above. Ru.

The plank is then transported to the production facility. FIG. 10 is a diagram showing an example of how a thick plate is processed in a production facility. In FIG. 10, production equipment 1000 is, for example, equipment for heat treatment and equipment for gas cutting. In FIG. 10, a case where thick plates 1002a and 1002b are transported using a transport roll 1001 is illustrated. Note that conveyance of the thick plate between the storage area and the conveyance roll 1001 is realized by using a conveyance device (not shown) or the like.

First imaging devices 430x to 430y are attached to a structure installed on a production line of a factory. The number and position of the first imaging devices 430x to 430y are set so that the display area 201 formed by the stencil of the thick plates 1002a, 1002b is imaged. The first imaging devices 430x to 430y can also adjust the imaging conditions (imaging direction, angle of view, zoom magnification, aperture (F number), etc.) in the same way as the first imaging devices 430a to 430m. ing. Note that the thick plates 1002a and 1002b transported on the transport roll 1001 are not overlapped. Moreover, the conveyance path of the thick plates 1002a and 1002b on the conveyance roll 1001 is assumed in advance. Therefore, in this embodiment, it is assumed that the first imaging devices 430x to 430y do not fail to capture images of the display areas of the labels 202 of the thick plates 1002a and 1002b. However, the display area of the labels 202 of the planks 1002a and 1002b may be imaged using an unmanned aerial vehicle equipped with a first imaging device. In addition, in FIG. 10, the broken line shows the conveyance route of the thick plates 1002a and 1002b on the conveyance roll 1001, and does not actually exist.

A thick plate 1002a transported from a storage area in a factory is placed on a transport roll 1001 and transported by the transport roll 1001. A region of the thick plate 1002a being transported on the transport rolls 1001 before entering the production facility 1000, including the stencil display region 201, is imaged by the first imaging device 430x. Further, after the processing in the production equipment 1000 is completed and the production equipment 1000 is exited, an area of the thick plate 1002b being transported on the transport rolls 1001, including the display area 201 by the stencil, is imaged by the first imaging device 430y. be done.

Thereafter, the planks are placed in a warehouse as illustrated in FIGS. 9A to 9C. As described above, the area including the stencil display area 201 (or the label 202 display area) of a thick plate placed in a storage area in a factory is imaged by the first imaging devices 430r to 430w. During shipping, planks are moved from a warehouse to a shipping area. An area including the stencil display area 201 (or the label 202 display area) of the thick plate placed in the shipping area is imaged by the first imaging devices 430r to 430w.

In each storage area within the factory, multiple planks may be stacked on top of each other. In some cases, one plate is placed in each storage area within the factory. Furthermore, at least one of the thickness, width, and length of the planks placed in each storage area within the factory may be different.

FIG. 11 is a diagram showing an example of the original thick plate and the thick plate after cutting. FIG. 11 illustrates a case where thick plate 1100 is cut to produce thick plates 1110, 1120, 1130, and 1140. After the planks 1110, 1120, 1130, 1140 are manufactured in this manner, markings are applied to each plank 1110, 1120, 1130, 1140. In FIG. 11, display areas 1111, 1121, 1131, 1141 for planks 1110, 1120, 1130, 1140 are shown. Information including the IDs of the planks 1110, 1120, 1130, and 1140 is displayed in the display areas 1111, 1121, 1131, and 1141. In the display areas 1111, 1121, 1131, and 1141, display is performed using, for example, a stencil. In addition to the display using a stencil, at least one of a stamp and a label may be used. Furthermore, the display in the display areas 1111, 1121, 1131, and 1141 may be handwritten.

Furthermore, the ID of the original thick plate 1100 may be displayed in the display areas 1111, 1121, 1131, and 1141. In the display areas 1111, 1121, 1131, and 1141, display is performed using, for example, a stencil. However, the display in the display areas 1111, 1121, 1131, and 1141 may be handwritten. This also applies to the display on the display area 201 and the display on the label 202. In the following, it is assumed that display areas 1111, 1121, 1131, and 1141 are display areas using stencils.

When the thick plates 1110, 1120, 1130, and 1140 are manufactured, for example, the display areas 1111, 1121, 1131, and 1141 of the desired thick plates are displayed using a stencil using a first imaging device held by an operator (worker). A captured image of the area including the area may be captured. Note that the first imaging device held by the operator (worker) may be, for example, an information terminal device (tablet terminal, smartphone, etc.) including the first imaging device. Further, the operator (worker) operates the unmanned aerial vehicle equipped with the first imaging device to capture the captured image of the region of the desired thick plate including the stencil display regions 1111, 1121, 1131, and 1141. The image may be captured by one imaging device.

As described above, in this embodiment, at least when placed on the transport ship 400, when unloaded from the transport ship 400 and loaded onto the transport vehicle 700, and at each storage area of the gantry crane 410 and the factory. A case will be exemplified in which a captured image of the area including the display areas 201, 1111, 1121, 1131, and 1141 by the stencils and the area including the display area by the label 202 of each thick plate is obtained.

In this embodiment, the position of the plank is automatically specified by the article management device using the captured image captured as described above. Note that when the thick plates are transported one by one, the display area 201 by the stencil can be imaged, so the captured image of the area including the display area by the label 202 becomes unnecessary. In the following description, when stencil display and label 202 display are collectively referred to, they will be referred to as stencil (label) display as necessary. Furthermore, when the stencil display areas 201, 1111, 1121, 1131, and 1141 and the label 202 display area are collectively referred to as a stencil (label) display area, if necessary.

<Articles management device>
FIG. 12 is a diagram showing an example of the functional configuration of the article management device 1200. The hardware of the article management device 1200 is realized by using, for example, an information processing device including a CPU, ROM, RAM, HDD, and various interfaces, or dedicated hardware. FIG. 13 is a diagram showing an example of a management table 1300 managed by the article management device 1200. Note that, as described above, in this embodiment, tracking is performed by the article management device 1200. Therefore, in this embodiment, the article management device 1200 becomes a tracking device.

As shown in FIG. 12, in this embodiment, the article management device 1200 includes a first image acquisition unit 1201, a recognition unit 1202, an acquisition unit 1203, a collation unit 1204, an identification unit 1205, and a position derivation unit 1206. , a storage unit 1207 , an output unit 1208 , and a second image acquisition unit 1209 .

The first image acquisition unit 1201 acquires a first captured image of an area including a display area by a stencil (label). The recognition unit 1202 recognizes components, positions, etc. of display items in a stencil (label) display from the first captured image acquired by the first image acquisition unit 1201. The acquisition unit 1203 acquires information stored in the ID column of the actual item display column in the management table 1300, which will be described later. The collation unit 1204 collates the information recognized by the recognition unit 1202 and the information obtained by the acquisition unit 1203. The identification unit 1205 identifies one plank from among the planks managed in the management table 1300 based on the result of the verification performed by the verification unit 1204 . In this embodiment, an identification information acquisition unit 1210 that acquires identification information of a thick plate includes a recognition unit 1202, an acquisition unit 1203, a collation unit 1204, and a specification unit 1205. The position deriving unit 1206 derives the xyz coordinates (three-dimensional coordinates) of the display area by the stencil (label) as the three-dimensional position of the thick plate specified by the specifying unit 1205. The storage unit 1207 stores information indicating the three-dimensional position of the thick plate in the management table 1300. The output unit 1208 outputs information indicating the three-dimensional position of the thick plate. The second image acquisition unit 1209 acquires a second captured image for specifying the position in the height direction (z-axis direction) of the display area by the stencil (label) in the position derivation unit 1206. A detailed example of the article management device 1200 will be described later with reference to the flowchart in FIG. 17.

In FIG. 13, a management table 1300 stores information used by the article management device 1200 to manage the position of the plank. The information stored in the management table 1300 includes information managed by the thick plate manufacturer. However, some of the information stored in the management table 1300 may be information that can also be used in distribution and the like. Here, the information managed by the thick plate manufacturer is information set or acquired by the thick plate manufacturer. The information managed by the thick plate manufacturer includes information that allows an operator who is an employee of the thick plate manufacturer to understand the meaning of the information. More specifically, the information managed by the thick plate manufacturer is, for example, information created by the thick plate manufacturer and stored in the management table 1300. In this embodiment, the information stored in the management table 1300 is an example of article management data, and by storing each piece of information in the management table 1300, an example of the data structure of article management data is realized. .

In FIG. 13, information in the same column becomes identification information for the same plank (columns under the same No. become columns in which identification information for the same plank is stored, respectively).
In FIG. 13, information regarding each display item of the stencil (label) display is stored in the actual item display column. The non-actual item display column stores information other than information regarding each display item displayed by a stencil (label). In the present embodiment, the information stored in the item display column is an example of display information included in article management data, and the information stored in the non-item item display column is hidden information included in article management data. This is an example.

An example of information stored in the actual item display column of the management table 1300 will be explained.
In the stencil column of the actual product display column, information regarding the stencil is stored.
In the ID column of the stencil column, information indicating the content and position of the component of ID 305 included in the stencil display is stored. Note that the term "component element" refers to each piece of information that constitutes a display item, such as letters, numbers, symbols, and marks.
In the customer name column of the stencil column, information indicating the content and position of the constituent elements of the customer name 302 included in the stencil display is stored.

The size column of the stencil column stores information indicating the content and position of the component of size 304 included in the stencil display.
The standard column of the stencil column stores information indicating the contents and positions of the constituent elements of the standard 303 included in the stencil display.
The customer code column in the stencil column stores information indicating the content and position of the component of the customer code 306 included in the stencil display.

The order number column in the stencil column stores information indicating the content and position of the component of order number 307 included in the stencil display.
In addition to the display items in the stencil display described above, the contents of display items that may be displayed as display items in the stencil display are also stored in the management table 1300. The "..." below the additional information column in the stencil column in FIG. 13 indicates this. Furthermore, display items themselves may be added and/or deleted.

In the column of actual item display, information regarding the label is stored in the column of label.
In the ID column of the label column, information indicating the content and position of the component of ID 312 included in the label display is stored.
In the size column of the label column, information indicating the content and position of the component of size 313 included in the label display is stored.

The standard column of the label column stores information indicating the contents and positions of the constituent elements of the standard 311 included in the label display.
In the delivery date column of the label column, information indicating the content and position of the component of the delivery date 314 included in the label display is stored.
The customer code column in the label column stores information indicating the content and position of the component of the customer code 315 included in the label display.

In addition to the display items in the label display described above, the contents of display items that may be displayed as display items in the label display are also stored in the management table 1300. "..." below the customer code column in the label column of FIG. 13 indicates this. Furthermore, display items themselves may be added and/or deleted.

Next, an example of information stored in the non-product display column of the management table 1300 will be described.
Information indicating the position of the display area when a predetermined position on the thick plate is set as the origin (reference) is stored in the column of position within the article.
In the stencil column of the intra-article position column, information indicating the position of the stencil display area when a predetermined position on the thick plate is set as the origin (reference) is stored.
In the label column of the intra-article position column, information indicating the position of the label display area when a predetermined position on the thick plate is set as the origin (reference) is stored.

In the article position column, information indicating the three-dimensional position of the plank is stored. The information stored in the item position column is information indicating the three-dimensional position of the plank specified by the item management device 1200. Therefore, the three-dimensional position of the same plank identified by the article management device 1200 may change not only the x-coordinate and y-coordinate but also the z-coordinate as the plank moves. Therefore, as the plank moves, the three-dimensional position of the information display position of the same plank identified by the article management device 1200 changes not only in the x and y coordinates (position in the horizontal direction) but also in the z coordinate (height). direction position) may also change.

The registered time column stores information indicating the date and time when the plank was placed at the current position (timing determined from the date and time).

Information indicating the ID of the original plank is stored in the parent ID column. Planks may be cut in a factory. Original plank refers to the plank before cutting. Therefore, if information indicating the ID of the original plank is stored in the parent ID column, the information stored in the column in the same column as the parent ID column is the information for the plank after cutting. become. When thick plates 1110, 1120, 1130, and 1140 are manufactured as shown in FIG. , 1131, and 1141 are stored. Furthermore, information indicating the ID of the thick plate 1100 is stored in the parent ID column for the thick plates 1110, 1120, 1130, and 1140. As described above, in this embodiment, the information on the thick plate after cutting is exemplified as being managed as a new column (No) in the management table 1300 shown in FIG. 13. In the management table 1300, information regarding the thick plate after cutting is stored in the actual item display and non-actual item display columns.

Here, for example, the plank 1110 may be cut to produce a plurality of additional planks. In this case, information indicating the ID of the thick plate 1110 is stored in the parent ID column for the other plurality of thick plates. In the parent ID column for the thick plate 1110, information indicating the ID of the thick plate 1100 is stored. In this way, by tracing the information stored in the parent ID column, the changes in each plank can be identified. The sequence of planks obtained by tracing the information stored in the parent ID column for a certain plank in this manner will be referred to as the cut plate series of the plank. For example, the cut plate series of plank 1110 becomes planks 1100 and 1110. In addition, in the parent ID column of each plank, for example, in order to know the cutting board series, not only information indicating the ID of the most recent plank but also the IDs of all planks belonging to that cutting series are shown. Information may also be stored. In this case, any ID defined by the manufacturer is used.

The ID of the thick plate carrier is stored in the column of carrier ID. In this embodiment, a case is illustrated in which the carriers are a transport ship 400 and a transport vehicle 700.
When the planks are loaded onto the transport ship 400, the article management device 1200 may store information indicating the ID of the transport ship 400, for example, as follows.
First, the article management device 1200 acquires the ID of the plank and the ID (so-called ship number) of the transport ship 400. Then, the article management device 1200 identifies, from the ID column of the management table 1300, a column in which information indicating the same ID as the obtained ID of the thick plate is stored. Then, the article management device 1200 stores information indicating the acquired ID of the transport ship 400 in the vehicle ID column of the specified column.

For example, the first imaging devices 430a, 430b, and 430n capture an image including a stencil display area 201 or a label 202 display area, and an area where information indicating the ID of the transport ship 400 is displayed. Good too. In this case, the article management device 1200 may recognize and acquire the ID of the plank and the ID of the transport ship 400 from the captured image.

Furthermore, the first imaging devices 430a, 430b, and 430n capture a captured image including a display area 201 using a stencil or a display area using a label 202, and a captured image including an area where information indicating the ID of the transport ship 400 is displayed. may be imaged. For example, a captured image including an area where information indicating the ID of the transport ship 400 is displayed may be captured next to the captured image including the display area 201 by the stencil or the display area by the label 202. In this case, the article management device 1200 includes an ID recognized from a captured image including a stencil display area 201 or a label 202 display area, and an area where information indicating the ID of the transport ship 400 is displayed. The ID recognized from the captured image may be acquired as the ID of the plank and the ID of the transport ship 400.

Furthermore, for example, the operator (worker) of the transport ship 400 may input the ID of the plank and the ID of the transport ship 400 into a handheld information processing terminal (tablet terminal, etc.). In this case, by transmitting the ID of the plank and the ID of the transport ship 400 from the information processing terminal to the goods management device 1200, the goods management device 1200 can send the ID of the plank and the ID of the transport ship 400. You may also obtain the ID of

Further, when the planks are loaded onto the transportation vehicle 700, the article management device 1200 may store information indicating the ID of the transportation vehicle 700, for example, as follows.
First, the article management device 1200 acquires the ID of the plank and the ID of the transportation vehicle 700 (for example, license plate information). Then, the article management device 1200 identifies, from the ID column of the management table 1300, a column in which information indicating the same ID as the obtained ID of the thick plate is stored. Then, the article management device 1200 stores information indicating the acquired ID of the transport vehicle 700 in the column of the vehicle ID of the specified column.

For example, the first imaging devices 430o to 430q may capture an image including a stencil display area 201 or a label 202 display area, and an area where information indicating the ID of the transportation vehicle 700 is displayed. . In this case, the article management device 1200 may recognize and acquire the ID of the thick plate and the ID of the transportation vehicle 700 from the captured image.

Furthermore, the first imaging devices 430o to 430q capture a captured image including the stencil display area 201 or the label 202 display area, and a captured image including the area where information indicating the ID of the transportation vehicle 700 is displayed. You may. For example, a captured image including an area where information indicating the ID of the transport vehicle 700 is displayed may be captured next to the captured image including the display area 201 by the stencil or the display area by the label 202. In this case, the article management device 1200 includes an ID recognized from a captured image including a stencil display area 201 or a label 202 display area, and an area where information indicating the ID of the transportation vehicle 700 is displayed. The ID recognized from the captured image may be acquired as the ID of the thick plate and the ID of the transport vehicle 700.

Furthermore, for example, the operator (worker) of the transportation vehicle 700 may input the ID of the plank and the ID of the transportation vehicle 700 into a handheld information processing terminal (tablet terminal, etc.). In this case, by transmitting the ID of the thick plate and the ID of the transportation vehicle 700 from the information processing terminal to the article management device 1200, the article management device 1200 can send the ID of the plate and the ID of the transportation vehicle 700. You may also obtain the ID of

The data structure of the management table 1300 may be such that the information stored in each column of the management table 1300 (for example, article position and registration time) is updated every time a plank is placed in a new position. good. Further, the data structure of the management table 1300 may be a data structure to which information is added. In the latter case, there is an advantage that the history of the position of the article can be tracked.

Next, an example of the contents stored in the actual item display column of the management table 1300 and the content stored in the intra-article position column of the non-actual item display column will be explained in more detail.
FIG. 14 is a diagram showing an example of the position of the stencil in the article and the position of the constituent elements of each display item in the display using the stencil. FIG. 14 illustrates a case in which stencil display is performed with the display position shown in FIG. 2 and the content shown in FIG. 3A.
In FIG. 14, among the eight vertices of a rectangular parallelepiped thick plate, the vertex 203 shown in FIG. The position of the stencil in the article and the position of the constituent elements of each display item by the stencil are expressed in xy coordinates with the direction along the horizontal direction being the positive direction of the x-axis.

In this embodiment, the circumference determined by the position of the maximum value of the x-axis, the minimum value of the x-axis, the maximum value of the y-axis, and the minimum value of the y-axis among the positions of the constituent elements of each display item in the stencil display. Among the four vertices of the rectangle, the xy coordinates of two vertices on the diagonal are defined as the position of the stencil in the article. In FIG. 14, the xy coordinates (Ls10, Ws10) and (Ls11, Ws11) of the two vertices 1401a and 1401b of the circumscribed rectangle indicate the position of the stencil within the article. In order to simplify the explanation, it is assumed here that the circumscribed rectangle is a rectangle parallel to the coordinate axes (x-axis and y-axis in the example shown in FIG. 14) expressing the position within the article.

In addition, the xy coordinates of the center of gravity (or centroid) of each component (letter, number, symbol, mark) of each display item in the stencil display is taken as the position of the component. In FIG. 14, the y-axis coordinates of 5, 6, 3, 4, 9, 7, -, 0, and 1, which are the components of ID 305 included in the stencil display, are Ws1, Ws2, Ws3, Ws4, and Ws5, respectively. , Ws6, Ws7, Ws8, and Ws9, indicating that the x-axis coordinates are all Ls1.
As described above, the position of each component of each display item displayed by the stencil is expressed by xy coordinates.

FIG. 15 is a diagram showing an example of the position of the label in the article and the position of the constituent elements of each display item in the display by the label. FIG. 15 illustrates a case in which labels are displayed in the display position shown in FIG. 2 and the display content shown in FIG. 3B.
In FIG. 15, among the eight vertices of a rectangular parallelepiped thick plate, the vertex 204 shown in FIG. The position of the label within the article and the position of the constituent elements of each display item on the label are expressed in yz coordinates, with the direction along the thickness direction being the positive direction of the z-axis.

In this embodiment, it is assumed that the label is rectangular. Among the four vertices of the label, the yz coordinates of two diagonal vertices are taken as the position of the label within the article. Furthermore, the yz coordinates of the position of the center of gravity (or centroid) of the component of each display item in the label display is taken as the position of the component. In FIG. 15, the yz coordinates of the two vertices 1501a and 1501b of the label indicate the position of the label within the article.

As described above, in the management table 1300 shown in FIG. 13, the ID column of the stencil column of the actual product display column stores information indicating the content and position of the component of ID 305 included in the stencil display. . Therefore, as shown in FIG. 16A, in the example shown in FIG. 14, the ID column of the stencil column of the actual item display column contains "5", " 6'', ``3'', ``4'', ``9'', ``7'', ``-'', ``0'', and ``1'' are stored. In addition, in the ID column of the stencil column of the actual item display column, the positions of the relevant components are (Ls1, Ws1), (Ls2, Ws1), (Ls3, Ws1), (Ls4, Ws1), (Ls5 , Ws1), (Ls6, Ws1), (Ls7, Ws1), (Ls8, Ws1), and (Ls9, Ws1) are stored.

Similarly to the ID column of the stencil column, information indicating the content and position of the component of the display item is stored in the other display item column of the actual item display column. Note that the information indicating the position of the component of each display item in the label column is not the xy coordinate but the yz coordinate.

In addition, as shown in FIG. 16B, in the example shown in FIG. 14, the stencil column in the column for position within the article in the column for displaying non-actual items includes (Ls10, Ws10), as information indicating the position of the stencil within the article. (Ls11, Ws11) is stored.
Similarly to the stencil column of the intra-article position column, information indicating the position of the label within the article is also stored in the label column of the intra-article position column. Note that, as described above, the information indicating the position of the label within the article is not the xy coordinate but the yz coordinate.

When creating the management table 1300, for example, the operator may operate the user interface of the article management device 1200 to input information to be stored in each column of the management table 1300 into the article management device 1200. Additionally, the article management device 1200 may receive information to be stored in each column of the management table 1300 from an external device such as a computer that manages factory operations.

The information stored in the item display column of the management table 1300 and the information stored in the item position and parent ID column of the non-item item display column may be determined after the display on the thick plate is performed. For example, in some cases, the requirements are determined before the marking is applied to the plank. Therefore, if the information to be stored in the column is determined before displaying on the plank, the article management device 1200, for example, acquires the information to be stored in the column as described above. and stored in each column of the management table 1300.

On the other hand, the information stored in the item position and registration time columns of the non-actual product display column may not be determined before the display on the plank is performed. Therefore, the article management device 1200 acquires the information to be stored in the relevant column and stores it in each column of the management table 1300, for example, at the timing when the information to be stored in the relevant column is obtained. As described above, the registration time column stores the date and time when the information stored in the article position column was changed. In this case, the date and time when the plank was placed at the current position is registered in the registration time column.

Regarding items in the non-actual item display column, information is stored in the column for the item only if the information for the item has been acquired, and if the information for the item has not been acquired, the information for the item is stored in the column for the item. No information is stored in the field. Furthermore, if the information stored in the article position column is not updated even after a predetermined period of time has elapsed from the date and time stored in the registration time column, the article management device 1200 determines that the identification of the position of the plank has been interrupted. It is preferable to make a judgment and reset the information stored in the columns of the article position and registration time. Furthermore, at the same position, a plank with an earlier date and time stored in the registered time column will not be transported before a plank with a later date and time stored in the registered time column (i.e., If there are overlapping boards, the board below will not suddenly disappear). Therefore, when the article management device 1200 resets the information stored in the article position and registration time columns, it is preferable to do the following. In other words, the article management device 1200 determines whether a plank whose date and time is later than the date and time written in the column of registered time to be reset is stored in the column of registered time, and which is stored in the column of article position to be reset. If there is a plank in the same three-dimensional position as the current position, it is desirable to issue a sign to confirm it with an alarm or the like.

Next, an example of the functions of the article management device 1200 will be explained. FIG. 17 is a flowchart illustrating an example of processing of the article management device 1200 of this embodiment. An example of the functions of the article management device 1200 will be described with reference to FIG. 17. Note that, as described above, in this embodiment, tracking is performed by the article management device 1200. Therefore, in this embodiment, the article management method implemented using the flowchart illustrated in FIG. 17 is a tracking method.

In step S1701 of FIG. 17, the first image acquisition unit 1201 acquires the first captured image. The first captured image is captured by any one of the first imaging devices 430a to 430y, the first imaging devices included in the unmanned aerial vehicles 440a to 440c, and the first imaging device held by the operator (worker). This is a captured image captured by the first imaging device. The first captured image may be directly transmitted from the first imaging device to the article management device 1200, or may be transmitted from the first imaging device to the article management device 1200 via an information terminal device. The process according to the flowchart in FIG. 17 is performed on any one of the first imaging devices 430a to 430y, the first imaging devices included in the unmanned aerial vehicles 440a to 440c, and the first imaging device held by the operator (worker). This process is executed every time the article management device 1200 receives a captured image captured by one imaging device. In this embodiment, it is assumed that the ID of the first imaging device that captured the first captured image and the imaging conditions under which the first captured image was captured are transmitted together with the first captured image. .

The first image acquisition unit 1201 includes the first captured image, the ID of the first imaging device that captured the first captured image, the imaging conditions under which the first captured image was captured, and the The imaging date and time of the first captured image are acquired. Among the information set by the imaging device during image capturing, the imaging conditions include information necessary for specifying the position of the stencil display area or the label display area, as will be described later.

Further, if the position of the first imaging device when the first captured image is captured is not fixed, the position information of the first imaging device when the first captured image is captured is Sent with the image. For example, if the unmanned aerial vehicles 440a to 440c can position the three-dimensional coordinates (xyz coordinates) of the unmanned aerial vehicle using GPS (Global Positioning System), the first The three-dimensional coordinates may be transmitted as the position information of the imaging device. Regarding the first imaging device 430n attached to the transport ship 400, the first imaging devices 430g to 430m attached to the cargo handling device 411, and the first imaging device 430q attached to the transportation vehicle 700, Similarly to the unmanned aerial vehicles 440a to 440c, the position information of the first imaging device may be transmitted together with the first captured image. This also applies to the first imaging device held by the operator (worker).
Note that if the first imaging device is unable to capture an area that should be included in the first captured image, the imaging conditions of the first imaging device (imaging direction, angle of view, zoom magnification, aperture (F number, etc.) are adjusted.

Next, in step S1702, the recognition unit 1202 recognizes the constituent elements (letters, numbers, symbols, marks) of display items in the stencil (label) display from the first captured image acquired in step S1701. The process of recognizing the contents of the constituent elements of the display items can be realized, for example, by a technique using well-known OCR (Optical Character Recognition), so detailed explanation thereof will be omitted here. Note that the process in step S1702 may be realized using a technology using AI (Artificial Intelligence) such as NN (Neural Network) or a technology that combines AI and OCR (so-called AI OCR).

Furthermore, the recognition unit 1202 identifies whether the component of the display item is displayed by a stencil or displayed by a label. For example, the recognition unit 1202 performs edge detection processing based on the first captured image acquired in step S1701, detects the edge of the thick plate, and identifies the area of the thick plate based on the detected result. Then, the recognition unit 1202 determines whether the identified region is a plate surface portion of the thick plate or a side surface (plate thickness) portion of the thick plate. This determination is realized, for example, based on the aspect ratio of the specified area. As a result of this determination, if the identified area is a plate surface portion of a thick plate, the recognition unit 1202 recognizes that the component of the display item is displayed by a stencil. On the other hand, if the identified area is the side surface (thickness) of a thick plate, the recognition unit 1202 recognizes that the component of the display item is displayed by a label.

Further, the recognition unit 1202 performs edge detection processing based on the first captured image, and if the area of the rectangular label 202 is detected, the recognition unit 1202 determines that the component of the display item is displayed by the label. If this is not the case, the component of the displayed item may be recognized as being displayed using a stencil.
In this embodiment, in order to simplify the explanation, it is assumed that for the thick plate located at the top level, the first captured image includes a display area 201 using a stencil instead of a display area using a label 202. .

Additionally, the recognition unit 1202 derives the position of the display item. An example of a method by which the first recognition unit 1202 derives the position of a display item will be described. First, the recognition unit 1202 performs edge detection processing based on the first captured image acquired in step S1701 to detect edges of the thick plate. Then, the recognition unit 1202 detects the origin of the position within the article (predetermined position on the thick plate) in the first captured image. For example, the recognition unit 1202 identifies the article in the captured image based on a preset positional relationship between the display area of the stencil (label) and the origin of the position within the article among the eight vertices of a rectangular parallelepiped thick plate. Derive the vertex that is the origin of the internal position (predetermined position on the thick plate).

For example, the following information may be preset in the article management device 1200 as the positional relationship between the display area of the stencil (label) and the origin of the position within the article. In other words, when the component of the display item of the stencil (label) display is viewed so that the top of the component is located on the upper side, the stencil (label) display is on the same surface as the stencil (label) display. Information that the vertex located on the lower right side of is the origin of the position within the article may be set in advance in the article management device 1200. The recognition unit 1202 derives the position of the component of the display item in the stencil (label) display based on the origin of the position within the article in the captured image. When the process in step S1702 is completed, the process in step S1703 is executed.

In step S1703, the acquisition unit 1203 acquires information stored in the ID column of the actual item display column in the management table 1300 stored in the storage unit 1207.
Next, in step S1704, the matching unit 1204 matches the information recognized in step S1702 and the information obtained in step S1703. If the information recognized in step S1702 is recognized as stencil display information, the collation unit 1204 selects the ID column of the stencil column of the actual product display column of the information obtained in step S1703. The stored information is compared with the information recognized in step S1702. On the other hand, if the information recognized in step S1702 is recognized as label display information, the collation unit 1204 checks the ID in the label column of the actual product display column among the information obtained in step S1703. The information stored in the column is compared with the information recognized in step S1702.

Next, in step S1705, the identification unit 1205 identifies one plank from the planks managed in the management table 1300, based on the result of the verification in step S1703. Specifically, the identifying unit 1205 identifies information that matches the information recognized in step S1702 from among the information obtained in step S1703.

Here, matching means that the information recognized in step S1702 completely matches the content and arrangement order of the components stored in the stencil column of the actual item display column or the ID column of the label column. It means that there is a part that is In the example shown in FIGS. 3A and 3B, the contents of the components of IDs 305 and 312 displayed by the stencil or label are 5, 6, 3, 4, 9, 7, -, 0, 1, and the arrangement order is , from left to right: 5, 6, 3, 4, 9, 7, -, 0, 1. Therefore, the identifying unit 1205 identifies information in which 5, 6, 3, 4, 9, 7, -, 0, and 1 are arranged in this order from the left from the ID column managed in the management table 1300. . In the information recognized in step S1702, not only when some of these constituent elements are missing, but also when constituent elements other than these constituent elements are lined up, the information recognized in step S1702 is It is assumed that the information does not completely match the content and arrangement order of the components stored in the stencil column of the actual item display column or the ID column of the label column.

Next, in step S1706, the position deriving unit 1206 determines whether the position of the first imaging device that captured the first captured image obtained in step S1701 is known, based on the information obtained in step S1701. Determine whether In this embodiment, the first imaging devices 430a to 430f are attached to the gutter 412 and the leg structure 413, and the first imaging device 430o is attached to the structure installed at the standby position of the transport vehicle 700. -430p, first imaging devices 430r to 430w attached to structures installed in a storage area in the factory, and first imaging devices attached to structures installed on the production line of the factory It is assumed that the three-dimensional positions 430x to 430y are known. For each of these first imaging devices, the ID of the first imaging device and the position information (three-dimensional coordinates (xyz coordinates)) of the first imaging device are correlated with each other. It is assumed that it has been registered in the article management device 1200 in advance. If the position information (three-dimensional coordinates) associated with the same ID as the ID of the first imaging device acquired in step S1701 is registered in the article management device 1200 in advance, the position derivation unit 1206 determines the position information in step S1701. It is determined that the position of the first imaging device that captured the acquired first captured image is known, and if not, the position of the first imaging device that captured the first captured image acquired in step S1701 is determined to be known. Determine that it is not known.

As a result of the determination in step S1706, if the position of the first imaging device that captured the first captured image acquired in step S1701 is not known, the process in step S1707 is executed. In step S1707, the position derivation unit 1206 determines whether the position information of the first imaging device that captured the first captured image has been acquired in step S1701. As a result of this determination, if the position information of the first imaging device that captured the first captured image has not been acquired in step S1701, the process advances to step S1712, which will be described later.

As a result of the determination in step S1706, the position of the first imaging device that captured the first captured image acquired in step S1701 is known, and as a result of the determination in step S1707, the location of the first imaging device that captured the first captured image is known. If the position information of the first imaging device is acquired in step S1701, the process of step S1708 is executed. In step S1708, the position deriving unit 1206 determines whether the position in the height direction (z-axis direction) of the display area by the stencil (label) can be specified. Note that a specific example in which the position in the height direction (z-axis direction) of the display area using a stencil (label) can be specified will be described later. As a result of the determination in step S1708, if the position in the height direction (z-axis direction) of the display area by the stencil (label) cannot be specified, the process proceeds to step S1712, which will be described later.

On the other hand, as a result of the determination in step S1708, if the position in the height direction (z-axis direction) of the display area by the stencil (label) can be specified, the process in step S1709 is executed. In step S1709, the position derivation unit 1206 derives the three-dimensional coordinates (xyz coordinates) of the display area by the stencil (label) as the three-dimensional position of the thick plate specified in step S1705.

Next, in step S1710, the storage unit 1207 converts the information stored in the item position column of the non-in-stock item display column of the management table 1300 as information for the thick plate identified in step S1705 into the information derived in step S1709. The information is updated to indicate the three-dimensional position of the plank. As described above, this embodiment exemplifies a case where the information stored in the article position column of the non-actual item display column of the management table 1300 is the information display position. Further, in this embodiment, the information indicating the three-dimensional position of the thick plate derived in step S1709 is an example of the information display position newly derived by the position deriving means. Furthermore, the information stored in the article position column of the non-in-stock item display column of the management table 1300 before the process of step S1710 is performed is an example of the information display position already stored in the storage medium. The information stored in the item position column of the non-actual item display column of the management table 1300 is the xyz coordinates of the display area by the stencil (label). Therefore, the information display position is a three-dimensional position. As described above, the information display position of the same thick plate changes not only the x-coordinate and y-coordinate but also the z-coordinate (position in the height direction) as the position of the thick plate changes. In this embodiment, the process of step S1710 makes it possible to track the position of the plank whose existing position changes.

Next, in step S1711, the output unit 1208 outputs information indicating the three-dimensional position of the thick plate identified in step S1705. For example, the output unit 1208 causes the computer display to display information indicating the three-dimensional position of the plank identified in step S1705. Then, the process according to the flowchart of FIG. 17 ends.

Here, an example of a method for deriving the three-dimensional position of the stencil (label) display area in step S1709 will be described using the stencil display area 201 shown in FIG. 2 as an example. In the example shown in FIG. 2, the stencil display area 201 is a region of the plate surface of the plate near the center in the width direction (y-axis direction) of the plate and in the longitudinal direction of the plate. It is located in a region on one end side (negative direction side of the x-axis). In this embodiment, the three-dimensional position of the display area 201 by the stencil is an example of the information display position.
FIG. 18 is a diagram illustrating an example of a method for deriving the three-dimensional position of the display area 201 using a stencil.

In FIG. 18, point P indicates the three-dimensional position of the first imaging device (the xyz coordinates of point P are (x _p , y _p , z _p )). The three-dimensional position of the first imaging device is, for example, the principal point of the first imaging device. In the example shown in FIG. 18, it is assumed that the z-axis is perpendicular to the surface of the thick plate (the surface on which the display area 201 formed by the stencil is formed). Further, it is assumed that the y-axis is parallel to the direction in which constituent elements of the same display item are lined up in the stencil display area 201. Further, it is assumed that the x-axis is parallel to the direction in which different display items are lined up in the display area 201 using the stencil. Further, it is assumed that the x-axis, y-axis, and z-axis are perpendicular to each other. Furthermore, here, in order to simplify the explanation, it is assumed that the width direction of the thick plate is parallel to the y-axis, and the length direction of the thick plate is parallel to the x-axis.

Point S indicates the position of the vertex that is located on the negative side of the x-axis and on the positive side of the y-axis among the four vertices of the display area 201 formed by the stencil (the x- The yz coordinates are (x _s , y _s , z _s )). Point O is the foot of the perpendicular to point P (with respect to the xy plane) (the intersection of the perpendicular to point P and the xy plane). In the example shown in FIG. 18, point O is assumed to be the origin. Point Q is the intersection of a straight line passing through point O and parallel to the x-axis and a straight line passing through point S and parallel to the y-axis.

If ∠OPQ is β (rad) and the length of line segment OP (distance from point O to point P) is H, then the length of line segment OQ (distance from point O to point Q) L is as follows. It is expressed by equation (1).
L=Htanβ...(1)

Further, if ∠OPS is α (rad), the length of the line segment OS (the length from point O to point S) l is expressed by the following equation (2). The length of the line segment SQ (distance from point S to point Q) w _y is expressed by the following equation (3).
l=Htanα...(2)
w _y = (l ² - L ² ) ^0.5 ...(3)

Therefore, the xyz coordinate S (x _s , y _s , z _s ) of the point S is expressed by the following equation (4).
S(x _s , y _s , z _s )=(L, w _y , 0) (4)

In this embodiment, when the length H of the line segment OP, ∠OPQ (=β), and ∠OPS (=α) are determined, the position derivation unit 1206 calculates the xyz coordinates (x _s , y _s , z _s ) are derived as the position of the display area 201 using a stencil.

The length H of the line segment OP is the value obtained by subtracting the z-coordinate _zs of the display area 201 by the stencil from the z-coordinate zp of the point _P. Therefore, if the z-coordinate of the stencil display area 201 (the position in the height direction (z-axis direction) of the stencil display area 201) is specified, the length H of the line segment OP can be derived. Note that FIG. 18 illustrates a case where the z-coordinate _zs of the stencil display area 201 is 0 (zero), so the length H of the line segment OP is equal to the z-coordinate _zp of the point P.

The position of point S in the first captured image is derived, for example, as follows. The position deriving unit 1206 determines the position of the maximum value of the x-axis, the minimum value of the x-axis, the maximum value of the y-axis, and the minimum value of the y-axis among the components of the display item by the stencil recognized in step S1702. Derive the fixed circumscribed rectangle. Then, the position derivation unit 1206 determines, among the vertices of the derived circumscribed rectangle, the lower left side (the negative side of the x-axis and the The position of the vertex located in the positive direction of the y-axis is derived as the position of the point S in the first captured image. Note that when a label is displayed in the first captured image, the position deriving unit 1206 calculates the position deriving unit 1206 so that, among the vertices of the rectangular label, the top of the constituent element of the stencil display item is located on the upper side. In this case, for example, the vertex located on the lower left side is derived as the position of point S in the first captured image.

The position of point Q in the first captured image is a position on a straight line that passes through the position of point S in the first captured image and is parallel to the y-axis, and The position corresponds to the center of the corner). Note that the point Q may be specified, for example, as follows. That is, a display area 201 is formed by stenciling a straight line passing through the position of point S and parallel to the y-axis by irradiating a laser beam from point P with a laser pointer or the like on the plate surface of the thick plate. display on the screen. In addition, a laser beam is irradiated from point P using a laser pointer or the like, and a straight line passing through point O and parallel to the x-axis is drawn on the surface of the thick plate where the display area 201 by the stencil is formed. indicate. In this case, the point Q is the intersection of the two straight lines displayed by the laser beam. Therefore, the position derivation unit 1206 may identify the point Q from the first captured image captured while the intersection of two straight lines displayed by the laser beam is displayed. In this way, when point P, point O, and point Q are on the xz plane (on the plane where y=0), the display area 201 by the stencil is formed on the plate surface of the thick plate. Even if the plane is tilted with respect to the xy plane, the position of point Q can be specified.

For example, the position derivation unit 1206 may derive ∠OPS (=α), ∠OPQ (=β), and ∠SPQ (=δ) as follows. First, the position derivation unit 1206 calculates the lengths of the line segment PS and the point PQ based on the measurement value of a distance measurement sensor (for example, a TOF (Time-of-Flight) distance measurement sensor) included in the first imaging device. Derive. Then, the position deriving unit 1206 calculates the positions of the points S and Q assuming that the plane on which the stencil display area 201 is formed is the xy plane and the line segment QS is a straight line extending in the y-axis direction. Identify. Then, the position deriving unit 1206 derives ∠OPS (=α), ∠OPQ (=β), and ∠SPQ (=δ) based on the positions of point P, point S, and point Q.

Further, for example, the position deriving unit 1206 calculates the angle of the laser beam irradiated toward the point S from the first imaging device, the angle of the laser beam irradiated toward the point Q from the first imaging device, Based on the scanning angle when the laser beam is scanned from the first imaging device from point S to point Q, ∠OPS (=α), ∠OPQ (=β), and ∠SPQ=(=δ) are calculated. May be derived.

The position derivation unit 1206 also generates a first image that is imaged while irradiating a grid-shaped laser beam onto a region of the plate surface of the thick plate that includes the surface where the stencil display area 201 is formed and the point O. By counting the number of rectangles arranged in a grid in the captured image, ∠OPS (=α), ∠OPQ (=β), and ∠SPQ (=δ) may be derived. In this case, the length of one rectangle in the x-axis direction is, for example, the length in the x-axis direction that changes when ∠OPS (=α) and ∠OPQ (=β) change by 1 rad, and the length in the y-axis direction The length in the direction is, for example, the length in the y-axis direction that changes when ∠SPQ (=δ) changes by 1 rad. Based on the result of counting the number of rectangles from the rectangle located at the position corresponding to point O to the rectangle corresponding to points S and Q, ∠OPS (=α) and ∠OPQ (=β) are derived. Furthermore, ∠SPQ (=δ) is derived based on the result of counting the number of rectangles from the rectangle located at the position corresponding to point S to the rectangle located at the position corresponding to point Q. The number of rectangles is counted based on the result of image processing on the first captured image.

Note that the positions of the display areas 1111, 1121, 1131, and 1141 are also derived in the same manner as the method described with reference to FIG. Furthermore, since the display area 201 using the stencil and the display area using the label 202 differ only in the surface on which they are arranged on the thick plate, the position of the display area using the label 202 is also determined by the method described with reference to FIG. It can be derived according to the following. Therefore, detailed explanation thereof will be omitted here.

Next, an example will be described in which it is determined in step S1708 that the position in the height direction (z-axis direction) of the display area by the stencil (label) can be specified.
As a first example, the position deriving unit 1206 first derives the distance in the height direction (z-axis direction) from the first imaging device to the stencil display area using a laser range finder. Then, the position deriving unit 1206 calculates the position obtained by subtracting the distance from the position of the first imaging device in the height direction (z-axis direction) as the position of the display area 201 by the stencil in the height direction (z-axis direction). Derive it as

As a second example, the position deriving unit 1206 extracts the information indicating the position of the thick plate stored in the article position column of the management table 1300 and the thickness information stored in the registration time column of the management table 1300. Using information indicating the date and time when the board was placed at the current position, the position in the height direction (z-axis direction) of the display area by the stencil (label) is derived. An example of a method for deriving the position in the height direction (z-axis direction) of the display area by a stencil (label) in this manner will be described. Note that here, the three-dimensional position of the thick plate stored in the article position column of the management table 1300 is referred to as an article position, as necessary. In addition, here, in order to simplify the explanation, it is assumed that the lowest plank among a plurality of stacked planks is placed on the ground.

First, the position deriving unit 1206 calculates the imaging range (xy coordinates) of the first imaging device that captured the first captured image acquired in step S1701 from a plank other than the plank identified in step S1705. A plank plate that includes at least a portion of the board is identified based on the article position stored in the management table 1300.

When deriving the position in the height direction (z-axis direction) of the display area 201 using the stencil, the thick plate specified in step S1705 is at the top. Therefore, in step S1705, the position deriving unit 1206 determines that the thick plate, at least a part of which is included in the imaging range (xy coordinates) of the first imaging device that captured the first captured image acquired in step S1701, is Determine that it is under the identified plank. In this case, all the sizes (thicknesses) stored in the management table 1300 as information on the planks below the plank identified in step S1705 are added to the thickness of the plank specified in step S1705. The value is the position in the height direction (z-axis direction) of the display area 201 by the stencil.

On the other hand, when deriving the position in the height direction (z-axis direction) of the display area by the label, the position derivation unit 1206 uses the thickness specified in step S1705 among the thick plates specified as described above. A plank whose date and time is later than the date and time stored in the registration time column of the management table 1300 as plank information is a plank below the plank identified in step S1705. It is determined that In this case, all the added values of the size (thickness) stored in the management table 1300 as information on the plank below the plank specified in step S1705, and the information on the plank specified in step S1705. The value obtained by adding the value of the z-coordinate of the position within the article stored in the management table 1300 becomes the position in the height direction (z-axis direction) of the display area of the label 202.

As a third example, when information that allows the height direction (z-axis direction) to be specified is displayed in the first captured image, the position derivation unit 1206 generates a stencil (label) based on the information. ) can derive the position of the display area in the height direction (z-axis direction). An example of a method for deriving the position in the height direction (z-axis direction) of the display area by a stencil (label) in this manner will be described. For example, if the first captured image includes a structure on which a scale indicating the position in the height direction (z-axis direction) is displayed, the position derivation unit 1206 calculates the scale of the structure from the first captured image. Recognize. For example, the position derivation unit 1206 performs edge detection processing on the first captured image to extract edges of the object, and extracts the structure from the extracted object. For example, information in which information indicating the outline of an object and the name of the object are mutually associated is registered in the article management device 1200 in advance. In this case, the position deriving unit 1206 may extract the object corresponding to the structure by comparing the edge of the object extracted as described above with the information. Further, the position derivation unit 1206 may extract the structure from the objects extracted as described above using AI (Artificial Intelligence) such as NN (Neural Network).

Then, the position deriving unit 1206 calculates the stencil (label) based on the position of the display area of the stencil (label) in the first captured image and the position of the scale displayed on the structure in the first captured image. ) is derived from the height direction (z-axis direction) of the display area. Note that the numbers attached to the scale in the first captured image are recognized using a method similar to the method for recognizing the constituent elements of display items using stencils (labels).

As a fourth example, the position derivation unit 1206 uses the second captured image acquired by the second image acquisition unit 1209 to derive the position of the display area 201 by the stencil in the height direction (z-axis direction). You may. In this case, in the flowchart of FIG. 17, a step is added in which the second image acquisition unit 1209 acquires the second captured image before the processing in step S1708 is started. The second captured image was captured by a second imaging device that is different from the first imaging device that captured the first captured image (an imaging device that captures an area including a stencil (label) display area). This is a captured image. The second captured image includes both the first imaging device that captured the first captured image acquired in step S1701 and the thick plate identified in step S1705. An example of a method for deriving the position in the height direction (z-axis direction) of the display area 201 using a stencil in this manner will be described.

FIG. 19 is a diagram illustrating an example of a method for deriving the position in the height direction (z-axis direction) of the display area 201 using a stencil.
In the example shown in FIG. 19, similarly to FIG. 18, point P indicates the position of the first imaging device (xyz coordinates of point P are (x _p , y _p , z _p ) ). Further, the z-axis is a direction perpendicular to the surface of the thick plate (the surface on which the display area 201 by the stencil is formed). Further, the y-axis is a direction parallel to the direction in which constituent elements of the same display item are lined up in the stencil display area 201. Further, the x-axis is a direction parallel to the direction in which different display items are lined up in the stencil display area 201. Further, it is assumed that the x-axis, y-axis, and z-axis are perpendicular to each other. Again, to simplify the explanation, it is assumed that the width direction of the plank is parallel to the y-axis, and the length direction of the plank is parallel to the x-axis.

Further, point O' is the intersection of the perpendicular to point P (with respect to the xy plane) and the plane that includes the surface of the thick plate (the surface on which the display area 201 by the stencil is formed) (point O' corresponds to point O in FIG. 18). Point O is the foot of the perpendicular to point P (with respect to the xy plane) (the intersection of the perpendicular to point P and the xy plane). In the example shown in FIG. 19, point O is assumed to be the origin. Point R indicates the three-dimensional position of the second imaging device (the xyz coordinates of point R are (x _r , y _r , z _r )). The three-dimensional position of the second imaging device is, for example, the position of the principal point of the second imaging device.

It is assumed that the three-dimensional position of the second imaging device can be specified by the article management device 1200. For example, if the three-dimensional position of the second imaging device is fixed, the ID of the second imaging device and the three-dimensional coordinates (xyz coordinates) of the second imaging device are associated with each other. and registered in the article management device 1200 in advance. In addition, if the three-dimensional position of the second imaging device is not fixed, for example, the second imaging device can use the three-dimensional coordinates (x-y-z coordinates) of the second imaging device that are positioned using GPS. and the ID of the second imaging device. In this case, the second image acquisition unit 1209 acquires this information transmitted from the second imaging device.

In the example shown in FIG. 19, in order to simplify the explanation, it is assumed that the lowest plank among a plurality of stacked planks is placed on the ground. Point D is the vertex closest to point R among the vertices of the surface on which the stencil display area 201 is formed. Point B is the vertex closest to point R among the vertices of the bottom surface of the lowest plank among the plurality of stacked planks.

In FIG. 19, ∠BRD (=φ _B ) and ∠PRO (=φ _P ) are, for example, the positions of point P, point B, and point D in the second captured image, and the main character of the second captured image. It is determined based on the points and. The direction in which the line segment RP extends is specified by the straight line connecting the point P in the second captured image and the principal point of the second captured image. The direction in which the line segment RB extends is specified by the straight line connecting the point B in the second captured image and the principal point of the second captured image. The direction in which the line segment RT extends is specified by the straight line connecting the point D in the second captured image and the principal point of the second captured image. ∠BRD (=φ _B ) is determined based on the direction in which line segment RB extends and the direction in which line segment RD extends. ∠PRO (=φ _P ) is determined based on the direction in which the line segment RP extends and the y-axis direction.

Further, for example, the position derivation unit 1206 calculates the length of the line segment RP ( The lengths of the line segment RD (distance from point R to point D) and line segment RD (distance from point R to point D) are derived. Then, the position deriving unit 1206 calculates the position (z _r ) of the second imaging device in the height direction (z-axis direction), the length of the line segment RP, and ∠PRO (=φ _P ). The position (z _p ) of the No. 1 imaging device in the height direction (z-axis direction) is derived (z _p =z _r +length of line segment RP×sinφ _P ). Similarly, the position deriving unit 1206 calculates the stencil based on the position (z _r ) of the second imaging device in the height direction (z-axis direction), the length of the line segment RD, and ∠BRD (=φ _B ). The position of the display area 201 in the height direction (z-axis direction) is derived.

Note that FIG. 19 illustrates a case where the position of the second imaging device (point R) in the height direction (z-axis direction) is 0 (zero). Furthermore, if the second captured image is used as described above, the position deriving unit 1206 can calculate not only the position in the height direction (z-axis direction) of the display area 201 by the stencil, but also the position in the height direction of the first imaging device. The position (z _p ) (in the z-axis direction) can be derived. Further, the position deriving unit 1206 derives not only the position in the height direction (z-axis direction) but also the position in the x-axis direction (x _p ) and the position in the y-axis direction (y _p ) of the first imaging device. be able to. For example, x _p and y _p are derived based on the direction in which the line segment RP extends and y _r +length of the line segment RP×cosφ _P. That is, a vector connecting points R and P is derived based on the direction in which line segment RP extends and y _r +length of line segment RP x cosφ _P. The length of the x-axis component and the length of the y-axis component of the vector are x _p and y _p .

Further, here, a case has been described as an example in which the point P (first imaging device) is included in the second captured image captured by the second imaging device. However, if it is possible to derive the relative positional relationship (∠PRO (=φ _P ), etc.) between the first imaging device (point P) and the second imaging device (point R), it is not necessary to do it this way. There's no need. In this case, for example, a captured image captured by at least one imaging device different from the first imaging device and the second imaging device is used. A specific example of such a case will be explained. First, a third imaging device is arranged in addition to the first imaging device and the second imaging device. It is assumed that the third imaging device includes point P (first imaging device) and point R (second imaging device) in its imaging range, but does not include point D in its imaging range. Further, a point indicating the position of the third imaging device is designated as R' (not shown). Further, it is assumed that one or both of the position of point R indicating the position of the second imaging device and the position of point R' indicating the position of the third imaging device are determined in advance. In this case, the position derivation unit 1206 derives the elevation angle or depression angle when looking at point R and point P from point R′ based on the third captured image captured by the third imaging device, and The position derivation unit 1206 derives ∠PRO (=φ _P ) based on the derived elevation angle or depression angle. Derivation of angle, direction, and length is achieved using a method similar to that described with reference to FIG. 19. Furthermore, if the imaging range of the third imaging device includes point P (first imaging device) and does not include point R (second imaging device), for example, point R' (third imaging device) is included in the imaging range of the third imaging device. What is necessary is to further use a fourth captured image captured by a fourth imaging device whose imaging range includes the point R (the second imaging device) and the point R (the second imaging device). On the other hand, if the imaging range of the third imaging device includes point R (second imaging device) and does not include point P (first imaging device), for example, point R'( It is sufficient to further use a fifth captured image captured by a fifth imaging device whose imaging range includes the point P (the third imaging device) and the point P (the first imaging device).

Further, the position deriving unit 1206 identifies the position of the point P (first imaging device) using a sixth captured image captured by a sixth imaging device whose imaging range includes the first imaging device. It's okay. In this case, it is assumed that the sixth imaging device is included in the imaging range of at least one of the second imaging device, the third imaging device, and the fourth imaging device. Further, a point indicating the position of the sixth imaging device is designated as P' (not shown). Further, it is assumed that one or both of the position of the point P indicating the position of the first imaging device and the position of the point P' indicating the position of the sixth imaging device are determined in advance. In this case, for example, in the above description, calculation is performed using the sixth imaging device and point P' instead of the first imaging device and point P. Then, for example, based on the result of this calculation, the sixth captured image, and the position of one or both of point P and point P', the first imaging device (point P) and the second imaging device A relative positional relationship (∠PRO (=φ _P ), etc.) with (point R) is derived.

With the above method, even if the second imaging device cannot be arranged so that the imaging range of the second imaging device includes both the first imaging device and the thick plate, the stencil The position of the display area 201 in the height direction (z-axis direction) can be derived. Note that the third to sixth captured images are acquired by, for example, the second image acquisition unit 1209. However, in addition to the second image acquisition unit 1209, third to sixth image acquisition units that individually acquire the third to sixth captured images may be used. In this case, in the flowchart of FIG. 17, the step of acquiring the second captured image by the second image acquisition unit 1209 is executed at a timing before the timing at which the process of step S1708 is started. A step of individually acquiring the third to sixth captured images is added at a timing different from the timing at which the third to sixth captured images are acquired. Note that if the position in the height direction (z-axis direction) of the display area by the stencil (label) can be specified without acquiring the second captured image, etc., the second image acquisition unit 1209 becomes unnecessary.

As a fifth example, the position deriving unit 1206 converts the height direction (z-axis direction) of the stencil display area 201 of the thick plates 1002a and 1002b placed on the transport table 1001 into the position of the transport table 1001. and the size (thickness) stored in the management table 1300 as information on the thick plates 1000a and 1002b. In this case, the position of the transport table 1001 is registered in the article management device 1200 in advance.

For example, in at least one of the above examples, if the position in the height direction (z-axis direction) of the display area by the stencil (label) can be derived, in step S1708, the position of the display area by the stencil (label) is determined. It is determined that the position in the height direction (z-axis direction) can be specified.

Further, if the plank is on the carrier, in step S1709, the position derivation unit 1206 acquires the three-dimensional position of the plank on the carrier. In this embodiment, the carriers are a transport ship 400 and a transport vehicle 700. Therefore, in this embodiment, a case where the transport equipment is a transport ship 400 and a transport vehicle 700 will be exemplified.
For example, when a plank is loaded onto the transport ship 400, the position derivation unit 1206 acquires the ID of the plank and information indicating the three-dimensional position of the plank on the transport ship 400. Then, the position deriving unit 1206 identifies a column in which information indicating the same ID as the acquired thick plate ID is stored from the ID column of the management table 1300. Then, the position deriving unit 1206 stores information indicating the acquired three-dimensional position of the plank in the transport ship 400 in the article position column of the specified column. For example, the operator (worker) of the transport ship 400 may input the ID of the plank and information indicating the three-dimensional position of the plank in the transport ship 400 into a handheld information processing terminal (tablet terminal, etc.). . In this case, by transmitting the ID of the plank and information indicating the three-dimensional position of the plank in the transport ship 400 from the information processing terminal to the article management device 1200, the position derivation unit 1206 The ID of the plank and information indicating the three-dimensional position of the plank in the transport ship 400 may be acquired. The information indicating the three-dimensional position of the plank is, for example, the xyz coordinates of the display area by a stencil (label).

Furthermore, when the plank is loaded onto the transport vehicle 700, the position derivation unit 1206 acquires the ID of the plank and information indicating the three-dimensional position of the plank in the transport vehicle 700. Then, the position derivation unit 1206 specifies, from the ID column of the management table 1300, a column in which information indicating the same ID as the obtained ID of the thick plate is stored. Then, the position deriving unit 1206 stores information indicating the obtained three-dimensional position of the thick plate in the transportation vehicle 700 in the article position column of the specified column. For example, the operator (worker) of the transport vehicle 700 may input the ID of the plank and information indicating the three-dimensional position of the plank in the transport vehicle 700 into a handheld information processing terminal (tablet terminal, etc.). . In this case, by transmitting the ID of the plank and information indicating the three-dimensional position of the plank in the transport vehicle 700 from the information processing terminal to the article management device 1200, the position deriving unit 1206 can: The ID of the plank and information indicating the three-dimensional position of the plank in the transportation vehicle 700 may be acquired. The information indicating the three-dimensional position of the plank is, for example, the xyz coordinates of the display area by a stencil (label).

By acquiring the three-dimensional positions of the planks on the carrier as described above, it is possible to specify the carrier that transports each plank. Further, it is possible to accurately specify the three-dimensional position of the plank while the plank is being transported by the carrier.

As a sixth example, use an imaging device in which the number of pixels and size of the image sensor (length of each pixel [m/pixel]) and the focal length, which is the distance from the image sensor to the principal point, are known. By acquiring the position of the display area of the stencil (label) in real space, the position of the display area of the stencil (label) in the height direction (z-axis direction) may be derived.

Taking as an example a case where the stencil display area 201 is a rectangle and the real space lengths of the long side and short side are stored in advance in the article management device 1200, the sixth example will be explained using FIG. 20A and FIG. An example will be explained. Note that, as described above, information on the number of pixels of the sensor of the first imaging device, information on the size of the sensor, and focal length for each optical magnification used at the time of photographing are stored in advance in the article management device 1200. ing. The focal length is information determined at the time of imaging.

FIG. 20A is a diagram showing an example of a first captured image including a display area 201 formed by a stencil. When the display area 201 formed by the stencil is imaged from the front, it becomes a rectangle in the first captured image. On the other hand, when the display area 201 formed by the stencil is imaged from an oblique direction, it may become a scalene quadrilateral in the first captured image, as shown by the broken line (virtual line) in FIG. 20A. FIG. 20B is a diagram illustrating an example of the geometrical relationship between the inside of the first imaging device and the display area 201 formed by the stencil. In FIG. 20B, the position deriving unit 1206 first specifies the positions of the four corner vertices 002a to 2002d of the stencil display area 201' imaged by the image sensor 2002. Next, the position deriving unit 1206 specifies straight lines 2003a to 2003d, which are the straight lines connecting the four corner vertices 2002a to 2002d of the stencil display area 201' and the principal point FP, and are extended toward the stencil display area 201 side. . Next, the position deriving unit 1206 calculates that the principal point FP is the vertex of the head, the straight lines 2003a to 2003d are the four sides with the principal point FP as one end, and the bottom dimension is the display area 201 (rectangular) formed by the stencil. ) is specified as the length of the real space of the long side and short side of the square pyramid 2001. As described above, when the stencil display area 201 is a rectangle, the real space lengths of the long sides and short sides of the rectangle are stored in advance in the article management device 1200. The position derivation unit 1206 derives the three-dimensional positional relationship between the stencil display area 201 and the principal point FP (first imaging device) based on the square pyramid 2001 identified in this manner.

Here, for ease of explanation, an example is given in which the shape of the display area by the stencil is a rectangle whose dimensions are known in advance. However, in the method of the sixth example, the shape of the display area from which the three-dimensional positional relationship with the first imaging device is derived is not limited to a rectangle. For example, even if the shape and size of a character, symbol, mark, etc. are known in advance, if the stencil display area 201 shown in FIG. 20B is replaced with the display area of the display, the position and main A three-dimensional positional relationship with the point (first imaging device) can be derived.

Furthermore, here, a case has been exemplified in which the display area from which the three-dimensional positional relationship with the first imaging device is derived is a plane. However, in the method of the sixth example, the display area from which the three-dimensional positional relationship with the first imaging device is derived is not limited to a plane. If the shape of the surface of the display area is known, the three-dimensional positional relationship between the display area and the first imaging device can be derived. For example, when the position derivation unit 1206 derives the three-dimensional positional relationship between the display area of the display made on the surface of the cylindrical article and the first imaging device, this will be explained with reference to FIG. 20B. Instead of the dimensions of the long side and short side of the rectangle, the shape and dimensions of the display area and the curvature of the display surface of the display area may be stored in advance in the article management device 1200. This is because, based on this information, it is possible to specify what corresponds to the bottom surface of the square pyramid in the method of the sixth example described with reference to FIG. 20B.

Returning to the explanation of FIG. 17, if it is determined in step S1708 that the position in the height direction (z-axis direction) of the display area by the stencil (label) cannot be specified, the process in step S1712 is executed. Ru. In step S1712, the output unit 1208 outputs information indicating that the three-dimensional position of the plank cannot be specified and the ID of the first imaging device that imaged the plank. For example, the output unit 1208 causes the computer display to display information indicating that the three-dimensional position of the plank cannot be specified and the ID of the first imaging device that captured the image of the plank. When these pieces of information are output, the operator (worker) goes to the location where the corresponding first imaging device is located and specifies the three-dimensional position of the corresponding plank. Further, the operator (worker) may specify the three-dimensional position of the corresponding thick plate from the preview image of the corresponding first imaging device.

Next, in step S1713, the storage unit 1207 inputs information indicating the three-dimensional position of the thick plate specified by the operator (worker), and stores it in the non-product display column of the management table 1300 as information for the thick plate. The information stored in the item position column is updated to the input information. Note that the information indicating the three-dimensional position of the plank can be input in the following ways: input by operating the user interface of the article management device 1200, receiving information transmitted from an external device, and information stored in a portable storage medium. At least one of these readings can be adopted. Then, the process according to the flowchart of FIG. 17 ends.

<Summary>
As described above, in this embodiment, the article management device 1200 is configured to display a display area in which information displayed on a thick plate and managed by the manufacturer of the thick plate is displayed using a stencil (label). A first captured image of the area including the area is acquired. The article management device 1200 determines the three-dimensional position of the display area of the stencil (label) based on the acquired first captured image and the position of the first imaging device that captured the first captured image. Information indicating the three-dimensional position is stored in a storage medium in association with the ID of the plank. Therefore, for example, when a thick plate is transported using various transport devices (transport ship 400, transport vehicle 700, transport table 1001, etc.) or when a worker transports a thick board, or when a storage lot address is determined, The position of the planks can be managed even when the planks are placed in an empty storage area or when the planks are stacked. Therefore, the position of the thick plate can be appropriately managed.

Furthermore, in the present embodiment, the article management device 1200 uses the stencil (label) when the three-dimensional position of the display area by the stencil (label) and the ID of the thick plate are already stored in the storage medium in association with each other. When the three-dimensional position of the display area based on the (label) is newly derived, the three-dimensional position of the display area based on the stencil (label) already stored in the storage medium is updated to the newly derived three-dimensional position. Therefore, it is possible to improve the tracking accuracy of the position of the plank whose position (location) changes with the passage of time.

Furthermore, in this embodiment, the article management device 1200 recognizes the information (ID305, 312) displayed in the stencil (label) display area based on the first captured image. The article management device 1200 then collates the recognized information with the information (ID) stored in the management table 1300, and identifies the thick plate based on the collation result. Therefore, the plank itself whose position is to be managed can be reliably specified based on the first captured image. Therefore, for example, the position of the thick plate can be specified with high precision compared to the case where the three-dimensional position of the thick plate is specified based on the operation of the transport device.

Furthermore, in the present embodiment, the article management device 1200 acquires a second captured image captured by a second imaging device different from the first imaging device that captured the first captured image. The second captured image includes the first captured image and the plank (whose position is to be managed). The article management device 1200 derives the position of the stencil display area 201 in the height direction (z-axis direction) based on the acquired second captured image and the position of the second imaging device. The article management device 1200 then stores the position of the stencil display area 201 in the height direction (z-axis direction), the first captured image, and the position of the first imaging device that captured the first captured image. , the three-dimensional position of the display area 201 by the stencil is derived as the three-dimensional position of the thick plate. Therefore, even if the position of the thick plate in the height direction cannot be derived even if the first captured image or the sensor included in the first captured image is used, the position of the thick plate can be managed.

Furthermore, in this embodiment, the stencil display and the label display are information that can be identified by humans. Therefore, it becomes easy for the operator (worker) to manage the actual product. Therefore, the convenience of managing the actual item is improved.

Furthermore, in this embodiment, the stencil display and the label display include information that is individually determined for each thick plate (information unique to the thick plate). Therefore, the plank can be uniquely identified based on the first captured image. Therefore, thick plates can be easily and quickly identified.

Furthermore, in this embodiment, the article management device 1200 derives three-dimensional coordinates (xyz coordinates) as the position of the display area by the stencil (label). Therefore, for example, even if a plurality of planks are stacked on top of each other, the positions of the planks can be managed.

<Modified example>
<<First modification example>>
The method for deriving the position of the display area based on the stencil (label) included in the first captured image in step S1709 is not limited to the method described with reference to FIG. 18. For example, the position derivation unit 1206 derives the position of the origin when determining the position within the article in the management table 1300 from the first captured image. Then, the position derivation unit 1206 displays the stencil (label) included in the first captured image based on the derived origin position and the information stored in the column of the position within the article of the management table 1300. Derive the location of the region. An example of a method for deriving the position of the stencil (label) display area included in the first captured image in this manner will be described using the stencil display area 201 as an example.

FIG. 21 is a diagram illustrating an example of a method for deriving the position of the display area 201 using a stencil. Similar to FIG. 18, FIG. 21 exemplifies a case where the xyz coordinates S (x _s , y _s , z _s ) of the point S are derived as the position of the display area 201 by the stencil. As explained with reference to FIG. 14, the position of the stencil in the article is determined from the vertex 203 shown in FIG. It is expressed in xy coordinates with the direction along the length of the plate being the positive direction of the x-axis. Therefore, the position of point E in FIG. 21 is the origin of the position of the stencil within the article. Point F is a point among the vertices of the thick plate that is lined up in the width direction (y-axis direction) with respect to point E (point E and point F form one side along the width direction of the thick plate). Point T is the foot of the perpendicular line of point P to line segment EF (the intersection of the perpendicular line of point P and line segment EF). Point P and point O are the same as shown in FIG.

In FIG. 21, the length of the line segment PT is h _p . Also, the length of line segment PF is h ₁ , ∠FPT is θ ₁ (rad), the length of line segment PE is h ₂ , ∠EPT is θ ₂ (rad), and ∠EPF is θ (rad). ) (θ=θ ₁ +θ ₂ ). Also, let the length of the line segment FT be w ₀₁ and the length of the line segment ET be w ₀₂ .

Then, the x coordinate x _e of point E becomes the value obtained by adding the length l _p of the line segment OT to the x coordinate x _p of point P (x _e = x _p + l _p ). The y coordinate y _e of the point E is the value obtained by subtracting the length w _o2 of the line segment ET from the y coordinate y _p of the point P (y _e =y _p - w _o2 ). Therefore, if the length l _p of the line segment OT and the length w _o2 of the line segment ET are derived, the xy coordinates (x _e , y _e ) is derived.

The length l _p of the line segment OT is expressed by the following equation (5).
l _p =√(h _p ² - H ² )=√(h ₁ ² cos ² θ ₁ - H ² )=√(h ₂ ² cos ² θ ₂ - H ² )...(5)
The length w _o2 of the line segment ET is expressed by the following equation (6).
w _o2 = h ₂ sin θ ₂ ...(6)
Therefore, for example, if the length h ₂ and ∠EPT (=θ ₂ ) of the line segment PE are derived, the length l _p of the line segment OT and the length w _o2 of the line segment ET are derived. Note that the length H of the line segment OP is derived as described with reference to FIG. 18.

The position of point T in the first captured image is a position on the line segment EF in the first captured image, and corresponds to the center of the horizontal angle of view (angle of view in the y-axis direction) at the time of imaging. . ∠EPT (=θ ₂ ) is determined, for example, based on the positions of point E and point T in the first captured image acquired in step S1701 and the principal point of the first captured image. The direction in which the line segment PE extends is specified by the straight line connecting the point E in the first captured image and the principal point of the first captured image. The direction in which the line segment PT extends is specified by the straight line connecting the point T in the first captured image and the principal point of the first captured image. ∠EPT (=θ ₂ ) is determined based on the direction in which line segment PE extends and the direction in which line segment PT extends.
The length _h2 of the line segment PE is derived, for example, using a distance measurement sensor (for example, a TOF (Time-of-Flight) distance measurement sensor) included in the first imaging device. In addition, the length _h2 of the line segment PE may be derived based on the length H of the line segment OP and ∠OPE.

As described above, the x coordinate x _e and the y coordinate y _e of point E are derived. Then, the position deriving unit 1206 calculates the xy coordinates ( Ls10, Ws10) are read. Then, the position derivation unit 1206 derives the value obtained by adding LS10 to the x coordinate x _e of point E as the x coordinate x _s of point S (x _s =x _e +LS10). Furthermore, the position deriving unit 1206 derives the value obtained by adding WS10 to the y coordinate x _y of point E as the y coordinate y _s of point S (y _s = y _e +WS10).

For example, depending on at least one of the location where the plank is placed, the position of the stencil (label) display area, and the direction in which the plank is placed, the stencil (label) display area and the first There are cases where the mathematical expression representing the geometrical positional relationship between the imaging device and the expression described with reference to FIGS. 18 and 21 is not satisfied. For example, if a plurality of planks are stacked on a freight car and some of the plurality of planks cannot be imaged by the imaging device, the topmost of the plurality of planks is placed on top of the plurality of planks. The height position of the plank may not be determined. Further, the mathematical expression representing the geometrical positional relationship between the display area formed by the stencil (label) and the first imaging device may become complicated. Therefore, for example, the first imaging device may capture the luminance image and the distance image as the first captured images. Note that the brightness image is the captured image described so far in this embodiment. The brightness image has information on the intensity of reflected light from the subject (brightness of the subject), but does not have information on the distance from the imaging device to the subject. The distance image has information on the distance from the imaging device to the subject. Note that the first imaging device images the brightness image and the distance image so that the positions of the pixels of the brightness image and the positions of the pixels of the distance image correspond to each other. Further, as a method for measuring a distance image, a known method such as a TOF (Time Of Flight) method or a pattern irradiation method may be used.

In this case, the position derivation unit 1206 identifies pixels corresponding to the display area of the stencil (label) based on the luminance image. For example, the position derivation unit 1206 stores an image of a stencil (label) in advance as a template image. The position derivation unit 1206 specifies a display area for a stencil (label) from the brightness image by performing pattern matching between the brightness image and the template image. The position derivation unit 1206 specifies an area of the range image that corresponds to the display area of the stencil (label) specified from the brightness image. The position derivation unit 1206 extracts pixel values of the specified area from the distance image. The pixel value of the distance image is a value indicating the distance from the first imaging device to the subject. The pixel value extracted from the distance image in this manner indicates the distance from the first imaging device to the display area defined by the stencil (label). In this manner, the position deriving unit 1206 can obtain the distance from the first imaging device to the predetermined position of the display area defined by the stencil (label) based on the pixel values extracted from the distance image. The position deriving unit 206 calculates the three-dimensional position (x-y-z coordinates) of the first imaging device, the attitude of the first imaging device when capturing the first captured image, and the information from the first imaging device. The three-dimensional position (xyz coordinates) of the predetermined position in the display area by the stencil (label) is derived based on the distance to the predetermined position in the display area by the stencil (label).

<<Second variant>>
As in this embodiment, a person identifies the customer name 302, standard 303, size 304, ID 305, customer code 306, order number 307, standard 311, ID 312, size 313, delivery date 314, and customer code 315. It is preferable if the information is available because it makes it easier for the operator (worker) to manage the actual product. However, at least one of the customer name 302, standard 303, size 304, ID 305, customer code 306, order number 307, standard 311, ID 312, size 313, delivery date 314, and customer code 315 may be changed to a bar code or It may also be information that is converted (encoded) into information that cannot be identified by humans, such as a dimensional code.

<<Third modification example>>
In this embodiment, as shown in FIGS. 3A and 3B, a case has been exemplified in which the ID 305 displayed by the stencil and the ID 312 displayed by the label are the same. However, the expressions and written contents of these IDs may differ. For example, some formats do not have a "hyphen" in the ID. In such a case, information indicating that these IDs (stencil ID, label ID) are for the same thick plate is stored in the management table 1300. Regarding display items other than IDs, the expression and written content of the same display item may be different depending on whether it is displayed using a stencil or displayed using a label. For example, if a standard is displayed on both the stencil and the label, the abbreviation of the standard may be displayed on the label, for example. Further, although the number of digits of the customer code may differ depending on whether it is displayed using a stencil or a label, the customer code is managed by the manufacturer. Also, regarding sizes, not all digits of dimensions may be shown. Even in such a case, information indicating that the different expressions and written contents have the same meaning is stored in the management table 1300.

<<Fourth variation>>
In this embodiment, a case has been exemplified in which the three-dimensional position (xyz coordinates) of the display area by the stencil (label) is derived as the three-dimensional position of the thick plate. However, the three-dimensional position of the plank is not limited to the three-dimensional position (xyz coordinates) of the display area by the stencil (label). For example, the article management device 1200 recognizes only one element displayed on the surface of a plank, and converts the xyz coordinates of the position (for example, the center of gravity or centroid) of the recognized element into the three-dimensional coordinates of the plank. It may also be the original position. An element is information that is treated as a single piece of information, such as letters, numbers, symbols, and marks. For example, an element is information that is recognized as one piece of information by the recognition unit 1202. For example, the article management device 1200 may recognize one element among the components of a display item in a stencil (label) display. Furthermore, the article management device 1200 may recognize one element other than the constituent elements of the display items in the stencil (label) display. When the article management device 1200 derives the three-dimensional position of one element as the three-dimensional position of the thick plate in this way, for example, the recognized circumscribed rectangle of one element is regarded as the display area 201 by the stencil, and the present embodiment Perform the processing described in .

Additionally, the article management device 1200 may recognize only two or more elements displayed on the surface of the plank. The two or more elements displayed on the surface of the plank are, for example, two or more of the components of the display item in a stencil (label) display. However, the two or more elements displayed on the surface of the plank are not all of the components of the display item in the stencil (label) display. Furthermore, the two or more elements displayed on the surface of the plank may be elements different from the constituent elements of the displayed items in the stencil (label) display. When deriving the three-dimensional positions of two or more elements as the three-dimensional positions of the plank in this way, the article management device 1200, for example, calculates the circumscribed rectangle of the two or more elements displayed on the surface of the plank. The processing described in this embodiment is performed by regarding the display area 201 as a stencil display area.

Furthermore, even if the characters are of the same size, characters located farther from the imaging device are imaged smaller than characters located closer to the imaging device. Taking advantage of this, the position deriving unit 1206 may derive the three-dimensional position of the article based on the position and size of the information displayed in the first captured image.

<<Fifth modification example>>
Furthermore, the article management device 1200 may include a direction derivation section. The direction derivation unit derives information specifying the direction in which the article is placed based on a plurality of three-dimensional positions (xyz) on one surface of the article. For example, the direction derivation unit derives the angle between the plate surface of the thick plate and the reference plane based on the three-dimensional positions (xyz coordinates) of the two vertices 1401a and 1401b of the circumscribed rectangle shown in FIG. do. The reference plane is, for example, the ground. The reference surface may be a surface other than the ground. In this case, the angle between the plate surface of the thick plate and the reference plane is an example of information that specifies the direction in which the article is placed. However, the information specifying the direction in which the article is placed is not limited to the angle between the plate surface of the article and the reference plane. For example, the axis of symmetry (central axis) of the article may be used instead of the plate surface of the article.

In addition, instead of or in addition to the two vertices 1401a and 1401b of the circumscribed rectangle shown in FIG. (xyz coordinates) may also be used. In addition, instead of or in addition to these three-dimensional positions (xyz coordinates), the direction deriving unit calculates the three-dimensional position (x -yz coordinates) may also be used.

Furthermore, when this modification is adopted, a column storing information specifying the direction in which the article is placed is added to the non-product display column of the management table 1300 shown in FIG. 13. In the following description, the column in which information specifying the direction in which the article is placed is stored will be referred to as the article direction column. The storage unit 1207 stores information specifying the direction in which the thick plate is placed in the article direction column corresponding to the thick plate in the management table 1300. As a result, information specifying the direction in which the plank is placed, and the ID, three-dimensional position, etc. of the plank are stored in association with each other.

In addition, if the plate surface of the plank is inclined with respect to the ground, the position in the height direction (z-axis direction) of the position (point) to be derived from the three-dimensional position (x-y-z coordinates) is, for example, , is derived as follows. First, the position derivation unit 1206 uses a laser range finder to calculate the height direction (z-axis direction) from the first imaging device to the information (point) from which the three-dimensional position (xyz coordinates) is to be derived. Derive the distance. Then, the position deriving unit 1206 calculates the position obtained by subtracting the distance from the position of the first imaging device in the height direction (z-axis direction) as information for deriving the three-dimensional position (xyz coordinates). It is derived as the position of (point) in the height direction (z-axis direction).

Furthermore, as explained in <<First Modification>>, the position derivation unit 1206 obtains information (point ) may be derived.

<<Sixth variation>>
In this embodiment, a case has been exemplified in which the xyz coordinates of the display area by the stencil (label) are derived as the three-dimensional position of the display area by the stencil (label). However, the three-dimensional position of the display area by the stencil (label) is not limited to xyz coordinates. For example, the three-dimensional position of the display area by the stencil (label) may be a relative three-dimensional position with respect to the position of the first imaging device. In this case, it is not necessary to use the coordinates (x _p , y _p , z _p ) of the point P indicating the three-dimensional position of the imaging device. Note that when the origin of the xyz coordinates is the three-dimensional position of the first imaging device, the three-dimensional position of the display area by the stencil (label) may be expressed as the xyz coordinates. This indicates a relative three-dimensional position based on the three-dimensional position of the first imaging device.

<<Seventh variation>>
In this embodiment, the case where the identification information acquisition unit 1210 includes the recognition unit 1202, the acquisition unit 1203, the collation unit 1204, and the identification unit 1205 is illustrated. However, as long as the identification information acquisition unit 1210 has acquired the identification information of the plank to be associated with the three-dimensional position derived by the position derivation unit 1206, the recognition unit 1202, the acquisition unit 1203, the collation unit 1204, and the identification It is not necessary to have the section 1205. For example, the identification information acquisition unit 1210 may include only the recognition unit 1202. Further, the identification information acquisition unit 1210 may include only the acquisition unit 1203, for example. Further, the identification information acquisition unit 1210 may include only the recognition unit 1202 and the acquisition unit 1203, for example. For example, the identification information acquisition unit 1210 may acquire the ID recognized by the recognition unit 1202 as the identification information of the thick plate. In this case, the identification information acquisition section 1210 does not need to include the acquisition section 1203, the collation section 1204, and the identification section 1205. Further, for example, if the number of articles whose positions are to be managed is limited to one in advance, the identification information acquisition unit 1210 may acquire the ID acquired by the acquisition unit 1203 as the identification information of the thick plate. . In this case, the identification information acquisition unit 1210 does not need to include the recognition unit 1202, the verification unit 1204, and the identification unit 1205. For example, when identifying a thick plate only from the information recognized by the recognition unit 1202, the identification information acquisition unit 1210 uses the recognition unit 1202, and limits the number of articles whose positions are to be managed to one in advance. In this case, the acquisition unit 1203 may be used. In this case, the identification information acquisition unit 1210 does not need to have the collation unit 1204 and the identification unit 1205 (it may have only the recognition unit 1202 and the acquisition unit 1203).

<<Eighth modification example>>
In this embodiment, the case where the management table 1300 is stored in the article management device 1200 is illustrated. However, the management table 1300 may be located outside the article management device 1200.

<<Ninth modification example>>
In this embodiment, the case where the article whose existing position changes is a thick plate is exemplified. However, articles that change their location are not limited to planks. For example, the article whose position changes may be a product other than plank. The article whose existing position changes may be a manufactured product on which information managed by the manufacturer is displayed. The article of manufacture may be, for example, a slab, a billet, a sheet coil, or a steel tube. Moreover, it may be an article other than a manufactured product. The article whose existing position changes may be, for example, a container.

(Second embodiment)
Next, a second embodiment will be described. In the first embodiment, the article management device 1200 manages the position of an article whose existing position changes. In this embodiment, a case will be described in which the article management device 1200 manages the positions of articles installed in the installation area. As described above, in this embodiment, it is assumed that the article whose location is to be managed is an article whose location is not subject to change. As described above, the present embodiment and the first embodiment differ mainly in the configuration and processing due to the difference in whether or not the location of the article whose location is to be managed is assumed to change. Therefore, in the description of this embodiment, the same parts as those in the first embodiment are denoted by the same reference numerals as in FIGS. 1 to 21, and detailed description thereof will be omitted.

Examples of articles managed by the article management device 1200 of this embodiment include the articles shown in FIGS. 22 and 23.

FIG. 22 is a diagram showing an example of a steel frame of a structure installed on the ground. In the example shown in FIG. 22, the installation area is the ground. Structures are, for example, buildings and bridges. In addition, in FIG. 22, in order to simplify the notation, only a part of the steel frame of the structure is shown in a simplified manner. In addition, in FIG. 22, among the structural members 2201a to 2201d that constitute the steel frame of a structure installed on the ground, the structural members 2201a to 2201b are uniquely identified on the surfaces of the structural members 2201a to 2201b that become beam members. A case where IDs 2202a to 2202b are displayed as information for the purpose will be exemplified. Note that the structural material is, for example, a metal material such as steel. The structural members 2201a to 2201b are H-shaped steel. ID2202a to 2202b are information corresponding to ID305 and ID312. The IDs 2202a to 2202b are displayed, for example, by at least one of a stencil, handwriting, and engraving. Further, the IDs 2202a to 2202b may be displayed using labels. Further, information other than the IDs 2202a to 2202b may be displayed on the surfaces of the structural members 2201a to 2201b.

Note that, in order to simplify the notation, FIG. 22 exemplifies a case where the ID 2202 is displayed only on the structural members 2201a to 2201b among the structural members 2201a to 2201d. However, information including information (ID) for uniquely identifying the structural members 2201c to 2201d may also be displayed on the surfaces of the structural members 2201c to 2201d. Further, FIG. 22 illustrates a case where the connection between the structural member 2201a and the structural members 2201c to 2201d and the connection between the structural member 2201b and the structural members 2201c to 2201d are each performed by welding. However, the method of connecting the structural members 2201a to 2201d is not limited to welding. For example, two structural members may be connected using a splice plate.

In the example shown in FIG. 22, after the structural members 2201a to 2201b are installed and before the surfaces of the structural members 2201a to 2201b become invisible due to painting etc. of the structural members 2201a to 2201b, the structural members 2201a to 2201b including IDs 2202a to 2202b are A captured image of the area is captured using the first imaging device. In this way, the IDs 2202a to 2202b (information for uniquely identifying the article) are hidden and become invisible after installation. The role of the first captured image is the same as the role of the first captured image described in the first embodiment. As the first imaging device, for example, a first imaging device held by an operator (worker) is used. Further, as the first imaging device, a first imaging device provided in an unmanned aerial vehicle (drone) may be used. In the following description, the area of the structural members 2201a to 2201b including the IDs 2202a to 2202b will be referred to as the ID display area of the structural member, if necessary. The ID display area of the structural material corresponds to the display area by the stencil (label) of the first embodiment.

FIG. 23 is a diagram showing an example of a pipe installed underground. In the example shown in FIG. 23, the installation area is underground. The pipe is, for example, at least one of a metal pipe, a clay pipe, a PVC pipe, and an insulated pipe. FIG. 23 illustrates a case where an ID 2302 is displayed on the surface of the pipe 2301a among the pipes 2301a to 2301c installed underground as information for uniquely identifying the pipe 2301a. ID2302 is information corresponding to ID305 and ID312. The ID 2302 is displayed, for example, by at least one of a stencil, handwriting, and engraving. Further, the ID 2302 may be displayed by a label. Furthermore, as described in the first embodiment, information other than the ID 2302 may be displayed on the surface of the tube 2301a.

Note that, in order to simplify the notation, FIG. 23 exemplifies a case where the ID 2302 is displayed only on the tube 2301a among the tubes 2301a to 2301c. However, information including information (ID) for uniquely identifying the tubes 2301b to 2301c may also be displayed on the surfaces of the tubes 2301b to 2301c. Further, FIG. 23 illustrates a case where pipes 2301a to 2301c are connected using joint members 2303a to 2303b. However, the method of connecting the pipes 2301a to 2301c is not limited to the method using joint members 2303a to 2303b.

In the example shown in FIG. 23, after the tube 2301a is installed and before the hole 2304 is filled and the tubes 2301a to 2301c become invisible, a captured image of the region of the tube 2301a including the ID 2302 is captured using the first imaging device. Imaged. In this way, the ID 2302 (information for uniquely identifying the article) is hidden and becomes invisible after installation. The role of the first captured image is the same as the role of the first captured image described in the first embodiment. As the first imaging device, for example, a first imaging device held by an operator (worker) is used. Further, as the first imaging device, a first imaging device provided in an unmanned aerial vehicle (drone) may be used. In the following description, the area of the tube 2301a that includes the ID 2302 will be referred to as a tube ID display area, if necessary. The ID display area of the tube corresponds to the stencil (label) display area of the first embodiment.

An example of the functional configuration of the article management device 1200 of this embodiment is the same as an example of the functional configuration of the article management device 1200 of the first embodiment shown in FIG. 12.

Furthermore, an example of the management table managed by the article management device 1200 of this embodiment is the management table 2400 shown in FIG. 24. In the following description including the drawings, when managing the positions of structural members (structural members 2201a to 2201b, etc.), the article management device 1200 (storage unit 1207) Information used to manage the information is stored in the management table 2400. When managing the positions of structural members (structural members 2201a to 2201b, etc.), the article management device 1200 (storage unit 1207) stores information used for managing the positions of pipes (pipe 2301a, etc.) in the management table 2400. There's no need. On the other hand, when managing the position of a pipe (such as the pipe 2301a), the article management device 1200 (storage unit 1207) stores information used for managing the position of the pipe (such as the pipe 2301a) in the management table 2400. When managing the positions of pipes (pipe 2301a, etc.), the article management device 1200 (storage unit 1207) stores information used for managing the positions of structural members (structural members 2201a to 2201b, etc.) in the management table 2400. There's no need. Similarly to Fig. 13, the information in the same column in Fig. 24 is the identification information for the same structural material (pipe) (the column under the same No. (It becomes a column where information is stored respectively). In the following description, including the drawings, structural members (structural members 2201a to 2201b, etc.) and pipes (pipe 2301a, etc.) are referred to as position-fixing articles as necessary.

Furthermore, a flowchart illustrating an example of the processing of the article management device 1200 of this embodiment is the flowchart shown in FIG. 25. Note that the flowchart explaining an example of the process of the article management device 1200 when managing the position of a structural member is different from the flowchart explaining an example of the process of the article management device 1200 when managing the position of a pipe. It becomes a flowchart. When the flowchart shown in FIG. 25 is a flowchart illustrating an example of the processing of the article management device 1200 when managing the position of a structural material, the "position fixed article" in FIG. 25 means the "structural material". . When the flowchart shown in FIG. 25 is a flowchart illustrating an example of the processing of the article management apparatus 1200 when managing the position of a pipe, the "position-fixed article" in FIG. 25 means a "pipe."

FIGS. 22 and 23 illustrate a case where only IDs 2202a to 2202b and 2302 are displayed on structural members 2201a to 2201b and pipe 2301a. Therefore, in the management table 2400 shown in FIG. 24, only the ID column in which the IDs 2202a to 2202b and 2302 are stored is shown in the actual item display column. However, if information other than the IDs 2202a to 2202b and 2302 is displayed for the structural members 2201a to 2201b and the pipe 2301a, even if the column in which the information is stored is included in the actual product display column of the management table 2400. good.

The article position column of the management table 2400 stores information indicating the three-dimensional position of the position-fixed article. The information stored in the article position column is information indicating the three-dimensional position of the position-fixed article specified by article management device 1200. In this embodiment, a case will be exemplified in which the three-dimensional position of the ID display area of the structural material and the three-dimensional position of the ID display area of the pipe are the three-dimensional position of the structural material and the three-dimensional position of the pipe, respectively. Further, in this embodiment, the three-dimensional position of the ID display area of the structural material and the three-dimensional position of the ID display area of the pipe are examples of information display positions.

The registration time column of the management table 2400 stores information indicating the date and time (timing determined from the date and time) when the position-fixed article was placed at the current position. Note that the date and time when the position-fixed article was placed at the current position refers to the date and time when the operation for acquiring the date and time was performed. In the registration time column, for example, the imaging date and time of the first captured image is stored. However, the information stored in the registration time column is not limited to the imaging date and time of the first captured image (this also applies to the first embodiment). For example, the operator may input information on the date and time when the position-fixed article was placed at the current position to the article management device 1200 by operating the user interface of the article management device 1200. In this embodiment, the information stored in the registration time column is an example of the date and time when an operation to obtain the date and time when the article was placed at the current position was performed.

As described above, the IDs 2202a to 2202b displayed on the structural members 2201a to 2201b become invisible due to painting or the like. Further, the ID 2302 displayed on the pipe 2301a is buried underground and becomes invisible. Even in such a case, the operator can grasp the elapsed time since the position-fixed article was installed based on the information stored in the registration time column. Therefore, for example, it can be determined whether the position-fixing article needs to be repaired or replaced, based on the elapsed time since the position-fixing article was installed. Also, based on the information stored in the ID column and the article position column, it is possible to determine which position-fixed article exists in which position, regardless of whether or not the IDs 2202a to 2202b and 2302 are visible. You can understand it without any problem. Therefore, it is possible to grasp what kind of position-fixed article is the position-fixed article that requires repair or replacement, and at what position. Therefore, the information stored in the management table 2400 can be used, for example, to repair or replace a position-fixed article whose ID (information for uniquely identifying the article) cannot be visually recognized.

Note that the non-product display column of the management table 2400 may include columns other than the article position column and the registration time column. For example, the non-product display column of the management table 2400 may store a column of position within the article and a column of parent ID, similar to the management table 1300 of the first embodiment. In the first embodiment, an example is given in which the position within the article is a position whose origin is a predetermined position (vertex 203, 204) on the thick plate. A mark indicating the origin may be displayed at a predetermined position on the surface of the position-fixed article. For example, paint is used to display the mark. Further, a straight line parallel to the axis of symmetry (central axis) of the position-fixing article may be displayed on the surface of the position-fixing article. For example, paint is used to display the straight line. In this case, the first imaging device captures a first image that includes the ID display area of the position-fixed article, the mark, and the straight line. In this case, the position where the mark is displayed becomes the position within the article. Further, the straight line corresponds to the y-axis shown in FIGS. 14 and 15. Note that straight lines corresponding to the x-axis shown in FIG. 14 and the z-axis shown in FIG. 15 may be displayed on the surface of the position-fixed article. Further, for the tube, the distance from the origin on the x-axis shown in FIG. 14 and the z-axis shown in FIG. 15 is determined in consideration of the curvature according to the outer diameter of the tube.

Furthermore, the non-product display column of the management table 2400 may include the article direction column described in <<Fifth Modification>> of the first embodiment. Note that the tube does not have a plate surface. Therefore, for example, the angle between the axis of symmetry (central axis) of the tube and the reference plane is stored in the article orientation column. In this case, for example, the ID 2302 indicates that the direction in which the plurality of components ("4312" in the example shown in FIG. ) is displayed on the surface of the tube 2301a.

In addition, when the ID of a position-fixed article and the three-dimensional position of the position-fixed article are specified from the design drawing, a column for the assumed article position is included in the non-actual item display column of the management table 2400 in addition to the article position. You may be In the article assumed position column, the three-dimensional position of the position-fixed article specified from the design drawing is stored.

In the first embodiment, a first image acquisition unit 1201, recognition unit 1202, acquisition unit 1203, matching unit 1204, identification unit 1205, position derivation unit 1206, storage unit 1207, output unit 1208, and second image acquisition unit A unit 1209 performs processing using a display area using a stencil (label). In contrast, in this embodiment, the first image acquisition unit 1201, the recognition unit 1202, the acquisition unit 1203, the matching unit 1204, the identification unit 1205, the position derivation unit 1206, the storage unit 1207, the output unit 1208, and the second image The acquisition unit 1209 performs processing using the structural material ID display area and the pipe ID display area.

Therefore, in this embodiment, in step S2505 of FIG. 25, the specifying unit 1205 specifies one position-fixed article from the position-fixed articles managed in the management table 2300, based on the result of the collation in step S2504.

Furthermore, in this embodiment, the position derivation unit 1206 derives the three-dimensional position of the position-fixed article installed in the installation area. Therefore, for example, if it is assumed in advance from the design drawing which position-fixing article will be installed at which position, the determination in step S2508 in FIG. 25 may not be performed. In this case, the article management device 1200 does not need to include the second image acquisition unit 1209.

Further, in step S2509, the position deriving unit 1206 uses a mathematical formula expressing the geometric positional relationship between the ID display area of the position-fixed article and the first imaging device, as in the first embodiment. , the three-dimensional position of the position-fixed article may be derived. Note that the mathematical expression representing the geometrical positional relationship between the ID display area of the tube and the first imaging device is expressed using the curvature of the surface of the tube. Furthermore, the ID display area of the position-fixed article and the first A three-dimensional positional relationship with the imaging device may be derived. Note that, as described in the first embodiment, when deriving the three-dimensional positional relationship between the ID display area of the tube and the first imaging device, as information on the ID display area of the position-fixed article, The shape and dimensions of the ID display area of the tube and the curvature of the surface of the tube are used. Furthermore, the three-dimensional position of the position-fixed article may be derived using the brightness image and the distance image.

The position derivation unit 1206 also calculates the difference between the three-dimensional position of the fixed article derived as described above and the three-dimensional position stored in the assumed article position column of the non-actual item display column of the management table. It may be determined whether or not is within a predetermined range. In this case, the difference between the three-dimensional position of the fixed article derived as described above and the three-dimensional position stored in the assumed article position column of the non-actual item display column of the management table is within a predetermined range. If not, the process of step S2512 may be performed.

Additionally, the position derivation unit 1206 may perform the following processing in step S2509. That is, the position deriving unit 1206 calculates the three-dimensional position stored in the column corresponding to the position-fixed article specified in step S2505 in the article assumed position column in the non-actual product display column of the management table in step S2505. It may also be derived as the three-dimensional position of the position-fixed article specified in . However, if the three-dimensional position of the position-fixed article is derived in this way, if the position-fixed article is not installed at the designed position, the correct three-dimensional position of the position-fixed article will not be derived. There are cases. Therefore, when the three-dimensional position of the position-fixed article is derived in this way, the worker (operator) calculates the three-dimensional position of the position-fixed article based on the information indicating the three-dimensional position of the position-fixed article output in S2511. It is preferable to confirm whether the position fixing article is installed in its original position.

Furthermore, in this embodiment, the position derivation unit 1206 derives the three-dimensional position of the position-fixed article installed in the installation area. Therefore, after the three-dimensional position of the position-fixed article is derived by the position derivation unit 1206 in step S2509, the three-dimensional position of the position-fixed article does not necessarily need to be updated. This is indicated by the fact that "update" shown in steps S1710 and S1713 in FIG. 17 is changed to "storage" in steps S2510 and S2513 in FIG. 25. However, after the position derivation unit 1206 derives the three-dimensional position of the position-fixed article in step S2509, the three-dimensional position of the position-fixed article may be updated. For example, based on the information indicating the three-dimensional position of the position-fixed article output in step S2511, if the three-dimensional position is clearly different from the actual three-dimensional position, the worker (operator) The article management device 1200 may be instructed to derive the three-dimensional position again. In this case, the process according to the flowchart of FIG. 25 is executed again, and the three-dimensional position of the position-fixed article can be updated in step S2510.

As described above, in the present embodiment, the article management device 1200 stores information displayed on a position-fixed article installed on the ground or underground, and the ID managed by the manufacturer of the position-fixed article. A first captured image including the displayed display area is acquired. The article management device 1200 determines the three-dimensional position of the ID display area of the position-fixed article based on the acquired first captured image and the position of the first imaging device that captured the first captured image. Information indicating the three-dimensional position is stored in a storage medium in association with the ID of the position-fixed article. Thus, for example, which position-fixing articles are in which positions, even if it becomes practically impossible to check the information displayed on the position-fixing articles after the position-fixing articles have been installed. can be managed. Therefore, the position of the position fixing article installed in the installation area can be appropriately managed.
Note that the articles installed in the installation area are not limited to the structural material shown in FIG. 22 and the pipe shown in FIG. 23. In addition, various modifications described in the first embodiment may be adopted in this embodiment as well.

(Other variations)
Note that the embodiments of the present invention described above can be realized by a computer executing a program. Furthermore, a computer-readable recording medium on which the program is recorded and a computer program product such as the program can also be applied as an embodiment of the present invention. As the recording medium, for example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, magnetic tape, nonvolatile memory card, ROM, etc. can be used. Further, embodiments of the present invention may be realized by a PLC (Programmable Logic Controller) or by dedicated hardware such as an ASIC (Application Specific Integrated Circuit).
Furthermore, the embodiments of the present invention described above are merely examples of implementation of the present invention, and the technical scope of the present invention should not be construed as limited by these. It is something. That is, the present invention can be implemented in various forms without departing from its technical idea or its main features.

(hardware)
An example of the hardware of the article management device 1200 will be explained. In FIG. 26, the article management device 1200 includes a CPU 2601, a main storage device 2602, an auxiliary storage device 2603, a communication circuit 2604, a signal processing circuit 2605, an image processing circuit 2606, an I/F circuit 2607, a user interface 2608, a display 2609, and It has a bus 2610.

The CPU 2601 centrally controls the entire article management device 1200. The CPU 2601 uses the main storage device 2602 as a work area to execute programs stored in the auxiliary storage device 2603. Main storage device 2602 temporarily stores data. The auxiliary storage device 2603 stores various data in addition to programs executed by the CPU 2601. Note that a processor other than the CPU (eg, GPU) may be used.

The communication circuit 2604 is a circuit for communicating with the outside of the article management device 1200. The communication circuit 2604 may perform wireless communication or wired communication with the outside of the article management device 1200.

The signal processing circuit 2605 performs various signal processing on the signals received by the communication circuit 2604 and the signals input under the control of the CPU 2601.
The image processing circuit 2606 performs various image processing on the input signal under the control of the CPU 2601. The signal subjected to this image processing is output to, for example, a display 2609.
The user interface 2608 is a part through which an operator gives instructions to the article management device 1200. User interface 2608 includes, for example, buttons, switches, dials, and the like. Further, the user interface 2608 may include a graphical user interface using a display 2609.

Display 2609 displays an image based on the signal output from image processing circuit 2606. The I/F circuit 2607 exchanges data with devices connected to the I/F circuit 2607. FIG. 26 shows a user interface 2608 and a display 2609 as devices connected to the I/F circuit 2607. However, the devices connected to the I/F circuit 2607 are not limited to these. For example, a portable storage medium may be connected to the I/F circuit 2607. Furthermore, at least a portion of the user interface 2608 and the display 2609 may be located outside the article management device 1200.
The output unit 1208 is realized, for example, by using at least one of the communication circuit 2604, the signal processing circuit 2605, the image processing circuit 2606, the I/F circuit 2607, and the display 2609.

Note that the CPU 2601, main storage device 2602, auxiliary storage device 2603, signal processing circuit 2605, image processing circuit 2606, and I/F circuit 2607 are connected to the bus 2610. Communication between these components occurs via bus 2610. Further, the hardware of the article management device 1200 is not limited to that shown in FIG. 26 as long as it can realize the functions of the article management device 1200 described above.

The present invention can be used, for example, to manage the location of articles.

Claims (11)

1. An article management device that manages the location of articles,
   one or more image acquisition means for acquiring a captured image of an area including display information displayed on the article;
   identification information acquisition means for acquiring identification information of the article;
   position derivation means for deriving an information display position that is a three-dimensional position where the display information is displayed based on the captured image acquired by the image acquisition means;
   It is characterized by comprising a storage means for storing the information display position on the article derived by the position derivation means in a storage medium in association with the identification information of the article acquired by the identification information acquisition means. Goods management device.
2. The display information displayed on the article includes identification information of the article,
   The identification information acquisition means includes:
   recognition means for recognizing identification information of the article based on the captured image acquired by the image acquisition means;
   The article management apparatus according to claim 1, further comprising at least one means for acquiring identification information of the article stored in the storage means.
3. The information display position stored in the storage medium in association with the identification information of the article is information regarding the three-dimensional position where the display information is displayed. Contains information about location;
   3. The article management device according to claim 2, wherein the captured image acquired by the image acquisition means includes an area including a predetermined position of the article.
4. The identification information acquisition means includes:
   The recognition means;
   the acquisition means;
   collation means for collating the identification information of the article recognized by the recognition means and the identification information of the article obtained by the acquisition means;
   Claim 2 or 3, further comprising a specifying means for specifying identification information of the article included in the captured image acquired by the image acquiring means based on a result of the collation by the collating means. The article management device described in .
5. The position derivation means derives the position in the height direction of the surface of the article on which the display information is displayed, based on two or more captured images acquired by the two or more image acquisition means. The article management device according to any one of claims 1 to 4.
6. The storage means stores the information display position of the article derived by the position derivation means in the storage medium as the current position of the article, and records the date and time when the article was placed at the current position. The article management device according to any one of claims 1 to 5, wherein the information is stored in the storage medium in association with the information display position on the article.
7. The article management device manages the position of an article whose existing position changes,
   When the information display position on the article and the identification information of the article are already stored in the storage medium in association with each other, the storage means causes the position derivation means to display the information on the article. When a position is newly derived, the information display position already stored in the storage medium is updated to the information display position newly derived by the position derivation means. 6. The article management device according to any one of Item 6.
8. The article management device according to any one of claims 1 to 6, wherein the article management device manages the positions of articles installed in the installation area.
9. An article management method for managing the location of articles, the method comprising:
   one or more image acquisition steps of acquiring captured images of a region including display information displayed on the article;
   an identification information acquisition step of acquiring identification information of the article;
   a position derivation step of deriving an information display position, which is a three-dimensional position where the display information is displayed, based on the captured image acquired in the image acquisition step;
   A storage step of storing the information display position on the article derived in the position derivation step in a storage medium in association with identification information of the article acquired in the identification information acquisition step. How to manage goods.
10. display information displayed on the goods;
    and has hidden information that is not displayed on the article,
    The hidden information includes an information display position that is a three-dimensional position where the display information is displayed,
    A data structure of article management data used by an article management device to derive the information display position on the article based on a captured image of a region including display information displayed on the article.
11. A program for causing a computer to execute processing for managing the location of articles,
    one or more image acquisition steps of acquiring captured images of a region including display information displayed on the article;
    an identification information acquisition step of acquiring identification information of the article;
    a position derivation step of deriving an information display position, which is a three-dimensional position where the display information is displayed, based on the captured image acquired in the image acquisition step;
    causing a computer to execute a storing step of storing the information display position on the article derived in the position deriving step in a storage medium in association with the identification information of the article acquired in the identification information acquiring step; Featured programs.

Images