JP2022135731A - Product management system, product management method, and product management program - Google Patents

Abstract

To perform a product management system that differentiates loading from unloading while reducing installation expenses and installation space for product management using an RFID.SOLUTION: The product management system includes: a gate section that has an RFID reader function that reads the identification information of a product from an RF tag attached to the product; a photographing section that photographs around the gate part; and a management section that manages a product passing the gate section based on the data obtained by photographing with the photographing section and the identification information read by the gate section. The management section includes: an image processing unit that based on the image data obtained by photographing with the photographing section, performs a transportation direction determination process for determining a transportation direction that indicates whether the transported product has been loaded or unloaded and a quantity determination process for determining the number of transported products; and an excess and shortage determination unit that compares the quantity of the identification information read by the gate section with the number of transported products determined by the image processing unit and determines the excess or shortage of the transported products for each transportation direction.SELECTED DRAWING: Figure 4

Description

The present invention relates to a product management system, a product management method, and a product management program.

In recent years, product management using RFID (Radio-Frequency Identification) is increasing. RFID is a technology for contactlessly reading an RF tag attached to a product from a dedicated reader. Product management by RFID is used, for example, for management of incoming/outgoing products to/from a warehouse (incoming/outgoing management). Warehousing and shipping management is performed, for example, by installing a gate having an RF reader function through which products pass, and reading the RF tag of the product when passing through the gate.

Techniques related to product management using RFID are disclosed in the following prior art documents.

JP 2012-108777 A JP-A-2006-103812

However, if an attempt is made to separately manage entry and exit using RFID, it is necessary to install both an entry gate and an exit gate. Since the gate allows a plurality of products to pass through, the size of the gate is appropriate, and a large space may be required to install a plurality of gates. In addition, since the gate is large and has an RF reader function, it is expensive.

Therefore, in product management using RFID, a product management system, a product management method, and a product management program are provided that are capable of performing product management that distinguishes between incoming and outgoing shipments while suppressing installation costs and installation space.

According to one embodiment of the present invention, a gate section having an RFID reader function for reading product identification information from an RF tag attached to a product, a photographing section for photographing the periphery of the gate section, and the photographing section and a management unit that manages products passing through the gate based on the data photographed by and the identification information read by the gate, wherein the management unit is photographed by the photographing unit An image for executing, based on image data, a transport direction determination process for determining a transport direction indicating whether a product to be transported is incoming or outgoing, and a number determination process for determining the number of products to be transported. comparing the number of pieces of identification information read by the processing unit and the gate unit with the number of the products to be conveyed determined by the image processing unit, and determining excess or deficiency of the products to be conveyed for each of the conveying directions; and an excess/deficiency determination unit for determination.

According to one embodiment of the present invention described above, it is possible to manage products by distinguishing between incoming and outgoing, while suppressing installation costs and installation space.

FIG. 1 is a diagram showing a configuration example of a product management system 1. As shown in FIG. FIG. 2 is a diagram showing an example of the software configuration of the product management system 1. As shown in FIG. FIG. 3 is a diagram showing a configuration example of the product management system 1. As shown in FIG. FIG. 4 is a diagram showing an example of the sequence of product management processing. FIG. 5 is a diagram showing an example of a processing flowchart of the product management processing S100. FIG. 6 is a diagram showing an example of a processing flowchart of the excess/deficiency determination processing S30. FIG. 7 is a diagram showing an example of a processing flowchart of the excess/deficiency determination processing S60 in the second embodiment. FIG. 8 is a diagram showing an example of an entry/exit list and determination results. FIG. 9 is a diagram showing an example of a processing flowchart of the product management processing S200 in the third embodiment. FIG. 10 is a diagram showing an example of a processing flowchart of the learning model pre-learning processing for image analysis S300. FIG. 11 is a diagram showing an example of processed image data.

DETAILED DESCRIPTION OF THE INVENTION Embodiments for carrying out the invention will be described below.

<Configuration example of product management system 1>
FIG. 1 is a diagram showing a configuration example of a product management system 1. As shown in FIG. The product management system 1 has a management device 100 , a gate 200 and a camera 300 . The product management system 1 is a system that manages the storage and delivery of products that pass through the gate 200 . A product is determined to be out of the warehouse if it is conveyed through the gate 200 from the right to the left (D1), and that it is in the warehouse if it is conveyed in the opposite direction (D2).

The products T1 to T3 are packed, for example, in corrugated cardboard, and RF tags R1 to R3 are attached to the outer surface of the packing, respectively.

An RF tag has an IC (Integrated Circuit) chip that stores information about a product. The information about the product includes, for example, the identifier of the product. RF tags are, for example, passive tabs. A passive tab is a tab that is not equipped with a battery and uses radio waves transmitted from an RFID reader as a power source to transmit radio waves to the RFID reader. Also, the RF tag may be an active tab or semi-active tab that transmits radio waves using its own battery as a power source.

The gate 200 is a device that has an RFID reader function and reads the RF tags of products that pass through the gate. For example, the gate 200 transmits radio waves for reading the RF tag and receives radio waves returned from the RF tag, thereby recognizing the presence of the product in the vicinity (within a predetermined distance). The timing at which the gate 200 transmits radio waves is, for example, always, at a predetermined interval, or at timing instructed by an administrator's operation or another device.

The camera 300 is a photographing device that photographs an image within a predetermined range of the gate 200 and its surroundings. A video is, for example, one or both of a moving image and a still image. The shooting timing of the camera 300 is constant, a predetermined interval, or a timing instructed by an administrator's operation or another device. Also, the camera may be a camera with a motion function (a function capable of determining the movement direction of a moving article).

The management device 100 is a device that manages products transported (stored and shipped) through the gate 200, and is, for example, a computer or a server machine. The management device 100 acquires the image data of the video captured by the camera 300 . The management device 100 performs image analysis on the image data, and determines the number of products, the direction in which the products are conveyed (incoming or outgoing), and the like.

Also, the management device 100 acquires the RFID read by the gate 200 in a series of processes or within a predetermined period of time. Then, using the determination result of the image analysis and the readout result of the RFID, the warehousing and shipping of products are managed.

The management device 100, the gate 200, and the camera 300 are connected via a network or cable, for example, and communicate with each other. The management device 100 communicates with the gate 200 and the camera 300 to acquire image data and RFID readout results.

In FIG. 1, the management device 100, the gate 200, and the camera 300 are different devices, but all or any two devices may be the same device.

<Software configuration example of product management system>
FIG. 2 is a diagram showing an example of the software configuration of the product management system 1. As shown in FIG.

The camera 300 has an imaging section 30 . The photographing unit 30 photographs a predetermined range, stores the photographed image data in an internal memory, or transfers the photographed image data to the image processing unit 10 of the management device 100 .

A gate (gate section) 200 has an RFID reading section 20 . The RFID reader 20 transmits radio waves for reading the RF tag in response to a read trigger, and receives radio waves returned from the RF tag in response to the transmitted radio waves. Then, the RFID reading unit 20 converts information (for example, the number of read RFIDs and information about products) about the RFIDs successfully read from the RF tags included in the received radio waves into the product management data of the management device 100 . Hand over to Section 14.

A management device (management section) 100 has an image processing section 10 and a product management section 14 . The image processing unit 10 acquires image data captured by the camera 300 and performs image analysis. The product management unit 14 manages the warehousing and shipping of products based on the results of image analysis and RFID reading.

The image processing unit 10 has an image analysis learning model learning unit 11 , a number determination unit 12 , and a transport direction determination unit 13 .

The image analysis learning model learning unit 11 learns an image analysis learning model for performing image analysis. The image analysis model is, for example, AI (Artificial Intelligence) for image analysis. The image analysis learning model learning unit 11 improves the determination accuracy of the trained image analysis learning model during operation of the product management system 1 or pre-learns the image analysis learning model before operation of the product management system 1. .

The number determination unit 12 uses the learning model for image analysis to determine the number of products appearing in the image data. The number determination unit 12 hands over the determination result of the number to the product management unit 14 .

The conveying direction determination unit 13 determines the conveying direction of the product (incoming or outgoing). The transport direction determination unit 13 transfers the transport direction determination result to the product management unit 14 .

The product management unit 14 has an entry/exit list acquisition unit 15 , an excess/deficiency determination unit 16 , and a learning data creation unit 17 .

The incoming/outgoing list acquisition unit 15 acquires an incoming/outgoing list of information on products to be entered and delivered. The entry/exit list is obtained, for example, from a core system connected to the management device 100 .

The excess/deficiency determination unit 16 determines whether or not there is an excess/deficiency in the products to be conveyed at the time of warehousing and shipping. The excess/deficiency determination unit 16 performs excess/deficiency determination based on at least the number of products determined by the number determination unit 12 and the number of products whose RFIDs have been successfully read by the RFID reading unit 20 .

The learning data creating unit 17 creates teacher data (learning data) for learning an image analysis learning model from image data captured during operation of the product management system 1 and the result of excess/deficiency determination.

Note that each unit may be configured by, for example, dedicated hardware, or may be constructed by a processor executing a loaded program.

<Hardware configuration example of product management system>
FIG. 3 is a diagram showing a configuration example of the product management system 1. As shown in FIG. The product management system 1 has a CPU (Central Processing Unit) 110 , a storage 120 , a memory 130 , a camera 300 , a gate 200 and a display 160 .

The storage 120 is an auxiliary storage device such as flash memory, HDD (Hard Disk Drive), or SSD (Solid State Drive) that stores programs and data. The storage 120 stores an RFID reading program 121, a photographing program 122, an image processing program 123, a product management program 124, and a learning model learning program 125 for image analysis.

Memory 130 is an area into which programs stored in storage 120 are loaded. The memory 130 may also be used as an area where programs store data.

The CPU 110 is a processor that loads a program stored in the storage 120 into the memory 130, executes the loaded program, constructs each section, and executes each process.

The camera 300 is a device for photographing products around the gate 200 . The camera 300 may be connected to the outside of the computer that configures the product management system 1, or may be built in the computer, for example.

The gate 200 is a device that has an RFID reader function and uses the RFID reader function to receive radio waves emitted (returned) by an RF tag attached to a product to obtain information (product information) on the RF tag. . For example, the gate 200 may be connected to the outside of the computer that constitutes the product management system 1, or may be built in the computer. In addition, the gate 200 is, for example, a gate type, and RFID is read from RF tags of products located within the gate or within a predetermined distance from the gate 200 .

The display 160 is a display unit that displays images and moving images. The display 160 is, for example, liquid crystal or organic EL (Electro Luminescence). The display 160 displays, for example, the result of excess/deficiency determination.

By executing the RFID reading program 121, the CPU 110 constructs the RFID reading unit 20 and performs RFID reading processing. The RFID readout process uses the RFID readout function of the gate 200 to read the product information from the RF tag.

By executing the imaging program 122, the CPU 110 constructs the imaging unit 30 and performs imaging processing. The shooting process is a process of shooting an image including the area around the gate 200 using the camera 300 .

The CPU 110 executes the image processing program 123 to construct the image processing unit 10 and perform image processing. Image processing is a process of analyzing image data and determining the number of products appearing in the image data and the carrying-in direction (warehousing or shipping) of the products.

By executing the product management program 124, the CPU 110 constructs the product management unit 14 and performs product management processing. The product management process is a process of judging excess/deficiency of products in warehousing and delivery.

By executing the image analysis learning model learning program 125, the CPU 110 constructs the image analysis learning model learning unit 11 and the learning data creation unit 17, and performs image analysis learning model learning processing. The image analysis learning model learning process performs processing related to learning of the image analysis learning model, such as selection of teacher data to be learned by the image analysis learning model, input of teacher data, and the like. Note that the image analysis learning model learning process may include prior learning of the image analysis learning model before the product management system 1 is operated.

<Product management processing sequence>
FIG. 4 is a diagram showing an example of the sequence of product management processing. Processing of each unit will be described below.

The photographing unit 30 photographs the area around the gate 200 (including the product), and transmits the photographed image data to the image processing unit 10 (S10).

The image processing unit 10 uses the image analysis learning model 50 (S11), executes the number determination process S12, and transmits the execution result to the product management unit 14 (S13). The image analysis learning model 50 is, for example, AI (Artificial Intelligence).

The number determination process S12 is a process for determining the number of products appearing in the image data. The execution result of the number determination process S12 includes the number of products (hereinafter sometimes referred to as the number of images). The number determination process S12 determines the number of products that are moving (conveyed). Products that are not moved (products that stay in the same position) are not considered products that are being transported and are not counted in the number of products that are being transported.

In addition, the image processing unit 10 acquires moving direction information regarding the moving direction of the product from the camera with motion function (S14), executes the conveying direction determination processing S15, and transmits the execution result to the product management unit 14 (S16). ).

The transport direction determination process S15 is a process for determining the transport direction of the product (whether it is stored or unloaded) based on the movement direction information. The execution result of the transport direction determination process S15 includes information indicating that the product is entering or leaving the warehouse (hereinafter sometimes referred to as the image transport direction).

The RFID reading unit 20 reads the RFID and transmits the read result to the product management unit 14 (S20). The read result includes the number of successfully read RFIDs (hereinafter sometimes referred to as the number of readouts) and product information included in the read RFIDs.

When the product management unit 14 receives the number of images, the number of readout images, and the image conveying direction, the product management unit 14 performs an excess/deficiency determination process S30 and a learning data creation process S40.

The excess/deficiency determination process S30 is a process for determining whether or not there is an excess/deficiency in the products to be transported for each incoming and outgoing warehousing, based on the number of images, the number of readout images, and the direction of image transport. Moreover, the excess/deficiency determination process S30 may include a process of displaying the result of the excess/deficiency determination on the display 160 .

The learning data creation process S40 is a process of using the image data used for image analysis by the image analysis learning model 50 and the result of correctness or wrongness of the image analysis of the image data as teacher data (creating teacher data). be. The learning data creation process S40 transmits teacher data to the image analysis learning model learning process S42 (S41). Note that the teacher data created in the learning data creating process S40 relates to determination of the number of images. When the moving direction is determined by a camera with a motion function, teacher data regarding transport direction determination is not created in the learning data creating process S40.

The image analysis learning model learning process S42 is a process of inputting teacher data, learning the image analysis learning model 50, and updating the image analysis learning model 50 (S43). The learning timing may be, for example, during operation of the product management system 1 or while the product management system 1 is stopped (nighttime, system maintenance time, etc.). In the product management system 1, the image data subjected to the image analysis is used, and the image analysis learning model 50 is repeatedly trained to improve the accuracy of the image analysis (percentage of correct execution results of the number determination processing S12). .

When images are captured by a camera that does not have a motion function, for example, the image analysis learning model 50 may be used to determine the direction in which the product moves and whether the product is in or out of the warehouse. In this case, learning data may also be generated and learned for the conveying direction determination processing S15.

Furthermore, learning of the image analysis learning model 50 during system operation may not be performed. Examples of not learning during operation include cases where sufficient pre-learning has been performed and AI accuracy is considerably high, and cases where learning during operation is difficult due to the processing load and communication speed. Further, the system configuration does not necessarily have to have a learning function during operation.

[First embodiment]
A first embodiment will be described. In the first embodiment, moving image data is used as shooting data. The management device 100 detects the movement of the product based on the moving image data, and executes product management processing with the detection of the movement of the product as a trigger.

FIG. 5 is a diagram showing an example of a processing flowchart of the product management processing S100 in the first embodiment.

The management device 100 analyzes the captured data of the moving image and waits for the detection of the movement of the product (that the product is about to be carried in or out) (No in S100-1). When the management device 100 detects that the product has started to move (Yes in S100-1), the management device 100 executes the number determination process S12.

In the number determination process S12, the management device 100 determines the number of product images from a still image at a certain moment in the moving image data. The number determination process S12 is, for example, a process of determining the number of images based on a plurality of still images at a plurality of timings. Also, the number determination process S12 may be, for example, a process of determining the number of images using video data captured from multiple directions using a plurality of cameras 300 . Furthermore, when the camera 300 has a depth measurement function, the number determination process S12 may include a process of using the depth measurement result to determine the number of images. The depth measurement function is a function of detecting the distance to an object, and is a function that makes it possible to determine, for example, whether an area surrounded by a plurality of cardboard boxes is a space or a cardboard box.

The management device 100 executes the transport direction determination process S15. In the conveying direction determination process S15, the management device 100 determines whether the conveying direction of the product is warehousing or warehousing according to the information about the moving direction acquired from the camera. The transport direction determination process S15 is a process for determining whether the product is in or out of the warehouse from the moving direction acquired by the motion function of the camera.

The management device 100 waits until the reading of the RFID is completed (No in S100-2). RFID reading continues until a predetermined time elapses after one RFID is read, for example. Continuing to read the RFID for a predetermined period of time may require a certain amount of time to successfully read the RI tags of multiple products, depending on, for example, the size of the product itself and the radio wave conditions near the gate 200. This is because of the nature of

When the reading of the RFID is completed (Yes in S100-2), the management device 100 executes the excess/deficiency determination process S30. The details of the excess/deficiency determination process S30 will be described later.

In the excess/deficiency determination process S30, if there is an excess/deficiency in the product (Yes in S100-3), the management device 100 confirms the product and responds to the excess/deficiency (S100-4). In the process S100-4, for example, the mismatch between the number of images and the number of readouts and the image transfer direction such as warehousing or warehousing are displayed on the display 160 or the like so that the administrator can confirm or respond to excess or deficiency of the product. including prompting to

The management device 100 creates teacher data from the image data and the excess/deficiency determination result, executes the learning model learning process for image analysis S42, and again waits for detection of movement of the product (S100-1). Note that the teacher data is created, for example, in the learning data creating process S40.

<Excess/shortage determination process>
FIG. 6 is a diagram showing an example of a processing flowchart of the excess/deficiency determination processing S30. The management device 100 determines whether or not the number of images matches the number of readouts (S30-1).

When the number of images and the number of readouts match (Yes in S30-1), the management device 100 determines that there is no shortage or excess of products in shipping if the product is shipped in the image conveying direction (“Delivery” in S30-2). (S30-3), and if the image transport direction is warehousing ("warehousing" of S30-2), it is determined that there is no shortage or excess of products in the warehousing (S30-4), and the process ends.

On the other hand, if the number of images does not match the number of readouts (No in S30-1), the management device 100 determines whether there is an excess or deficiency in the products in the delivery if the product is delivered in the image conveying direction (“Delivery” in S30-5). (S30-7), and if the image transport direction is warehousing ("warehousing" of S30-5), it is determined that there is an excess or deficiency of products in the warehousing (S30-6), and the process ends. .

In the first embodiment, when the movement (conveyance) of the product is detected from the moving image data, the management device 100 determines the number of images and the image conveying direction from the moving image data. Then, the management device 100 acquires the read result of the RFID, compares the number of readouts and the number of images, and determines whether or not there is an excess or deficiency of products for each image conveying direction. As a result, by installing one gate 200, it is possible to determine the excess or deficiency of products in both directions of warehousing and warehousing, thereby enabling product management. In addition, since the management apparatus 100 determines whether or not the item is in or out of the warehouse based on the image, it is possible to reduce the time and effort of the administrator to input the information of whether or not the item is in or out of the warehouse, or to reduce the risk of inputting the information incorrectly.

[Second embodiment]
A second embodiment will be described. In the second embodiment, the management device 100 executes an excess/deficiency determination process S60 instead of the excess/deficiency determination process S30 in the first embodiment. The excess/deficiency determination process S60 is a process of using a storage/delivery list (product list) including a list of products to be stored and delivered for excess/deficiency determination. The warehousing/delivery list includes, for example, information such as the conveying direction, identification information including identifiers of products to be conveyed, and the number of products to be conveyed (quantity information).

FIG. 7 is a diagram showing an example of a processing flowchart of the excess/deficiency determination processing S60 in the second embodiment. The management device 100 acquires the entry/exit list (S60-1). The management device 100, for example, acquires an entry/exit list from a core system that creates and manages the entry/exit list. Also, the management device 100 may acquire the incoming/outgoing goods list, for example, by the input of the administrator.

The management device 100 determines whether or not the conveying direction of the incoming/outgoing list matches the image conveying direction (S60-2). If the transport direction of the incoming/outgoing list does not match the image transport direction (No in S60-2), the management device 100 determines that there is an excess or deficiency in the product (S60-5), and terminates the process. Note that the determination that there is an excess or deficiency in the product includes the determination that an error other than the excess or deficiency of the product has occurred, such as an error in the conveying direction or the movement of excess products.

On the other hand, if the conveying direction of the entry/exit list matches the image conveying direction (Yes in S60-2), the management device 100 determines that the number of images, the number of readouts, and the number of conveyances of the entry/exit list all match. It is determined whether or not there is (S60-3).

When the number of images, the number of readouts, and the number of conveyances in the storage/delivery list all match (Yes in S60-3), the management device 100 determines that there is no excess or deficiency in the product (S60-4), and processes the exit.

On the other hand, if the number of images, the number of readouts, and the number of transported items in the storage/delivery list partially do not match (No in S60-3), the management device 100 determines that there is an excess or deficiency in the product (S60-5), End the process.

FIG. 8 is a diagram showing an example of an entry/exit list and determination results. FIG. 8A is a diagram showing an example of a case where it is determined that there is no excess or deficiency. The management device 100 determines that the transport directions match because the transport direction of the storage/delivery list L1 is storage receipt and the image transport direction is also storage storage (Table 1, Yes in S60-2 of FIG. 7). The management device 100 determines that the number of transported items, the number of images, and the number of images to be read out in the storage/delivery list L1 is 7, which are all the same (Yes in S60-3 of Table 2 and FIG. 7). It is determined that there is no excess or deficiency (S60-4 in FIG. 7).

FIG. 8B is a diagram showing an example of a case in which it is determined that there is excess or deficiency. The management device 100 determines that the transport direction of the storage/delivery list L1 is warehousing, and the image transport direction is also warehousing, so that the transport directions match (Yes in S60-2 of Table 3 and FIG. 7). The management device 100 determines that the number of transported items in the storage/delivery list L1 is 7, the number of images is 6, and the number of readouts is 6, and all of them do not match (No in S60-3 in Table 4 and FIG. 7). It is determined that there is an excess or deficiency (S60-5 in FIG. 7).

<Modification>
In the excess/deficiency determination process S60, the management device 100 may use the identifier (ID) of the transported product in the receipt/dispatch list in place of the number of items to be transported in the receipt/dispatch list. In this case, the management device 100 confirms whether or not the identifiers of the transported products in the incoming/outgoing list and the identifiers of the products in the read results of the RFID match (match exactly). When the identifiers all match, the management device 100 further determines whether or not the number of images matches the number of transported items (same as the read number) in the incoming/outgoing inventory list. The management device 100 determines that there is no excess or deficiency in the product if the identifier and the number of conveyed items all match, and that there is an excess or deficiency in the product if even a portion of the identifiers do not match.

Also, the comparison with the entry/exit list may be performed only at the time of delivery. For warehousing, since it may be unclear when and how many products will be brought in, there are cases where a warehousing/dispatching list is not created for warehousing. On the other hand, shipping is performed at the timing of delivering the product to the customer, for example, so the timing and target product to be carried out are clear and strictly managed, so they are often listed.

Furthermore, the excess/deficiency determination process may be performed only at the time of leaving the warehouse. This is because, as described above, stricter product management is required at the time of delivery. For example, in the transport direction determination process S15 in the product management process S100 of FIG. good too.

[Third embodiment]
A third embodiment will be described. In the third embodiment, still image data is used as shooting data. When the management device 100 detects the readout of the RFID, the management device 100 determines the transport direction of the product based on the still image data captured at least at two timings, and executes product management.

FIG. 9 is a diagram showing an example of a processing flowchart of the product management processing S200 in the third embodiment.

The management device 100 waits for detection of RFID reading (No in S200-1). When the management device 100 detects the readout of the RFID (Yes in S200-1), it acquires photographed data (still image data) photographed at a plurality of timings (S200-2).

The management device 100 executes the number determination process S12 based on the photographed data. In the number determination process S12, the management device 100 may determine the number of images from one image data having a plurality of image data, or may determine the number of images comprehensively from a plurality of image data.

The management device 100 executes the transport direction determination process S15. In the transport direction determination process S15, the management device 100 determines whether the transport direction of the product is warehousing or warehousing based on the still image data. The transport direction determination processing S15 determines the moving direction of the product by grasping the position in the image data of the photographed product in chronological order.

The management device 100 executes an excess/deficiency determination process S30. In the excess/deficiency determination process S30, if there is an excess/deficiency in the product (Yes in S200-3), the management device 100 confirms the product and responds to the excess/deficiency (S200-4).

The management device 100 creates teacher data from the image data and the excess/deficiency determination result, executes the learning model learning process for image analysis S42, and waits for RFID reading detection again (S200-1).

In the first embodiment, the management device 100 executes the product management process triggered by the detection of the movement of the product by the image analysis of the moving image. However, when the image to be shot is a still image, it may not be possible to detect the movement of the product in real time like a moving image. Therefore, in the third embodiment, the management device 100 detects the movement of the product by capturing (continuously capturing) still images when the RFID is read out, without constantly capturing still images. can do.

[Fourth embodiment]
A fourth embodiment will be described. In the fourth embodiment, a method of learning a learning model for image analysis will be described.

<Learning processing for learning model for image analysis>
The image analysis learning model learning process S42 will be described. The image analysis learning model learning process S42 is a process executed in the following two patterns.

The first pattern is a pattern in which learning is performed using captured image data during operation of the product management system 1 . For example, the number of images and the number of readouts may differ. In this case, it is conceivable that an error occurred in the reading of the RFID, such as an omission of the reading of the RFID, or erroneous reading of the RFID of a product other than the target product. On the other hand, in this case, it is conceivable that the image analysis learning model erroneously determined the number of images. Here, for example, assuming that the possibility of an error in which an RFID reads an RFID other than a product is considerably low, if the number of readouts is greater than the number of images, there is a high possibility that the determination of the number of images is erroneous. . Therefore, if the number of readouts is greater than the number of images, the management device 100 assumes that the number of readouts is correct. A learning model learning process for image analysis is performed as learning data (teacher data).

When the learning model for image analysis determines the image transport direction using an image captured by a camera that does not have a motion function, the management device 100 learns the image transport direction (product movement direction). good too. The direction of image transport is determined from the direction of travel of the product. Therefore, if the determination of the movement direction is erroneous, the image transport direction is also erroneously determined. Therefore, the management device 100 learns the moving direction of the product as learning of the image conveying direction. Note that when a camera with a motion function is used, the management device 100 acquires determination of the movement direction from the camera, and therefore does not learn the image transport direction.

The second pattern is a pattern in which pre-learning is performed before the product management system 1 is operated. FIG. 10 is a diagram showing an example of a processing flowchart of the learning model pre-learning processing for image analysis S300.

The management device 100 captures images of the product (plurality, singularity) from multiple angles (S300-1). In step S300-1, for example, a plurality of products are stacked and photographed from four directions: front, rear, left, and right.

The management device 100 takes an image of the individual article (S300-2). Individual items are, for example, products (including packaging) of various shapes and sizes. In addition to products, individual items include carts for carrying products, people who carry products, objects in warehouses, and the like. In other words, the individual items are objects other than products that may exist (locate) around the gate 200 . By using video data of individual articles for pre-learning, the accuracy of determining whether or not they are products is improved.

The management device 100 generates image data (processed image data) in which boundaries between products are processed (S300-3). FIG. 11 is a diagram showing an example of processed image data. In step S300-3, the management device 100 processes the image data of FIG. 11(A) as shown in FIGS. 11(B), (C), and (D).

FIG. 11B is a diagram showing an example of processed data emphasizing a line where products come into contact with each other. The thick line is the boundary line where the product below and the product stacked above contact each other.

FIG. 11C is a diagram showing an example of processed data in which boundaries between products on an image are emphasized. In addition to the boundary line where the lower product and the upper product actually come into contact, the boundary line where the products overlap on the image is also made thick.

FIG. 11D is a diagram showing an example of processed data in which all lines of the product are emphasized. For example, if the product is a rectangular parallelepiped, thick lines are used for all sides of the range where both products are shown.

In this way, by using processed data in which the shapes of products, between products, between products, or between products, or between products, are made clearer, the accuracy of product number recognition is improved.

The management device 100 uses the photographed data and processed data as teacher data, learns the learning model for image analysis (S300-4), and ends the process.

In this way, the management device 100 does not learn the number of products itself in the pre-learning. This requires a huge amount of teacher data in order to learn the number, increasing the time and man-hours. In the fourth embodiment, the management device 100 learns individual products and articles, and learns boundary lines using processed data in which lines are partially emphasized, so that the number of products itself does not need to be learned. Also, it is possible to construct a learning model for image analysis that can determine the number of products (can recognize individual products).

[Other embodiments]
The embodiments described above may be implemented in combination with each other. For example, a still image may be used instead of the moving image in the first embodiment. In this case, still images may be taken periodically to determine whether the product has moved. In addition, prior learning, learning during operation, and the like may be performed as appropriate in consideration of the time and man-hours required for learning.

Reference Signs List 1: Product management system 10: Image processing unit 11: Image analysis learning model learning unit 12: Number determination unit 13: Conveyance direction determination unit 14: Product management unit 15: Entry/exit list acquisition unit 16: Excess/deficiency determination unit 17: Learning data creating unit 20 : RFID reading unit 30 : Imaging unit 50 : Learning model for image analysis 100 : Management device 110 : CPU
120: Storage 121: RFID reading program 122: Imaging program 123: Image processing program 124: Product management program 125: Learning model learning program for image analysis 130: Memory 160: Display 200: Gate 300: Camera

Claims (5)

a gate unit having an RFID reader function to read product identification information from an RF tag attached to the product;
a photographing unit for photographing the periphery of the gate;
A product management system comprising a management unit that manages products passing through the gate unit based on data captured by the imaging unit and identification information read by the gate unit,
The management department
Based on the image data captured by the imaging unit, a transport direction determination process for determining a transport direction indicating whether the product to be transported is entering or leaving the warehouse, and a number determination process for determining the number of products to be transported. an image processing unit that performs
The number of pieces of identification information read by the gate unit is compared with the number of the conveyed products determined by the image processing unit, and excess or deficiency of the conveyed products is determined for each conveying direction. A product management system having a determination unit. The image data is video data,
2. The product management system according to claim 1, wherein the image processing unit instructs the gate unit to read the RF tag when recognizing in the moving image data that the transportation of the product to be transported has started. . The excess/deficiency determination unit further acquires a product list containing information about the number of products to be conveyed, the number of products to be conveyed in the product list, the number of identification information read by the gate unit, and 2. The product management system according to claim 1, wherein the number of products to be transported determined by the image processing unit is compared with the number of the products to be transported, and if all match, it is determined that there is no shortage or excess of the products to be transported. the product list includes identification information of the products to be conveyed;
The excess/deficiency determination unit determines that the identification information of the product to be conveyed in the product list matches the identification information read by the gate unit, and furthermore, the identification information of the product to be conveyed determined by the image processing unit matches. 4. The product management system according to claim 3, wherein when the number of pieces matches the number of pieces of identification information read by the gate unit, it is determined that there is no excess or deficiency in the conveyed products. the product list includes information about the direction of transport;
5. The excess/deficiency determination unit further determines that there is no excess/deficiency in the transported products when the transport direction of the product list matches the transport direction determined by the image processing unit. product management system.

Images