JP7130945B2 - Inventory detection program, inventory detection method and inventory detection device - Google Patents

Description

The present invention relates to an inventory detection program, an inventory detection method, and an inventory detection device.

2. Description of the Related Art In a store such as a supermarket, there is known a technique of capturing an image of a shelf and detecting a product from the image when managing shelf allocation information and inventory information of products displayed on the shelf. For example, techniques are known for receiving a realog image containing a plurality of organized objects and detecting and identifying objects in the realog image of one or more items on a retail shelf. The technology identifies the front of the shelf and labels it, identifies empty space under the shelf, identifies areas where there may be unidentified products, and products are "out of stock." Identify the area where

JP 2016-115348 A

However, with the above technology, there is a high possibility of erroneous inventory detection. For example, in the above technology, inventory is detected using the brightness or gray scale value of the image. However, since the luminance is easily affected by external factors such as the brightness of the outside, there is a high possibility that the above technology will cause erroneous detection of products and shelf boards, or between products and shelves.

An object of one aspect of the present invention is to provide an inventory detection program, an inventory detection method, and an inventory detection device capable of accurately detecting product inventory.

In one aspect, the inventory detection program causes the computer to detect the shelf board of the shelf on which the product is displayed using the information about the distance from the reference point. The inventory detection program causes the computer to calculate the depth from the shelf board at the measurement point using the information about the distance from the reference point. The inventory detection program uses the depth to cause the computer to determine the inventory status of the product at the position corresponding to the measurement point.

According to one aspect, product inventory can be detected with high accuracy.

FIG. 1 is a diagram illustrating an example of a process of acquiring distance measurement data according to the first embodiment. FIG. 2 is a diagram showing an example of inventory status. FIG. 3 is a diagram showing an example of distance measurement data corresponding to inventory status in the first embodiment. FIG. 4 is a diagram showing an example of distance measurement data converted into point cloud data. FIG. 5 is a diagram showing an example of a detection device according to the first embodiment. FIG. 6 is a diagram illustrating an example of a distance measurement data DB according to the first embodiment; FIG. 7 is a diagram illustrating an example of a depth DB according to the first embodiment; FIG. 8 is a diagram illustrating an example of a planogram DB according to the first embodiment. 9A is a diagram illustrating an example of shelf board detection processing according to the first embodiment; FIG. 9B is a diagram illustrating an example of shelf board detection processing according to the first embodiment; FIG. 9C is a diagram illustrating an example of shelf board detection processing according to the first embodiment; FIG. FIG. 10 is a diagram illustrating an example of depth calculation processing according to the first embodiment. FIG. 11 is a diagram illustrating an example of an output result according to the first embodiment; FIG. 12 is a flowchart illustrating an example of inventory detection processing according to the first embodiment. FIG. 13 is a flowchart illustrating an example of shelf board detection processing according to the first embodiment. FIG. 14 is a flowchart illustrating an example of inventory determination processing according to the first embodiment. FIG. 15 is a diagram illustrating an example of depth calculation processing according to the second embodiment. FIG. 16 is a diagram showing an example of distance measurement data corresponding to inventory status in the second embodiment. FIG. 17 is a diagram illustrating an example of a detection device according to the second embodiment; FIG. 18 is a diagram illustrating an example of a distance measurement data DB according to the second embodiment; FIG. 19 is a diagram illustrating an example of a depth DB according to the second embodiment; FIG. 20 is a flow chart showing an example of inventory detection processing in the second embodiment. FIG. 21 is a flowchart illustrating an example of depth identification processing according to the second embodiment. FIG. 22 is a diagram illustrating an example of a computer that executes a detection program;

Exemplary embodiments of an inventory detection program, an inventory detection method, and an inventory detection device disclosed in the present application will be described in detail below with reference to the drawings. In addition, this invention is not limited by this Example. Moreover, each embodiment shown below may be appropriately combined within a range that does not cause contradiction.

The detection device 100 in this embodiment detects the inventory of products using distance measurement data obtained by measuring the distance from the reference point to the measurement point. FIG. 1 is a diagram illustrating an example of a process of acquiring distance measurement data according to the first embodiment. As shown in FIG. 1, the detection device 100 in this embodiment is, for example, a self-propelled robot. The detection apparatus 100 acquires distance measurement data from the reference point 920 to the measurement point using a distance measurement sensor that measures the distance to the object by irradiating, for example, infrared rays, sound waves, or the like. Note that the ranging data is an example of information about distance.

The detecting device 100 specifies the position of the shelf board 901 of the shelf 900 shown in FIG. Similarly, the detection device 100 moves from the position of the shelf board 901 shown in FIG. Depth is measured to indicate depth to point 910 . When the depth is 0 or small, the detection device 100 determines that there is a sufficient inventory of products, and when the depth is large, the detection device 100 determines that there is little or no inventory of products.

The relationship between the depth and product inventory will be described with reference to FIGS. 2 and 3. FIG. FIG. 2 is a diagram showing an example of inventory status. In FIG. 2, the upward direction in the drawing corresponds to the depth direction of the shelf, and the downward direction in the drawing corresponds to the front direction of the shelf. As shown in FIG. 2, for example, a store clerk 700 grasps the inventory of products 810 to 850 by viewing the shelves from the front side.

In FIG. 2, for example, items 810 and 850 are fully stocked on the shelf, but item 820 is understocked. On the other hand, product 840 is out of stock.

FIG. 3 shows the depth measurement result indicating the depth from the shelf plate 520 in such a product shelf. FIG. 3 is a diagram showing an example of distance measurement data corresponding to inventory status in the first embodiment. In FIG. 3, the x-axis indicates the horizontal direction of the shelf, and the z-axis indicates the depth direction of the shelf. On the z-axis, higher numbers indicate greater depth.

As shown in FIG. 3, for fully stocked items 810 , the corresponding depth 811 is less than the first threshold 801 . The same is true for depth 851 corresponding to item 850 . On the other hand, the depth 821 corresponding to the product 820 with low inventory is included in the range 891 that is greater than or equal to the first threshold 801 but less than the second threshold 802 . Also, the depth 841 corresponding to the out-of-stock product 840 is included in the range 892 above the second threshold 802 .

In addition, when point cloud data acquired as distance measurement data is illustrated, an image as shown in FIG. 4 is obtained. FIG. 4 is a diagram showing an example of distance measurement data converted into point cloud data. In the image 500 shown in FIG. 4, the distance measurement data of the portion where the distance is small is shown in a color close to white, and the longer the distance is, the color close to black is shown. It should be noted that parts where the distance cannot be measured, such as transparent products and products containing liquid, are shown in black.

In this manner, the detection apparatus 100 in this embodiment detects the shelf board of the display shelf using the distance measurement data indicating the distance from the measurement point, and detects the shelf board using the distance measurement data above the shelf board by a predetermined coordinate. Since the depth from the board is calculated and the inventory status of the product is determined, the inventory can be easily detected.

[Function block]
Next, functions of the detection device 100 in this embodiment will be described with reference to FIG. FIG. 5 is a diagram showing an example of a detection device according to the first embodiment. As shown in FIG. 5, the detection device 100 in this embodiment has an external I/F 110, a storage section 120, and a control section .

The external I/F 110 controls communication with other computers such as a store clerk's terminal (not shown) and other database servers (not shown), regardless of whether they are wired or wireless. The external I/F 110 is, for example, a communication interface such as a NIC (Network Interface Card). The external I/F 110 acquires ranging data from an external device having a ranging sensor, for example. In addition, as shown in FIG. 1, when the detection device 100 is implemented by a robot or the like having a distance measurement sensor, the external I/F 110 may be replaced with the distance measurement sensor.

The storage unit 120 stores various data such as programs executed by the control unit 130, for example. The storage unit 120 also has a ranging data DB 121 , a depth DB 122 and a planogram DB 123 . The storage unit 120 corresponds to a semiconductor memory device such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, or a storage device such as a HDD (Hard Disk Drive). In addition, in the following description, a database (Database) may be written as "DB."

The distance measurement data DB 121 stores the distance measurement data in association with the coordinates of the measurement point. FIG. 6 is a diagram illustrating an example of a distance measurement data DB according to the first embodiment; As shown in FIG. 6, the distance measurement data DB 121 stores, for example, "storage number", "x coordinate", "y coordinate", and "distance" in association with each other. Information stored in the distance measurement data DB 121 is input by, for example, a distance acquisition unit 131 to be described later. The distance measurement data DB 121 stores, for example, one record for each measurement point.

In FIG. 6, "shelve number" stores an identifier that uniquely identifies a product display shelf. "x coordinate" and "y coordinate" store the coordinates of the measurement point. Note that the y-coordinate indicates the vertical direction of the shelf. "Distance" stores the distance at the measurement point.

The depth DB 122 stores the depth calculated from the ranging data in association with the position on the shelf. FIG. 7 is a diagram illustrating an example of a depth DB according to the first embodiment; As shown in FIG. 7, the depth DB 122 stores "shelve number", "shelf level", "x coordinate", and "depth" in association with each other. Information stored in the depth DB 122 is input by, for example, a depth calculation unit 133 to be described later.

In FIG. 7, "shelf level" stores the position of the shelf level in the commodity display shelf of the shelf number. For example, the "shelf level" is set to "1" for the uppermost level. "Depth" stores the depth, which is calculated from the distance measurement data and indicates the depth from the shelf board to the measurement point.

The planogram DB 123 stores the arrangement of products on product display shelves. FIG. 8 is a diagram illustrating an example of a planogram DB according to the first embodiment. As shown in FIG. 8, the shelving allocation DB 123 stores "ID", "shelf number", "shelf level", "product name", and FACE number in association with each other. Information to be stored in the planogram DB 123 is input in advance by, for example, an administrator (not shown) of the detection device 100 .

In FIG. 8, "ID" is an identifier that uniquely identifies the product displayed on the product display shelf. "Left end" and "right end" store the range of the display position of the product in the horizontal direction, for example, the x-axis direction. "Product name" stores the name of the product to be displayed.

Returning to FIG. 5 , the control unit 130 is a processing unit that controls overall processing of the detection device 100 . The control unit 130 is implemented by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like executing a program stored in an internal storage device using a RAM as a work area. Also, the control unit 130 may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The control unit 130 has a distance acquisition unit 131 , a shelf detection unit 132 , a depth calculation unit 133 , an inventory determination unit 134 and an output unit 135 . Note that the distance acquisition unit 131, the shelf board detection unit 132, the depth calculation unit 133, the inventory determination unit 134, and the output unit 135 are examples of electronic circuits possessed by the processor and processes executed by the processor.

The distance acquisition unit 131 acquires ranging data. For example, the distance acquisition unit 131 acquires distance measurement data from a distance measurement sensor through the external I/F 110, associates the coordinates of the measurement point with the measured distance, and stores them in the distance measurement data DB 121.

The shelf detection unit 132 detects the shelf using the distance measurement data. Processing by the shelf detection unit 132 will be described with reference to FIGS. 9A to 9C. In addition, the shelf board detection part 132 is an example of a detection part.

9A to 9C are diagrams illustrating an example of shelf board detection processing according to the first embodiment. First, as shown in FIG. 9A, the shelf detection unit 132 identifies the central portion in the left-right direction, for example, the median line 511 on the x-coordinate straight line in the distance measurement data acquired as the point cloud data. Next, the shelf board detection unit 132 specifies, among the points on the median line 511, the point with the shortest distance, for example, the point 521 with the smallest distance measurement data as the first point.

Next, the shelf board detection unit 132 identifies points obtained by moving the first point in the left-right direction by predetermined coordinates as the second point and the third point. For example, when the point 521 is specified as the first point, the shelf detection unit 132 specifies points 522 and 523 shown in FIG. 9B as the second and third points.

Then, the shelf detection unit 132 determines whether the difference between the distance measurement data at each of the second point and the third point and the distance measurement data at the first point is less than a predetermined threshold. do. When the shelf board detection unit 132 determines that the difference in distance is less than the predetermined threshold value, the shelf board detection unit 132 generates a new point that is moved by a predetermined coordinate in the horizontal direction from each of the second point and the third point. identified as a second point and a third point. The shelf detection unit 132 repeats the process of identifying the second point and the third point until the new second point and the third point reach the left end 532 and the right end 533 of the point cloud data.

When the new second and third points reach the left and right ends of the point group data, the shelf detection unit 132 detects a straight line in the left-right direction connecting the second point and the third point as a shelf. Detect as the position of the board. Then, the shelf board detection unit 132 specifies, among the points on the median line 511, the point having the smallest distance measurement data after the first point as a new first point. Also, when the shelf detection unit 132 determines that the difference between the distance measurement data at each of the second point and the third point and the distance measurement data at the first point is equal to or greater than a predetermined threshold, , similarly identify a new first point.

The shelf detection unit 132 starts the process of identifying the first point from the bottom end of the point cloud data in the vertical direction, for example, the Y-axis direction, and reaches a point moved downward by a predetermined coordinate from the top end of the Y-axis direction. A plurality of shelf boards included in the product display shelf are detected by repeating the processing until the product display shelf is reached. For example, in FIG. 9C, the shelf detection unit 132 starts processing from the lower end Ymin in the Y-axis direction and moves down from the upper end Ymax in the Y-axis direction by a predetermined coordinate h1 to a point Ymax-h1. repeat. Thereby, the shelf detection unit 132 detects the shelves 540 to 580 in addition to the shelf 520 . It should be noted that the reason why the process is not repeated up to the upper end Ymax in the Y-axis direction and the process ends at a point moved downward by a predetermined coordinate is that if there is no space between the upper ends of the shelves, This is because the products cannot be displayed.

The depth calculator 133 calculates the depth at the measurement point using the distance measurement data. Processing by the depth calculation unit 133 will be described with reference to FIG. 10 . Note that the depth calculator 133 is an example of a calculator.

FIG. 10 is a diagram illustrating an example of depth calculation processing according to the first embodiment. First, the depth calculator 133 identifies a straight line 611 at a position moved upward by a predetermined coordinate from the position where the shelf board 520 was detected in the point cloud data. Next, the depth calculator 133 acquires distance measurement data at a specific x-coordinate point 612 on the straight line 611 .

The depth calculator 133 then calculates the difference between the distance measurement data at the point 612 and the distance measurement data at the shelf board 520 as the depth at the point 612 . The depth calculator 133 stores the calculated depth in the depth DB 122 in association with the x-coordinate of the point 612 and the shelf corresponding to the shelf 520 .

For example, the depth calculation unit 133 starts processing from the left end of the shelf in the x-coordinate direction, calculates the depth, moves rightward by a predetermined coordinate, and repeats the processing.

The inventory determination unit 134 uses the depth to determine product inventory. Note that the inventory determination unit 134 is an example of a determination unit.

The inventory determination unit 134 acquires depth from the depth DB 122 . In addition, the inventory determination unit 134 refers to the planogram DB 123 to identify the products to be displayed at the position corresponding to the acquired depth. The inventory determination unit 134 then compares the acquired depth with the first threshold and the second threshold.

When the depth is less than the first threshold, the inventory determining unit 134 determines that the product displayed at the corresponding position is "in stock". When the depth is greater than or equal to the first threshold and less than the second threshold, the inventory determining unit 134 determines that the product displayed at the corresponding position is "out of stock". When the depth is equal to or greater than the second threshold, the inventory determination unit 134 determines that the product displayed at the corresponding position is "out of stock".

The output unit 135 outputs the inventory determination result at each measurement point in association with the information on the planogram. The output unit 135 acquires the “shelve number”, “shelve stage” and “x coordinate” of each depth from the depth DB 122 . Next, the output unit 135 refers to the shelving allocation DB 123 to specify the products to be displayed at the positions corresponding to each "shelve number", "shelf level" and "x coordinate". Then, the output unit 135 outputs the inventory status of the products displayed at each position.

For example, the output unit 135 specifies that the shelf number “1”, the shelf level “1”, and the x-coordinate “80” in the depth DB 122 correspond to the ID “A001” in the shelf allocation DB 123 . Then, the output unit 135 outputs information indicating that the product is "in stock".

FIG. 11 is a diagram illustrating an example of an output result according to the first embodiment; As shown in FIG. 11, the output unit 135 generates an output result including information indicating that the product "●● Ramen" with the ID "A001" is "in stock". Similarly, the output unit 135 associates each position in the depth DB 122 with each product in the planogram DB 123, and outputs the inventory status of each product. For example, if the first threshold is "30" and the second threshold is "50", it is indicated that the product with a depth of "40" at the corresponding display position is "short of stock". Items with a depth of "50" at the corresponding display position are indicated as "out of stock".

[Process flow]
Next, processing in this embodiment will be described with reference to FIGS. 12 to 14. FIG. FIG. 12 is a flowchart illustrating an example of inventory detection processing according to the first embodiment. As shown in FIG. 12, the distance acquisition unit 131 of the detection device 100 waits until the distance measurement data is acquired through, for example, the external I/F 110 (S10: No).

When determining that the distance measurement data has been acquired (S10: Yes), the shelf detection unit 132 executes the shelf detection process shown in FIG. 13 (S11). The shelf board detection processing will be described later. Next, the depth calculator 133 calculates the depth (S12). Next, the inventory determination unit 134 executes inventory determination processing using the calculated depth (S13). The stock determination process will be explained later.

Next, the inventory determination unit 134 identifies a measurement point moved by w coordinates in the x direction (S14), and determines whether the x coordinate of the measurement point is equal to or greater than a predetermined threshold (S15). When the depth calculation unit 133 determines that the x-coordinate is less than the predetermined threshold value (S15: No), the process returns to S12 and repeats the process.

On the other hand, when the inventory determination unit 134 determines that the x-coordinate is equal to or greater than the predetermined threshold (S15: Yes), it determines whether or not the processing has been completed for all the shelf boards (S16). When it is determined that the processing has not been completed for all the shelf boards (S16: No), the depth calculation unit 133 identifies the next detected shelf board (S17), returns to S12, and repeats the process.

On the other hand, when the output unit 135 determines that the processing has been completed for all the shelf boards (S16
: Yes), the planogram DB 123 is referred to match the coordinates stored in the depth DB 122 with the planogram information (S18). Then, the output unit 135 outputs the matching result (S19), and ends the process.

Next, shelf board detection processing will be described. FIG. 13 is a flowchart illustrating an example of shelf board detection processing according to the first embodiment. As shown in FIG. 13, the shelf detection unit 132 identifies the median line on the x-coordinate straight line (S110). Next, the shelf board detection unit 132 identifies the closest point on the median line as the first point (S111).

And the shelf board detection part 132 specifies a 2nd point and a 3rd point (S113). The shelf detection unit 132 compares the distance measurement data of the first point with the distance measurement data of the second and third points to determine whether the difference in distance is less than a predetermined threshold. Determine (S114). When the shelf detection unit 132 determines that the distance difference is equal to or greater than the predetermined threshold value (S114: No), the process proceeds to S117.

When the shelf detection unit 132 determines that the distance difference is less than the predetermined threshold (S114: Yes), the second point and the third point reach the left and right ends in the point cloud data. , return to S113 and repeat the process (S115: No).

Then, when the shelf detection unit 132 determines that the second point and the third point have reached both the left and right ends in the point group data (S115: Yes), the second point and the third point is detected as a shelf plate (S116).

Then, the shelf board detection unit 132 specifies the next closest point on the median line (S117), and determines whether or not the y-coordinate of the point is equal to or greater than Ymax-h1 (S118). When the shelf detection unit 132 determines that the y-coordinate is less than Ymax-h1 (S118: No), the process returns to S113 and repeats the process. On the other hand, when the shelf detection unit 132 determines that the y-coordinate is equal to or greater than Ymax-h1 (S118: Yes), it returns to the inventory detection process.

FIG. 14 is a flowchart illustrating an example of inventory determination processing according to the first embodiment. First, the inventory determination unit 134 determines whether or not the acquired depth is equal to or greater than the first threshold (S130). When the inventory determining unit 134 determines that the depth is less than the first threshold (S130: No), it determines that the product displayed at the corresponding position is "in stock" (S132), and performs inventory detection processing. back to

On the other hand, when the inventory determination unit 134 determines that the depth is greater than or equal to the first threshold (S130: Yes), it determines whether or not the depth is greater than or equal to the second threshold (S131). When the inventory determination unit 134 determines that the depth is less than the second threshold (S131: No), it determines that the product displayed at the corresponding position is "shortage" (S133), and the inventory detection process return.

When the inventory determination unit 134 determines that the depth is equal to or greater than the second threshold (S131: Yes), it determines that the product displayed at the corresponding position is "out of stock" (S134), and performs the inventory detection process. return.

[effect]
As described above, the inventory detection device in this embodiment uses information about the distance from the reference point to detect the shelves of the shelves on which the products are displayed. The inventory detection device uses information about the distance from the reference point to calculate the depth from the shelf board at the measurement point. The inventory detection device uses the depth to determine the inventory status of the item at the location corresponding to the measurement point. As a result, product inventory can be detected with high accuracy.

Further, the inventory detection device in the present embodiment determines that the product is in short supply when the depth is greater than or equal to the first threshold and less than the second threshold, and determines that the product is in short supply when the depth is greater than or equal to the second threshold. , the product is determined to be out of stock. Furthermore, the inventory detection device may output the result of the judging process. This makes it possible to more subdivide and specify the product inventory status.

In addition, the inventory detection device in this embodiment detects the first point determined to be closest to the reference point in the central portion in the left-right direction of the point cloud data acquired as the information on the distance from the reference point. Identify. The inventory detection device is a second point and a third point that are moved by a predetermined coordinate in the horizontal direction of the first point, and the distance from the reference point is the distance from the reference point to the first point. Detect points that are determined to be close. Then, when the second point and the third point are detected at both ends of the point cloud data in the left-right direction, the inventory detection device identifies a straight line connecting the second point and the third point as the shelf board. do. Thereby, a shelf board can be specified more accurately.

In addition, when the shelf board is specified, or when the second point or the third point is not specified, the inventory detection device in this embodiment detects the reference point at the center of the point cloud data in the horizontal direction. A point determined to be close to the point may be newly set as the first point. Furthermore, the inventory detection device may start the detection process from the point at the lower end of the point cloud data in the vertical direction and repeat it until it reaches a point moved by a predetermined coordinate from the upper end of the point cloud data in the vertical direction. . Thereby, a plurality of shelf boards can be specified with high accuracy.

In addition, the inventory detection device in this embodiment identifies the first measurement point that has moved upward by a predetermined coordinate from the shelf board, and provides information on the distance from the reference point to the first measurement point and the distance from the shelf board to the shelf board. is calculated as the depth. This makes it possible to identify the depth of the product based on the position of the shelf board.

By the way, as shown in FIG. 4, there are cases where the distance cannot be measured due to the characteristics of the distance measuring sensor, the characteristics of the product at the measuring point, and the like. If the ranging data cannot be obtained, it is difficult to properly detect product inventory.

Therefore, in the present embodiment, a configuration will be described in which, when distance measurement data cannot be acquired, the measurement point is changed and the distance acquisition is attempted again. FIG. 15 is a diagram illustrating an example of depth calculation processing according to the second embodiment. Point cloud data 699 shown in FIG. 15 indicates distance measurement data of a position where, for example, a PET bottle 690 is displayed. Also, the PET bottle shown in FIG. 15 includes a portion covered with the label and a transparent portion not covered with the label.

As shown in FIG. 15, if the measurement point 691 corresponding to the straight line 611 is a transparent portion that is not covered with the PET bottle label, the distance cannot be obtained correctly for that measurement point. Therefore, in FIG. 15, the distance measurement data 613 corresponding to the measurement point 691 in the point cloud data indicates that the distance could not be measured.

In this case, the detection device 300 in this embodiment, which will be described later, further identifies a straight line 621 at a position moved upward by a predetermined coordinate from the straight line 611 . Then, the detection device 300 acquires distance measurement data at a specific x-coordinate measurement point 623 on the straight line 621 . In this case, since the portion 692 of the PET bottle 690 corresponding to the measurement point 623 is the portion covered with the label, distance measurement data at the measurement point 623 can be obtained.

For example, if the distance cannot be measured at the portion 692 as well, the detection device 300 identifies a straight line 631 at a position that is further moved upward by a predetermined coordinate from the straight line 621 . Then, the detection device 300 acquires distance measurement data at a specific x-coordinate measurement point 633 on the straight line 631 . In this case, the portion 693 of the PET bottle 690 corresponding to the measurement point 633 is a transparent portion that is not covered with a label, so the distance cannot be obtained at the measurement point 633 either.

FIG. 16 shows the depth measurement results in this example. FIG. 16 is a diagram showing an example of distance measurement data corresponding to inventory status in the second embodiment. In the following embodiments, the same reference numerals are given to the same parts as the parts shown in the previously described drawings, and overlapping explanations will be omitted.

In FIG. 16, for example, “●” marks 8021, 8041 and 8051 indicate that distances were obtained at measurement points on the straight line 611. In FIG. A "*" mark 8011 indicates that the distance is obtained at a measurement point on the straight line 621 at a position shifted upward from the straight line 611 by a predetermined coordinate. Also, the "x" mark 8031 indicates that the distance could not be obtained at any of the measurement points on the straight line.

[Function block]
Next, regarding the function of the detection device 300 in the present embodiment, FIG. 17 is a diagram showing an example of the detection device in the second embodiment. As shown in FIG. 17 , the detection device 300 in this embodiment has an external I/F 110 , a storage section 320 and a control section 330 . In the following embodiments, the same reference numerals are given to the same parts as the parts shown in the previously described drawings, and overlapping explanations will be omitted.

The storage unit 320 stores various data such as programs executed by the control unit 330, for example. The storage unit 320 also has a distance measurement data DB 321 , a depth DB 322 and a planogram DB 123 . The storage unit 320 corresponds to semiconductor memory elements such as RAM, ROM, and flash memory, and storage devices such as HDD.

The ranging data DB 321 stores ranging data in the same manner as the data shown in FIG. Note that the distance measurement data DB 321 further stores information indicating that the distance measurement data was not acquired (for example, a "-" sign) when the distance measurement data is not acquired by the distance acquisition unit 331, which will be described later. do. FIG. 18 is a diagram illustrating an example of a distance measurement data DB according to the second embodiment; As shown in FIG. 18, the distance measurement data DB 321 stores information indicating that the distance measurement data was not obtained at the measurement point at the x-coordinate of "200" and the y-coordinate of "0" in the storage bin "1". .

The depth DB 322 stores the depth calculated from the distance measurement data shown in FIG. 7 in association with the position on the shelf and the height at which the distance was measured. FIG. 19 is a diagram illustrating an example of a depth DB according to the second embodiment; As shown in FIG. 19, the depth DB 322 stores "shelve number", "shelf level", "x-coordinate", and "depth" in association with "measurement height".

In FIG. 19, "measurement height" stores the height in the Y-axis direction of the measurement point at which the distance was measured. In the example shown in FIG. 19, for example, measurement height "1" indicates that the distance was obtained at a measurement point on line 611 shown in FIG. Indicates that the distance was obtained with a point. Note that "-" indicates that the distance could not be obtained at any measurement point. In the example shown in FIG. 19, the depth DB 322 acquires the distance at the measurement point on the straight line 611 at the x-coordinate "80" of the shelf number "1" and the x-coordinate "160" and " 240” stores that the distance was obtained at the measurement point on the straight line 621 .

The control unit 330 is a processing unit that controls overall processing of the detection device 300 . The control unit 330 is implemented, for example, by executing a program stored in an internal storage device using a RAM as a work area by a CPU, MPU, or the like. Also, the control unit 330 may be implemented by an integrated circuit such as an ASIC or FPGA, for example.

The control unit 330 has a distance acquisition unit 331 , a shelf detection unit 132 , a depth calculation unit 333 , an inventory determination unit 334 and an output unit 135 . Note that the distance acquisition unit 331, the shelf detection unit 132, the depth calculation unit 333, the inventory determination unit 334, and the output unit 135 are examples of electronic circuits possessed by the processor and processes executed by the processor.

When the distance measurement data cannot be acquired, the distance acquisition unit 331 stores information indicating that the distance measurement data has not been acquired in the distance measurement data DB 321 in association with the target coordinates.

The depth calculation unit 333 calculates the depth at the measurement point using the distance measurement data, and when storing it in the depth DB 322, also stores the measurement height at which the distance measurement data used was acquired. If the distance measurement data at the measurement point cannot be obtained, the depth calculation unit 333 tries to calculate the depth again using the distance measurement data at a different measurement point.

For example, when the distance measurement data cannot be acquired at the measurement point on the straight line 611 shown in FIG. . For example, when the distance measurement data can be acquired at the measurement point on the straight line 621, the depth calculation unit 333 calculates the depth using the distance measurement data, and associates it with the measured height "2" corresponding to the measurement point. Store in the depth DB 322 .

Further, if the distance measurement data cannot be acquired at any of the candidate measurement points, the depth calculation unit 333 stores information indicating that the depth could not be specified in the depth DB 322 . For example, when the distance measurement data cannot be acquired even at the measurement points on the straight line 631 shown in FIG. -” is stored.

[Process flow]
Next, processing in this embodiment will be described with reference to FIGS. 20 and 21. FIG. In the following description, the same reference numerals as the steps shown in FIGS. 12 to 14 are the same steps, and detailed description thereof will be omitted.

FIG. 20 is a flow chart showing an example of inventory detection processing in the second embodiment. As shown in FIG. 20, in the inventory detection process in the second embodiment, depth specifying process is executed instead of S12 and S13 shown in FIG. 12 (S22).

FIG. 21 is a flowchart illustrating an example of depth identification processing according to the second embodiment. First, the depth calculator 333 identifies a straight line at a position moved upward by a predetermined coordinate from the detected shelf board (S221). Next, the depth calculation unit 333 refers to the distance measurement data DB 321 and determines whether or not the distance has been acquired at the measurement point on the specified straight line (S222).

When determining that the distance has been acquired (S222: Yes), the depth calculation unit 333 calculates the depth using the acquired ranging data (S223). Then, the depth calculation unit 333 performs the inventory determination process shown in FIG. 14 using the calculated depth (S13), and returns to the inventory detection process.

On the other hand, if the depth calculation unit 333 determines that the distance could not be obtained (S222: No), it identifies a straight line at a position moved upward by a predetermined coordinate from the identified measurement point (S224). Then, the depth calculation unit 333 determines whether or not the y-coordinate of the identified straight line is equal to or greater than a predetermined threshold (S225). When the depth calculation unit 333 determines that the y-coordinate is less than the predetermined threshold value (S225: No), the process returns to S222 and repeats the process.

On the other hand, if the depth calculation unit 333 determines that the y-coordinate is equal to or greater than the predetermined threshold (S225: Yes), it determines that there is no depth information, and stores the determination result in the depth DB 322 (S226). , return to inventory detection processing.

[effect]
As described above, the inventory detection device in this embodiment compares the shelving allocation data including the number of products displayed on the shelf with the inventory status and information on the size of the product, thereby identifying the product. Identify the type and number of items in stock. As a result, it is possible to identify the inventory status in line with the planogram.

Further, if the inventory detection device in this embodiment cannot specify the depth at the first measurement point, it may specify a second measurement point that is moved upward by a predetermined coordinate from the first measurement point. . Then, the inventory detection device may calculate the difference between the information regarding the distance from the reference point to the second measurement point and the information regarding the distance to the shelf as the depth. As a result, for example, even if the distance measurement data cannot be obtained at a specific position, it is possible to improve the possibility of detecting inventory.

Although the embodiments of the present invention have been described so far, the present invention may be implemented in various different forms other than the embodiments described above. For example, the detection device 100 in Embodiment 1 is implemented by a self-propelled robot as shown in FIG. can also Also, the server may acquire an image taken by a store clerk (not shown) instead of the image taken by the robot or the like.

Also, in FIG. 7, the depth calculation unit 133 stores the calculation result when, for example, after calculating the depth, it moves rightward by "80". You may change suitably according to the width of goods.

Also, in FIG. 9C, a configuration has been described in which processing ends at a point Ymax-h1 that is moved downward by a predetermined coordinate h1 from the upper end Ymax in the Y-axis direction, but the embodiment is not limited to this. For example, when products are hung from the upper end of the shelf, products may be displayed at the upper end of the shelf, so the shelf detection process may be repeated up to the upper end Ymax in the Y-axis direction.

If the second point and the third point are not detected in the shelf board detection process shown in FIG. have a nature. Therefore, if the second point and the third point are not detected, the shelf detection unit 132 may output information indicating that the product is displayed abnormally. The same applies when the depth is determined to be a negative value in the inventory determination process shown in FIG.

[system]
Also, each constituent element of each part illustrated does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each part is not limited to the one shown in the figure, and all or part of it can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. can be configured as For example, the depth calculation unit 133 and the inventory determination unit 134 may be integrated. Also, the process of comparing the inventory status and the planogram information of the output unit 135 and the process of outputting the result of matching may be divided and provided as separate processing units.

Furthermore, the various processing functions performed by each device may be executed in whole or in part on a CPU (or a microcomputer such as an MPU or MCU (Micro Controller Unit)). Also, various processing functions may be executed in whole or in part on a program analyzed and executed by a CPU (or a microcomputer such as an MPU or MCU) or on hardware based on wired logic. It goes without saying that it is good.

By the way, the various processes described in the above embodiments can be realized by executing a prepared program on a computer. Therefore, an example of a computer that executes a program having functions similar to those of the above embodiments will be described below. FIG. 22 is a diagram illustrating an example of a computer that executes a detection program; Although the detection device 100 according to the first embodiment will be described below as an example, the detection device 300 according to the third embodiment can also be realized by a similar computer.

As shown in FIG. 22, the computer 200 has a CPU 201 that executes various arithmetic processes, an input device 202 that receives data input, and a monitor 203 . The computer 200 also includes a medium reading device 204 for reading programs and the like from a storage medium, an interface device 205 for connecting to various devices, and a communication device 206 for connecting to other information processing devices by wire or wirelessly. have The computer 200 also has a RAM 207 that temporarily stores various information, and a hard disk device 208 . Each device 201 - 208 is also connected to a bus 209 .

The hard disk device 208 stores a detection program having the same functions as the distance acquisition unit 131, the shelf board detection unit 132, the depth calculation unit 133, the inventory determination unit 134, and the output unit 135 shown in FIG. be. The hard disk device 208 also stores a distance measurement data DB 121, a depth DB 122, a planogram DB 123, and various data for implementing the detection program. The input device 202 receives input of various information such as management information from an administrator of the computer 200, for example. The monitor 203 displays, for example, management information screens and various screens for the administrator of the computer 200 . The interface device 205 has, for example, the same function as the external I/F 110 shown in FIG. 5, is connected to an external device having a sensor such as a camera or a distance measuring sensor, and exchanges various information. The communication device 206 has, for example, the same functions as the external I/F 110 shown in FIG. 5, is connected to an external database or the like, and exchanges various information.

The CPU 201 performs various processes by reading each program stored in the hard disk device 208, developing it in the RAM 207, and executing it. These programs can also cause the computer 200 to function as the distance acquisition unit 131, the shelf detection unit 132, the depth calculation unit 133, the inventory determination unit 134, and the output unit 135 shown in FIG.

Note that the detection program described above does not necessarily have to be stored in the hard disk device 208 . For example, computer 200 may read and execute a program stored in a storage medium readable by computer 200 . Examples of storage media readable by the computer 200 include portable recording media such as CD-ROMs, DVD discs, USB (Universal Serial Bus) memories, semiconductor memories such as flash memories, and hard disk drives. Alternatively, the detection program may be stored in a device connected to a public line, the Internet, a LAN, etc., and the computer 200 may read and execute the detection program therefrom.

100, 300 detector 110 external I/F
120, 320 storage section 121, 321 ranging data DB
122, 322 Depth DB
123 Planogram DB
130, 330 control unit 131, 331 distance acquisition unit 132 shelf board detection unit 133, 333 depth calculation unit 134, 334 inventory determination unit 135 output unit

Claims (10)

Using the information about the distance from the reference point indicating the position of the distance measuring sensor to the measurement points included in the shelves where the products are displayed, the measurement points in the left and right direction are specified from the measurement points closest to the reference point. to detect the shelf plate of the shelf,
Using information about the distance from the reference point to the measurement point included in the shelf, the distance from the shelf board detected from the reference point to the measurement point in the upward direction, and the distance from the reference point to the shelf board Calculate the difference from the shortest distance as the depth from the shelf board at the measurement point in the upward direction ,
An inventory detection program that causes a computer to execute a process of determining an inventory status of a product at a position corresponding to the upward measuring point from which the depth is calculated, according to the depth of the depth. causing the computer to further execute a process of outputting the result of the determination process;
The determination processing determines that the product is in short supply when the depth is greater than or equal to a first threshold and less than a second threshold, and determines that the product is out of stock when the depth is greater than or equal to the second threshold. 2. The inventory detection program according to claim 1, wherein it is determined that the product is out of stock. The output processing includes depth data storing the position of the shelf board at the measurement point, the position corresponding to the measurement point, and the depth from the shelf board at the measurement point, and the display positions of the products displayed on the shelf board. A process of identifying the position of the shelf board at the measurement point and the product displayed at the position corresponding to the measurement point by collating with the stored planogram data, and outputting the inventory determination result of the identified product 3. The inventory detection program according to claim 2, further causing said computer to execute: In the detecting process, a first point determined to be closest to the reference point is specified in the central portion in the horizontal direction of the point cloud data acquired as information about the distance from the reference point, A second point and a third point that are moved by a predetermined coordinate in the left-right direction of the first point are specified, and the distance from the reference point at each of the second point and the third point; If the difference between the first point and the distance from the reference point is less than a predetermined threshold value, the second point and the third point, which are moved horizontally by a predetermined coordinate sentence, are newly specified. , when the second point and the third point are detected at both ends of the point group data in the horizontal direction, a straight line connecting the second point and the third point is specified as the shelf board. 4. The inventory detection program according to any one of claims 1 to 3, characterized by: In the detecting process, when the shelf board is specified, or when the second point or the third point is not specified, at the center in the horizontal direction of the point cloud data, then the reference point 5. The inventory detection program according to claim 4, wherein a point determined to be close to the first point is newly set as the first point. causing the computer to repeatedly execute the detection process, starting from a lower end point in the vertical direction of the point cloud data, until reaching a point moved by a predetermined coordinate from the upper end in the vertical direction of the point cloud data; 6. An inventory detection program according to claim 5. The calculating process specifies a first measurement point moved upward by a predetermined coordinate from the shelf board, information on the distance from the reference point to the first measurement point, and information on the distance to the shelf board. 7. The stock detection program according to any one of claims 1 to 6, wherein a difference from distance information is calculated as the depth. In the calculating process, when the depth cannot be specified at the first measurement point, a second measurement point further moved upward by a predetermined coordinate from the first measurement point is specified, and the depth is determined from the reference point. 8. The stock detection program according to claim 7, wherein the depth is calculated as a difference between the information on the distance to the second measurement point and the information on the distance to the shelf board. the computer
Using information about the distance from the reference point indicating the position of the distance measuring sensor to the measurement points included in the shelves where the products are displayed, the measurement points in the horizontal direction are specified from the measurement points closest to the reference point. to detect the shelf plate of the shelf,
Using information about the distance from the reference point to the measurement point included in the shelf, the distance from the shelf board detected from the reference point to the measurement point in the upward direction, and the distance from the reference point to the shelf board Calculate the difference from the shortest distance as the depth from the shelf board at the measurement point in the upward direction ,
An inventory detection method, comprising: determining an inventory status of a product at a position corresponding to the upward measuring point from which the depth is calculated, according to the depth of the depth. Using the information about the distance from the reference point indicating the position of the distance measuring sensor to the measurement points included in the shelves where the products are displayed, the measurement points in the left and right direction are specified from the measurement points closest to the reference point. a detection unit that detects the shelf plate of the shelf by
Using information about the distance from the reference point to the measurement point included in the shelf, the distance from the shelf board detected from the reference point to the measurement point in the upward direction, and the distance from the reference point to the shelf board a calculation unit that calculates the difference from the shortest distance as the depth from the shelf board at the measurement point in the upward direction ;
and a determination unit that determines, according to the depth of the depth, the inventory status of the product at a position corresponding to the measurement point in the upward direction from which the depth is calculated.

Images