JP2021025853A - Size measuring device, size measuring method, and program - Google Patents

Abstract

To provide a technology suitable for size measurement of a large number of objects.SOLUTION: A size measuring device has a camera that photographs an object present in a measurement area, a sensor that measures the distance from the camera to the object, and an information processing apparatus. The information processing apparatus has an extraction unit that extracts the object from an image obtained by the camera, and a size estimation unit that estimates the actual size of the object extracted from the image based on a measured distance measured by the sensor.SELECTED DRAWING: Figure 2

Description

The present invention relates to a technique for measuring the actual size of an object.

When a general consumer uses a parcel delivery service such as a courier service, the parcel delivery fee is set according to the size (for example, sum of width, length, height, etc.) and weight of the parcel. On the other hand, in the case of B2B logistics, it is not realistic to measure the size of each package because the number of packages is extremely large, and the shipping fee may be determined simply according to the number of packages. There are many. Even in B2B, it is desired to set an appropriate reward (delivery charge) according to the size of the package, but at present, there is no means to realize this.

In addition, in Patent Document 1, the baggage placed on the loading platform in the accommodation room is photographed by a camera installed in the accommodation room, and the width, length, and height of the baggage are measured from the obtained image. The device is disclosed. However, this measuring device has a problem of lacking versatility because the measurable luggage is limited to a size that can be accommodated in the accommodation chamber. Further, since the measurement must be performed with the luggage to be measured one by one positioned at a predetermined position in the accommodation chamber, it is not suitable for the purpose of measuring the size of a large number of luggage.

JP-A-2018-000386

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technique suitable for measuring the size of a large number of objects.

In order to achieve the above object, the present invention adopts the following configuration.

The first aspect of the present invention includes a camera that captures an object existing in a measurement area, a sensor that measures the distance from the camera to the object, and an information processing device. A size having an extraction unit that extracts the object from an image obtained by a camera, and a size estimation unit that estimates the actual size of the object extracted from the image based on the measurement distance measured by the sensor. A measuring device is provided.

According to this configuration, any object existing in the measurement area (for example, within the field of view of the camera and the sensor) can measure the actual size without strict positioning. Therefore, for example, a large number of objects are conveyed by a conveyor or the like, and the actual size of each object is measured while sequentially passing through the measurement area. Therefore, the actual size of a large number of objects can be measured efficiently and at high speed. Can be done.

The size estimation unit may calculate the actual size of the object in a plane parallel to the image by converting the length on the image into the length in the real space based on the measurement distance. .. Any conversion method may be used. For example, the size estimation unit uses a table in which the value of the distance from the camera to the object and the conversion ratio of the length on the image and the length in the real space are associated with each other on the image. The length of may be converted to the length in real space. Alternatively, the size estimation unit uses a model that learns the correlation between the value of the distance from the camera to the object and the conversion ratio of the length on the image and the length in the real space, and uses the image. The above length may be converted to the length in real space. Alternatively, the size estimation unit may convert the length on the image into the length in the real space by geometric calculation using the value of the distance from the camera to the object and the parameters of the camera. Good.

The size estimation unit may calculate the actual size of the object in the depth direction of the image from the difference between the measurement distance and the preset reference distance. It is possible to obtain the actual size of an object faster and more accurately by calculating the size of the object in the depth direction from the measurement distance of the sensor than by estimating the size of the object in the depth direction from the image of the camera.

The size estimation unit may set the reference distance based on the value measured by the sensor when the object is not present in the measurement area. Convenience can be improved by automating the setting of the reference distance.

The size measuring device measures the size of an object conveyed so as to pass through the measuring area, and the information processing device monitors the measured value of the sensor and measures the measured value according to a change in the measured value. It may further have a detection unit that detects the entry of an object into the area.

The information processing device may acquire an image used for estimating the actual size of the object, triggered by the detection of the entry of the object by the detection unit. As a result, the required image can be acquired at an appropriate timing.

In the detection unit, the measured value of the sensor is a first value smaller than the reference value and a second value smaller than the first value from the reference value obtained when no object exists in the measurement area. When the value gradually changes to the value of, it may be determined that a plurality of objects overlap.

When the detection unit detects the overlap of a plurality of objects, the size estimation unit puts the first value and the second on the first object having a height corresponding to the first value. The actual size of each of the first object and the second object may be estimated by assuming that the second objects having the height corresponding to the difference in the values of are overlapped. This makes it possible to measure the actual size of a plurality of objects that are transported in an overlapping manner. Alternatively, when the detection unit detects the overlap of a plurality of objects, the size estimation unit may perform control to notify that the actual size of the plurality of objects cannot be estimated. This is because the estimation accuracy of the actual size may decrease when a plurality of objects overlap. It can be expected that the operator who notices the notification can measure the actual size of each object with higher accuracy by taking measures such as eliminating the overlap of the objects.

The sensor may be an infrared sensor. By using an infrared sensor, it is possible to perform highly accurate distance measurement with a relatively inexpensive configuration.

The present invention may be regarded as a size measuring device having at least a part of the above means, or may be regarded as an information processing device constituting the size measuring device. Further, the present invention can be regarded as a size measuring method including at least a part of the above processing, a program for realizing such a method, or a recording medium on which the program is recorded. The present invention can be constructed by combining each of the above means and treatments with each other as much as possible.

According to the present invention, it is possible to provide a technique suitable for measuring the size of a large number of objects.

FIG. 1 is a diagram showing an application example of a size measuring device. FIG. 2 is a diagram showing a configuration example of a size measuring device. FIG. 3 is a flowchart showing the flow of the size measurement process by the size measuring device. FIG. 4 is a graph schematically showing changes in the measured values of the sensor. FIG. 5A is a diagram showing an example of an image captured from a camera, and FIG. 5B is a diagram showing an example of an extraction result of an object. FIG. 6 is a diagram showing the principle of size estimation. FIG. 7A is a diagram showing an example of a conversion ratio table, and FIG. 7B is a diagram showing an example of obtaining a conversion ratio by a model. FIG. 8A is a diagram showing two objects arranged apart from each other, and FIG. 8B is a diagram showing a change in the measured value of the sensor. FIG. 9A is a diagram showing two objects arranged on top of each other, and FIG. 9B is a diagram showing a change in the measured value of the sensor.

<Application example>
One of the application examples of the present invention will be described with reference to FIG. FIG. 1 shows an example in which the size measuring device 10 according to the embodiment of the present invention is applied to the automatic measurement of the size (dimensions) of an object.

In the example of FIG. 1, a size measuring device 10 is installed above a transport device (for example, a conveyor) 11, and an object (for example, a baggage such as a cardboard box) 14 transported one after another by the transport device 11 is a predetermined measurement area. As it passes through 13, the size measuring device 10 measures the size of the object 14 in real time in a non-contact manner. The measurement result (size information) is transmitted from the size measuring device 10 to the host system 12. The host system 12 is, for example, a database that manages information on each parcel, a POS system for settling a delivery fee for each parcel, a system that designs a load allocation of parcels to pallets, trucks, and the like.

According to such a size measuring device 10, the actual size of each of a large number of objects can be automatically measured. Therefore, for example, for the purpose of automatically setting a shipping fee according to the size of a package at a distribution center or the like. It can be preferably used.

<First Embodiment>
A specific configuration example of the size measuring device 10 according to the embodiment of the present invention will be described with reference to FIG. Hereinafter, for convenience of explanation, the surface on which the object 14 is placed (the surface of the conveyor in the example of FIG. 2) is referred to as a "reference surface R", and the transport direction of the object 14 is the X direction, parallel to the reference surface R and The direction perpendicular to the X direction is called the Y direction, and the direction perpendicular to the reference plane R is called the Z direction.

The size measuring device 10 generally includes a camera 20, a sensor 21, and an information processing device 22. FIG. 2 shows a configuration example in which the camera 20, the sensor 21, and the information processing device 22 are separately configured and each is connected by a cable, but the specific hardware configuration is not limited to this. For example, two or all of the camera 20, the sensor 21, and the information processing device 22 may be integrated. Alternatively, it may be connected wirelessly instead of being connected by cable. For example, a camera 20 and a sensor 21 may be installed on a plurality of rows of conveyors, and information obtained from them may be processed by one information processing device 22.

(Camera 20)
The camera 20 is an imaging device for photographing the object 14. In the present embodiment, the camera 20 is installed so that its optical axis is parallel to the Z direction, and the object 14 is photographed from above the reference plane R to obtain an upper view image of the object 14.

As the camera 20, either a monochrome camera or a color camera may be used. Further, depending on the application, a camera that captures electromagnetic waves other than visible light (for example, an infrared camera, an X-ray camera, etc.) may be used. The camera 20 may capture images at predetermined time intervals (for example, 30 fps), or may capture images according to a control signal from an external device (for example, the information processing device 22 or the sensor 21). The image obtained by the camera 20 is input to the information processing device 22.

(Sensor 21)
The sensor 21 is a measuring device for measuring the distance from the camera 20 to the object 14. The measurement result (distance information) of the sensor 21 is input to the information processing device 22.

As the sensor 21, any type of distance sensor may be used, and for example, an infrared sensor, an ultrasonic sensor, a laser displacement sensor, a ToF sensor, or the like may be used. In this embodiment, an infrared sensor is used because sufficient accuracy can be obtained, the cost is low, and the size is small.

The sensor 21 may be fixed to the camera 20 in a state where its optical axis is adjusted to be parallel to the optical axis of the camera 20. As a result, the relative positional relationship between the camera 20 and the sensor 21 is fixed, so that the work of installing and calibrating the camera 20 and the sensor 21 becomes easy.

(Information processing device 22)
The information processing device 22 is a device for estimating (calculating) the actual size of the object 14 based on the image of the object 14 obtained by the camera 20 and the distance information to the object 14 measured by the sensor 21. is there. As shown in FIG. 2, the information processing apparatus 22 of the present embodiment mainly includes an image acquisition unit 220, a distance information acquisition unit 221, a detection unit 222, an extraction unit 223, a size estimation unit 224, and an output unit 225. Have. The image acquisition unit 220 has a function of acquiring an image from the camera 20. The distance information acquisition unit 221 has a function of acquiring distance information from the sensor 21. The detection unit 222 has a function of detecting that the object 14 to be measured has entered the measurement area. The extraction unit 223 has a function of extracting the object 14 from the image of the camera 20. The size estimation unit 224 has a function of estimating the actual size of the object 14 based on the distance measured by the sensor 21. The output unit 225 has a function of outputting information on the actual size of the object 14 estimated by the size estimation unit 224.

The information processing device 22 may be composed of, for example, a general-purpose computer including at least a CPU (processor), a memory, and a non-volatile storage device. In that case, the above-described configuration of reference numerals 220 to 225 is realized by software by reading the program stored in the storage device into the memory and executing the program by the CPU. The configuration of the information processing device 22 is not limited to this, and for example, the configurations of reference numerals 220 to 225 may be realized by a plurality of devices by utilizing the techniques of distributed computing and cloud computing. Alternatively, part or all of the configurations of reference numerals 220 to 225 may be realized by a circuit such as FPGA or ASIC.

(Size measurement)
The flow of the size measurement process by the size measuring device 10 will be described with reference to the flowchart of FIG.

In step S30, the distance information acquisition unit 221 captures the measured value from the sensor 21. FIG. 4 is a graph schematically showing a change in the measured value of the sensor 21, in which the horizontal axis is the time [sec] and the vertical axis is the measured value [cm]. When the object 14 does not exist on the optical axis of the sensor 21, the distance from the sensor 21 (or the camera 20) to the reference plane R is obtained as a measured value, and while the object 14 crosses the optical axis of the sensor 21. , The distance from the sensor 21 (or the camera 20) to the upper surface of the object 14 is obtained as a measured value. Since the height of the upper surface of the rectangular parallelepiped object 14 such as a cardboard box is constant, the measured value of the sensor 21 changes in steps as shown in FIG.

In step S31, the detection unit 222 monitors the change in the measured value of the sensor 21. When the measured value changes (decreases) significantly as shown in FIG. 4, the detection unit 222 determines that the object 14 to be measured has entered the measurement area. Any method may be used to determine whether or not the measured value has changed significantly. For example, it may be determined whether or not the differential of the measured value is below the threshold value, or the average of the measured values of the last n times (n is an integer of 1 or more) and the measured value corresponding to the reference plane R stored in advance (n). It may be determined whether or not the difference from the reference distance) exceeds the threshold value.

Steps S30 and S31 are processes for monitoring the entry of the object 14 into the measurement area, and are repeatedly executed until the object 14 is detected. The repetition period may be appropriately set according to the transport speed of the object 14, the size of the object 14, and the like.

When the object 14 is detected by the detection unit 222, the image acquisition unit 220 captures an image from the camera 20 in step S32. FIG. 5A is an example of an image captured from the camera 20. It can be seen that an image of the object 14 viewed from above is obtained. It should be noted that the objects 14 do not have to be arranged in an aligned manner on the conveyor, and may be arranged at an arbitrary position and orientation as shown in FIG. 5A.

In step S33, the extraction unit 223 extracts the object 14 from the image. Any algorithm may be used for object extraction. For example, template matching, object detection by neural network, moving object detection by background subtraction method, contour detection, and the like can be preferably used. For example, when the precondition that the object 14 to be measured has a rectangular parallelepiped (box) shape is satisfied, the shape of the object 14 shown in the image is limited to a square shape. In the present embodiment, under such a premise, the extraction unit 223 extracts the outer shape of the object 14 by detecting the four vertices of the rectangle by the corner detection method. This method has the advantages that the processing is light and that the object 14 is robust with respect to the rotation, size, and shape. Therefore, it is suitable for applications such as measuring a large number of randomly arranged objects 14 on a conveyor in real time. FIG. 5B shows an example of the extraction result of the object 14. Points P1 to P4 are four vertices on the upper surface of the object 14.

In step S34, the size estimation unit 224 estimates the actual size of the object 14 in the XY plane (in the plane parallel to the image).

FIG. 6 shows the principle of size estimation. Here, assuming a rectangular parallelepiped object 14, the actual size is represented by a width W [cm], a length L [cm], and a height H [cm]. The height H is the dimension of the side parallel to the Z direction, and the width W and the length L are the dimensions of the two sides of the upper surface (rectangle) of the object 14. Further, the size on the image is represented by a width w [pixel] and a length l [pixel]. The two objects shown in FIG. 6 have the same width W and length L, but differ only in height H (H1 <H2). As shown in FIG. 6, if the height H of the object is different, the distance D from the camera 20 to the upper surface of the object changes (D1> D2), so even if the object has the same length L [cm], it can be seen on the image. The length l [pixel] of is different (l1 <l2). That is, the size on the image does not accurately represent the actual size. Therefore, the size estimation unit 224 obtains a conversion ratio of the length [pixel] on the image and the length [cm] in the real space based on the measurement distance D to the upper surface of the object obtained by the sensor 21. If this conversion ratio is known, the width W [cm] and the length L [cm], which are the actual sizes of the object 14, are calculated from the width w [pixel] and the length l [pixel] of the object 14 on the image. be able to. Not limited to the width W and the length L, the size in the XY plane can be calculated by the same method (for example, the length of the diagonal line on the upper surface of the object).

Any method can be used to obtain the conversion ratio between the length on the image and the length on the real space. For example, the measurement of the object 14 is performed with reference to the table 70 in which the distance D from the camera to the object and the conversion ratio of the length on the image and the length in the real space are associated with each other as shown in FIG. 7A. The conversion ratio according to the value may be acquired. The table 70 may be created from the experimental results obtained by measuring a plurality of samples having different heights, or may be created from the simulation results. Further, as shown in FIG. 7B, a model 71 in which the correlation between the value of the distance from the camera to the object and the conversion ratio of the length on the image and the length in the real space may be learned may be used. When the measured value of the object 14 is input to this model 71, a conversion ratio corresponding to the measured value is obtained. This model 71 may be generated by machine learning using experimental results obtained by measuring a plurality of samples having different heights, or may be created by simulation. Alternatively, the length on the image may be converted to the length in real space by geometric calculation using the value of the distance from the camera to the object and the parameters of the camera without using a table or a model.

In step S35, the size estimation unit 224 estimates the actual size of the object 14 in the Z direction (depth direction of the image). Specifically, the size estimation unit 224 calculates the height H [cm] of the object 14 from the difference between the measured value of the object 14 (the distance from the camera to the upper surface of the object) and the reference distance. The "reference distance" is the distance from the camera 20 to the reference plane R. The value of the reference distance may be set based on the measured value of the sensor 21 when no object exists in the measurement area, or may be manually input by the user. The installation height of the camera 20 (camera 20 and reference plane) based on the measured value of the sensor 21 when there is no object in the measurement area when the positions of the camera 20 and the sensor 21 are changed or when the device is maintained. The distance between R) may be automatically calibrated. Convenience can be improved by automating the setting of the reference distance.

In step S36, the output unit 225 outputs the measurement result. The measurement result may include information on at least one of the width W, the length L, and the height H of the object 14. Further, the measurement result may include information such as an image of the camera 20, a measured value of the sensor 21, a value of a reference distance, and a width w and a length l on the image. The output destination of the measurement result can be appropriately selected according to the purpose and purpose of the measurement result, such as the above-mentioned host system, display device, and terminal carried by the operator.

According to the configuration of the present embodiment described above, if the object exists in the measurement area (for example, within the field of view of the camera and the sensor), the actual size can be measured without strict positioning. Therefore, for example, a large number of objects are conveyed by a transfer device such as a conveyor, and the actual size of each object is measured while sequentially passing through the measurement area. Therefore, the actual size of a large number of objects can be measured efficiently and at high speed. Can be done. Further, since the height H of the object is calculated from the distance information obtained by the sensor, the height of the object is calculated faster and more accurately than other methods (for example, a method of calculating the depth from the image of the camera). The height can be calculated. Further, since the image acquisition and the size estimation process are performed only when necessary by using the detection of the object 14 by the detection unit 222 as a trigger, resources such as a CPU and a memory can be efficiently used.

<Second Embodiment>
In the second embodiment, the detection unit 222 determines "overlap of objects" based on the change in the measured value of the sensor 21. Hereinafter, the parts different from those of the first embodiment will be mainly described, and the description of the same configuration as that of the first embodiment will be omitted.

As shown in FIG. 8A, when the two objects 14a and 14b are arranged apart, the measured value of the sensor 21 changes as shown in FIG. 8B. On the other hand, as shown in FIG. 9A, when the two objects 14a and 14b are vertically overlapped, the measured value of the sensor 21 changes as shown in FIG. 9B. As can be seen from FIGS. 8B and 9B, when the objects 14a and 14b are separated from each other, the measured value is the value Va corresponding to the height Ha of the object 14a from the reference value Vr (measured value when the object does not exist). After returning to the reference value Vr again, the value decreases to the value Vb corresponding to the height Hb of the object 14b, whereas when the objects 14a and 14b overlap, the measured value is the reference value. The value Vr is gradually decreased from the first value Va to the second value (Va + Vb-Vr). Here, the first value Va represents the distance from the camera 20 to the upper surface of the lower object 14a, and the second value (Va + Vb-Vr) represents the distance from the camera 20 to the upper surface of the upper object 14b. Represent. Under the precondition that the object to be measured has a rectangular parallelepiped shape, when a change in the measured value as shown in FIG. 9B is observed, it can be considered that the two objects overlap.

When the detection unit 222 detects the overlap of a plurality of objects, the size estimation unit 224 may execute a process (exception process) different from that of the first embodiment. As exception handling, for example, the following processing can be performed.

(1) On the first object 14a of the height Ha corresponding to the first value Va, the second of the height Hb corresponding to the difference between the first value Va and the second value (Va + Vb-Vr). A process of estimating the actual size of each of the first object 14a and the second object 14b, assuming that the objects 14b of the first object 14b overlap.

(2) A process of controlling to notify that the actual size of a plurality of objects cannot be estimated.

The exception handling (1) has an advantage that the actual size of each of the objects being transported in an overlapping manner can be automatically measured. However, depending on the scene, such as when it is difficult to distinguish the boundary between objects from the image because the colors of the objects are very similar, the accuracy of estimating the actual size of the overlapping objects may be significantly reduced. .. In such a case, it is preferable to execute the exception handling (2). Then, it can be expected that the operator who notices the notification can measure the actual size of each object with higher accuracy by taking measures such as eliminating the overlap of the objects.

<Modification example>
The above-described embodiment is merely an example of a configuration example of the present invention. The present invention is not limited to the above-mentioned specific form, and various modifications can be made within the scope of its technical idea. For example, in the above embodiment, a rectangular parallelepiped object is illustrated, but the shape to be measured is not limited to the rectangular parallelepiped. If the shape of the object appearing in the image is not square, the actual size may be calculated assuming the circumscribed rectangle or bounding box of the object. If the height of the upper surface of the object is not constant, the sensor 21 may measure the distances at a plurality of locations on the upper surface, and the height of the object may be calculated from the statistical values thereof.

<Appendix 1>
(1) A camera (20) for photographing an object (14) existing in the measurement area (13), and
A sensor (21) that measures the distance from the camera (20) to the object (14),
It has an information processing device (22) and
The information processing device (22)
An extraction unit (223) that extracts the object (14) from the image obtained by the camera (20), and
A size measuring device including a size estimating unit (224) that estimates the actual size of the object (14) extracted from the image based on the measurement distance measured by the sensor (21). (10).

10: Size measuring device 11: Conveyor device (conveyor)
12: Upper system 13: Measurement areas 14, 14a, 14b: Object 20: Camera 21: Sensor 22: Information processing device 220: Image acquisition unit 221: Distance information acquisition unit 222: Detection unit 223: Extraction unit 224: Size estimation unit 225: Output unit R: Reference plane

Claims (15)

A camera that captures objects in the measurement area,
A sensor that measures the distance from the camera to the object,
Has an information processing device,
The information processing device
An extraction unit that extracts the object from the image obtained by the camera,
A size measuring device comprising a size estimating unit that estimates the actual size of the object extracted from the image based on the measurement distance measured by the sensor. The size estimation unit is characterized by calculating the actual size of the object in a plane parallel to the image by converting the length on the image into the length in the real space based on the measurement distance. The size measuring device according to claim 1. The size estimation unit uses a table in which the value of the distance from the camera to the object and the conversion ratio of the length on the image and the length in the real space are associated with each other, and the length on the image is used. The size measuring device according to claim 2, wherein the image is converted into a length in real space. The size estimation unit uses a model that learns the correlation between the value of the distance from the camera to the object and the conversion ratio of the length on the image and the length in the real space on the image. The size measuring device according to claim 2, wherein the length is converted into a length in real space. The size estimation unit is characterized in that the length on the image is converted into the length in the real space by geometric calculation using the value of the distance from the camera to the object and the parameters of the camera. The size measuring device according to claim 2. One of claims 1 to 5, wherein the size estimation unit calculates the actual size of the object in the depth direction of the image from the difference between the measurement distance and the preset reference distance. The size measuring device described in. The size measuring device according to claim 6, wherein the size estimating unit sets the reference distance based on a value measured by the sensor when the object does not exist in the measuring area. The size measuring device measures the size of an object conveyed so as to pass through the measuring area.
Of claims 1 to 7, the information processing device further includes a detection unit that monitors the measured value of the sensor and detects the entry of an object into the measurement area by changing the measured value. The size measuring device according to any one item. The size measuring device according to claim 8, wherein the information processing device acquires an image used for estimating the actual size of the object, triggered by the detection of the entry of the object by the detection unit. .. In the detection unit, the measured value of the sensor is a first value smaller than the reference value and a second value smaller than the first value from the reference value obtained when no object exists in the measurement area. The size measuring device according to claim 8 or 9, wherein it is determined that a plurality of objects are overlapped when the value gradually changes to the value of. When the detection unit detects the overlap of a plurality of objects,
In the size estimation unit, a second object having a height corresponding to the difference between the first value and the second value is superimposed on the first object having a height corresponding to the first value. The size measuring device according to claim 10, wherein the actual size of each of the first object and the second object is estimated. When the detection unit detects the overlap of a plurality of objects,
The size measuring device according to claim 10, wherein the size estimating unit controls to notify that the actual size of the plurality of objects cannot be estimated. The size measuring device according to any one of claims 1 to 12, wherein the sensor is an infrared sensor. The steps to acquire an image from a camera that captures an object in the measurement area,
The step of extracting the object from the image and
A step of measuring the distance from the camera to the object by a sensor, and
A size measuring method comprising a step of estimating the actual size of the object extracted from the image based on the measurement distance measured by the sensor. A program for causing a processor to execute each step of the size measuring method according to claim 14.

Images