WO2022138715A1

WO2022138715A1 - Load capacity rate measuring device, system, method, and program

Info

Publication number: WO2022138715A1
Application number: PCT/JP2021/047556
Authority: WO
Inventors: 悟己上野; 教之青木; 真則高岡; 研二河野; ゆり安達
Original assignee: 日本電気通信システム株式会社
Priority date: 2020-12-25
Filing date: 2021-12-22
Publication date: 2022-06-30
Also published as: JP7491615B2; JPWO2022138715A1

Abstract

This load capacity rate measuring device: acquires a point group existing in a monitored area; creates array data in which voxels, among voxels in a voxel grid obtained by dividing the monitored area into voxels having a voxel side length, in which at least a prescribed number of points from the point group are present are defined as valid voxels; calculates blind spot invalid voxel position information on the basis of imaging unit attachment position information, angle of view information relating to an imaging unit, the monitored area, and the voxel side length; creates estimated array data obtained by estimating valid voxels in a state in which an occlusion part, in which an obstacle at the front is hiding a measurement target object to the rear, has been removed, on the basis of the plurality of items of array data; and, if a prescribed condition is satisfied, sets all blind spot invalid voxels to the left and right or below the frontmost valid voxel in the estimated array data to be valid voxels.

Description

Floor area ratio measuring device, system, method, and program

[Description of related applications]
The present invention is based on the priority claim of Japanese patent application: Japanese Patent Application No. 2020-216527 (filed on December 25, 2020), and all the contents of the application are incorporated in this document by citation. It shall be.
The present invention relates to a floor area ratio measuring device, a system, a method, and a program.

In some transportation companies, in order to improve transportation efficiency, containers (including box-shaped loading platforms such as van bodies and wing bodies) are sequentially loaded from the back side to the front side at the truck berth, and the loading volume ratio is 100. I try to transport it in a state close to%. At the time of unloading, all the cargo is moved from the container to the inner warehouse at the truck berth so that the loading volume ratio is 0%. Furthermore, the loaded floor area ratio is measured visually by the driver, and the measured load capacity ratio is reported from the driver to the manager.

However, it is difficult to grasp an accurate value because individual differences are likely to occur in the visual measurement of the loaded floor area ratio. In addition, when the cargo is being loaded by the preceding vehicle, another waiting vehicle may exist in the truck berth, and the report of the loaded floor area ratio of the preceding vehicle has completed the loading of the luggage of the preceding vehicle. Since it is just before the departure after the departure, the movement of the waiting vehicle starts after the departure of the preceding vehicle, and the report of the loaded floor area ratio causes a time loss in the vast truck berth.

By automatically monitoring the floor area ratio from these events, it is possible to eliminate individual differences due to the measurement of the floor area ratio and the trouble of reporting the measured floor area ratio, and the floor area ratio is 100% for standby vehicles. It is desirable to be able to guide the position of the preceding vehicle near by in advance.

As a technique for automatically monitoring the load volume ratio, as in Patent Documents 1 to 4, the point cloud data (three-dimensional data) photographed or scanned by the photographing units (cameras, sensors) at a plurality of points is used to load the luggage. It is conceivable to apply a technique for recognizing volume.

Japanese Patent No. 6577767 Japanese Patent No. 6511681 Japanese Unexamined Patent Publication No. 2020-60451 Japanese Unexamined Patent Publication No. 2017-175483

The following analysis is given by the inventor of the present application.

However, in the techniques described in Patent Documents 1 to 4, since a plurality of photographing units are required to recognize the volume of the luggage, if a plurality of photographing units are used, the space for installing the photographing units increases. , The space in the container where luggage cannot be placed increases, which may reduce economic efficiency. Further, even if a plurality of photographing units are used, it may not be possible to compensate for the blind spots of each other depending on the specifications and installation positions of the photographing units.

Further, the techniques described in Patent Documents 1 to 4 only recognize the shape, edge, and dimensions of the luggage in the front row that can be photographed, and the condition of the entire luggage in the container (particularly, the condition of the luggage in a place where photography cannot be performed). May not be recognized accurately.

Further, in the techniques described in Patent Documents 1 to 4, since it is premised that at least point cloud data can be acquired within the angle of view, it becomes impossible to take a picture when a baggage approaches a close distance to the taking part, and the taking part cannot be taken. It may not be possible to accurately recognize the status of luggage at close range. In other words, unlike analog cameras and digital cameras that shoot non-point cloud data, a shooting unit such as a 3D sensor that shoots point cloud data has restrictions on the mechanism for distance measurement such as ToF (Time of Flight) and stereo. Therefore, it is not possible to shoot at a short distance of several tens of centimeters to 1 m due to the specifications. In particular, if there are multiple shooting units, it is easy for luggage to approach the shooting unit at a close distance, and the blind spots of one shooting unit cannot be covered by the other shooting units, so luggage at a close distance to the shooting unit. It is more likely that you will not be able to accurately recognize the situation.

Further, in the technique described in Patent Document 3, when the baggage is in the close distance of the sensor, the baggage in the blind spot portion of the sensor is uniformly interpolated, so that the baggage in the closest distance from the sensor and the baggage in the vicinity thereof are uniformly interpolated. On the same side, the load capacity of the load is measured by interpolating the impossible load, and there is a possibility that the difference between the measured load capacity and the actual load capacity will increase.

A main object of the present invention is to contribute to suppressing the discrepancy between the measured load capacity and the actual load capacity of the load on a volume basis without using a plurality of image pickup units even if the load is close to the photographing unit. It is to provide loading volume ratio measuring devices, systems, methods, and programs that can be used.

The loading volume ratio measuring device according to the first viewpoint acquires a group of points in the designated monitoring area in which the area where the measurement object is loaded in the predetermined space is present in the photographing data obtained by photographing the predetermined space by the photographing unit. Among the voxels in the voxel grid obtained by dividing the monitoring area by the voxel of the voxel side length specified in advance and the monitoring area designation unit configured to do so, a voxel having a predetermined number of points or more in the point group is valid. A voxelized unit configured to create voxelized array data, a photographing unit mounting position information relating to a position where the photographing unit is attached, an image angle information relating to an image angle of the photographing unit, the monitoring area, and the like. And, based on the voxel side length, in the voxel grid in the array data, the blind spot invalid voxel position information related to the position of the invisible blind spot invalid voxel of the occlusion portion which is the blind spot of the photographing unit is calculated. Based on the angle occlusion voxel calculation unit configured in, and the plurality of said sequence data within the predetermined interval time, the obstacle in the foreground hides the object to be measured behind and hides the occlusion. Estimated sequence data that estimates the effective voxels in the state where the portion is excluded is created, and when the predetermined conditions are satisfied, all the blind spot invalid voxels in the left-right direction or the downward direction of the frontmost effective voxels in the estimated sequence data. An array processing unit configured to create loading array data that enables the valid voxels and enables all the invalid voxels in the depth direction of all the valid voxels after the activation to the valid voxels. Based on the loading arrangement data, the loading volume ratio calculation unit configured to calculate the loading volume ratio, which is the ratio of the volume of the measurement object loaded in the monitoring area to the total volume in the monitoring area. And.

The loaded floor area ratio measuring system according to the second viewpoint includes a photographing unit configured to output imaging data obtained by photographing a predetermined space, and a loaded floor area ratio measuring device according to the first viewpoint.

The loading volume ratio measuring method according to the third viewpoint is a loading volume ratio measuring method for measuring the loading volume ratio using hardware resources, and is in the predetermined space among the photographing data obtained by photographing the predetermined space by the photographing unit. Of the voxels in the voxel grid, the step of acquiring a point group in which the area where the measurement object is loaded exists in the specified monitoring area and the voxels in which the monitoring area is divided by voxels having a predetermined voxel side length. A step of creating array data using voxels having a predetermined number of points or more as effective voxels, imaging unit mounting position information related to the position where the photographing unit is attached, and an image angle related to the image angle of the photographing unit. Based on the information, the monitoring area, and the voxel side length, the blind spot invalid voxel relating to the position of the invisible blind spot invalid voxel of the occlusion portion which is the blind spot of the photographing unit in the voxel grid in the array data. Based on the step of calculating the position information and the plurality of said sequence data within the predetermined interval time, the occlusion part where the obstacle in the foreground hides the measurement object behind and makes it invisible is excluded. Estimated sequence data that estimates the valid voxels of the state is created, and when a predetermined condition is satisfied, all the blind spot invalid voxels in the left-right direction or the downward direction of the previous valid voxels in the estimated sequence data are used as valid voxels. The monitoring area is based on the step of creating the loading array data in which all the invalid voxels in the depth direction of all the enabled and enabled voxels are enabled in the valid voxels, and the loading array data. Includes a step of calculating the loading volume ratio, which is the ratio of the volume of the measurement object loaded in the monitoring area to the total volume of the inside.

The program according to the fourth viewpoint is a program for causing hardware resources to execute a process of measuring the load capacity ratio, and the measurement target is loaded in the predetermined space among the shooting data taken by the photographing unit in the predetermined space. Of the voxels in the voxel grid in which the monitoring area is divided by the voxels having the specified voxel side length and the process of acquiring the point group existing in the specified monitoring area, the score of the point group is predetermined. Processing to create array data using a number of voxels as effective voxels, imaging unit mounting position information related to the position where the imaging unit is attached, image angle information related to the image angle of the imaging unit, the monitoring area, And, based on the voxel side length, in the voxel grid in the array data, the process of calculating the blind spot invalid voxel position information related to the position of the invisible blind spot invalid voxel of the occlusion portion which is the blind spot of the photographing unit. And, based on the plurality of said sequence data within the predetermined interval time, the effective voxel in the state where the obstacle in the foreground hides the measurement object behind and hides the occlusion part is estimated. When the estimated sequence data is created and the predetermined conditions are satisfied, all the blind spot invalid voxels located in the left-right direction or the downward direction of the front valid voxel in the estimated sequence data are enabled and valid as valid voxels. The process of creating loading sequence data in which all invalid voxels in the depth direction of all the valid voxels after conversion are enabled for valid voxels, and the process of creating the loading sequence data for the total volume in the monitoring area based on the loading sequence data. The process of calculating the loaded volume ratio, which is the ratio of the volume of the measured object loaded in the monitoring area, is performed by the hardware resource.

The program can be recorded on a storage medium that can be read by a computer. The storage medium may be a non-transient such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. Further, in the present disclosure, it is also possible to embody it as a computer program product. The program is input to a computer device via an input device or an external communication interface, stored in a storage device, drives a processor according to a predetermined step or process, and steps the processing result including an intermediate state as necessary. Each can be displayed via a display device, or can communicate with the outside via a communication interface. Computer devices for that purpose typically include, for example, a processor, a storage device, an input device, a communication interface, and, if necessary, a display device that can be connected to each other by a bus.

According to the first to fourth viewpoints, even if there is a load in a close distance to the photographing unit, it is possible to suppress the discrepancy between the measured load capacity and the actual load capacity on a volume basis without using a plurality of photographing units. Can contribute to.

It is a block diagram which shows typically the structure of the loading volume ratio measurement system which concerns on Embodiment 1. It is a block diagram which shows typically the structure of the loading volume ratio measuring apparatus in the loading volume ratio measuring system which concerns on Embodiment 1. FIG. It is an image diagram schematically showing the imageable range of the photographing unit, (A) is a perspective view, (B) is a view seen from the arrow D1 of FIG. 3 (A), and (C) is a view of FIG. 3 (A). It is a figure seen from the arrow view D2 of. It is an image diagram schematically showing an example of a situation in which a baggage in a container is photographed by a photographing unit. FIG. 3A is a view seen from the arrow D1 of FIG. 3A, and FIG. It is a figure seen from the arrow view D2 of. It is an image diagram of an example of a point cloud when the luggage in the container is photographed by the photographing unit, (A) is a view seen from the arrow view D1 of FIG. 3 (A), and (B) is an arrow of FIG. 3 (A). The view seen from the view D2, (C) is the view seen from the arrow view D3 of FIG. 3 (A). It is an image diagram seen from the arrow view D2 of FIG. 3A schematically showing the range that can be visualized from the image data taken by the photographing unit. It is an image diagram schematically showing the change of the visualization information of the baggage that can be visualized by the photographing unit in the process of loading the baggage in the container. It is an image diagram of the 1st example which schematically showed the flow from the photographing of the loading volume ratio measuring apparatus in the loading volume ratio measuring system which concerns on Embodiment 1 to voxel interpolation. FIG. 5 is an image diagram of a second example schematically showing a flow from photographing of the floor area ratio measuring device in the floor area ratio measuring system according to the first embodiment to voxel interpolation. FIG. 3 is an image diagram of a third example schematically showing a flow from photographing of the loaded floor area ratio measuring device in the loaded floor area ratio measuring system according to the first embodiment to voxel interpolation. It is a flowchart which schematically shows the operation of the loading volume ratio measuring apparatus in the loading volume ratio measuring system which concerns on Embodiment 1. FIG. It is a flowchart which schematically showed the detailed operation of the step A7 of FIG. 11 of the loading volume ratio measuring apparatus in the loading volume ratio measuring system which concerns on Embodiment 1. FIG. It is a block diagram which shows typically the structure of the loading volume ratio measuring apparatus which concerns on Embodiment 2. It is a block diagram which shows the structure of a hardware resource schematically.

Hereinafter, embodiments will be described with reference to the drawings. It should be noted that the reference numerals to the drawings in the present application are provided solely for the purpose of assisting understanding, and are not intended to be limited to the illustrated embodiments. Further, the following embodiments are merely examples, and do not limit the present invention. Further, the connecting line between blocks such as drawings referred to in the following description includes both bidirectional and unidirectional. The one-way arrow schematically shows the flow of the main signal (data), and does not exclude bidirectionality. Further, in the circuit diagram, block diagram, internal configuration diagram, connection diagram, etc. shown in the disclosure of the present application, although not explicitly stated, an input port and an output port exist at the input end and the output end of each connection line, respectively. The same applies to the input / output interface. The program is executed via the computer device, the computer device comprises, for example, a processor, a storage device, an input device, a communication interface, and a display device as needed, and the computer device is in the device or through the communication interface. It is configured to be able to communicate with external devices (including computers) regardless of whether it is wired or wireless.

[Embodiment 1]
The loaded floor area ratio measurement system according to the first embodiment will be described with reference to the drawings. FIG. 1 is a block diagram schematically showing the configuration of the loaded floor area ratio measurement system according to the first embodiment. FIG. 2 is a block diagram schematically showing the configuration of the floor area ratio measuring device in the floor area ratio measuring system according to the first embodiment.

The floor area ratio measurement system 1 is a system that measures (measures) the floor area ratio (or the load capacity) of the luggage 20 (measurement target) in the container 11 (corresponding to the monitoring area) of the truck 10 (Fig.). 1). The floor area ratio measurement system 1 can be used to manage transportation in the transportation field by truck, aircraft, ship, etc., or in the factory field. The loaded floor area ratio measuring system 1 has a configuration in which the photographing unit 300 and the loaded floor area ratio measuring device 200 are communicably connected via the network 400. In the loaded floor area ratio measuring system 1, the loaded floor area ratio measuring device 200 measures the loaded floor area ratio of the luggage 20 in the container 11 of the truck 10 based on the photographed data (100 in FIG. 2) photographed by the photographing unit 300. The loaded floor area ratio measuring system 1 quantifies the loaded floor area ratio according to the loading status or the unloading status of the luggage 20 of the container 11. The loaded floor area ratio measuring system 1 displays the measured numerical value of the loaded floor area ratio together with visualization information (see FIG. 7) that visualizes the photographing data 100.

The photographing unit 300 is a functional unit that senses and photographs the surface of the luggage 20 in the container 11 of the truck 10 (see FIG. 1). The photographing unit 300 is connected to the loaded floor area ratio measuring device 200 via a network 400 so that wireless communication is possible (wired communication is also possible). The photographing unit 300 outputs the photographing data (100 in FIG. 2) of the luggage 20 toward the loading floor area ratio measuring device 200. The shooting data 100 is point cloud data (three-dimensional data) drawn with a point cloud (three-dimensional coordinates of a large number of points). The photographing unit 300 can be selected according to the environmental conditions such as the photographing distance, the angle of view, indoor / outdoor, and the presence / absence of sunshine required for measuring the loaded floor area ratio, and the customer's request. Further, products sold by various manufacturers can be used for the photographing unit 300. As the photographing unit 300, for example, a stereo camera, a ToF (Time of Flight) camera, 3D-LIDAR (Three Dimensions-Laser Imaging Detection and Ringing), or the like can be used. The photographing unit 300 is attached to a position where the luggage 20 in the container 11 of the truck 10 can be photographed (can be installed), for example, for attaching to the container 11 or for loading and unloading the luggage between the truck 10 and the warehouse. It can be attached to the ceiling or wall surface of a truck berth that is in contact with the truck 10. If one unit can cover the monitoring area, one photographing unit 300 may be used, and if the monitoring area has a plurality of locations or a plurality of directions, there may be a plurality of units.

The network 400 is a wired or wireless communication network that communicably connects between the photographing unit 300 and the loaded floor area ratio measuring device 200. For the network 400, for example, a communication network such as PAN (Personal Area Network), LAN (Local Area Network), MAN (Metropolitan Area Network), WAN (Wide Area Network), GAN (Global Area Network) can be used. ..

The loaded floor area ratio measuring device 200 is a device that measures the loaded floor area ratio of the luggage 20 in the container 11 of the truck 10 based on the photographing data 100 of the predetermined space photographed by the photographing unit 300 (see FIGS. 1 and 2). ). Here, the loaded floor area ratio means the ratio (percentage) of the volume of the loaded luggage 20 to the total volume in the container 11. The loaded floor area ratio measuring device 200 has a function of visualizing the luggage 20 in the container 11. The loaded floor area ratio measuring device 200 is used by an observer. As the loaded floor area ratio measuring device 200, a device (computer device) having a functional unit (for example, a processor, a storage device, an input device, a communication interface, and a display device) constituting the computer can be used. As the loaded floor area ratio measuring device 200, for example, hardware resources such as a computer, a personal computer, a notebook personal computer, and a tablet terminal can be used. The load volume ratio measuring device 200 realizes a pretreatment unit 210, a load volume management unit 220, a result generation unit 230, and an interface unit 240 by executing a predetermined program (software).

The pre-processing unit 210 is a functional unit that performs predetermined pre-processing (here, format conversion and noise removal) on the photographing data 100 from the photographing unit (300 in FIG. 1) (see FIG. 2). The preprocessing unit 210 preprocesses the imaging data 100 from the imaging unit (300 in FIG. 1), and preprocesses the preprocessed imaging data 100 into the monitoring area designation unit 221 of the loading volume management unit 220 and the interface unit 240. Is output to the display unit 242 of. The preprocessing unit 210 includes a format conversion unit 211 and a noise removal unit 212.

The format conversion unit 211 is a functional unit that converts the format of the photographing data 100, which differs depending on the type of the photographing unit (300 in FIG. 1), into a common format that can be commonly used in the loaded floor area ratio measuring device 200 (see FIG. 2). The format conversion unit 211 outputs the converted common format shooting data 100 toward the noise removal unit 212.

The noise removing unit 212 is a functional unit that removes noise (point cloud unnecessary for measurement) from the point cloud in the shooting data 100 from the format conversion unit 211 (see FIG. 2). The noise removing unit 212 outputs the noise-removed shooting data 100 toward the monitoring area designation unit 221 of the loading volume management unit 220 and the display unit 242 of the interface unit 240. Examples of the noise removing method include smoothing processing, filtering (for example, moving average filtering processing, median filtering processing, etc.), outlier removal processing (for example, outlier removal processing by chi-square test), and the like.

The loading volume management unit 220 is a functional unit that manages the loading volume of the load (20 in FIG. 1) in the container (11 in FIG. 1) (see FIG. 2). The loading volume management unit 220 creates and created the loading arrangement data 112 based on the shooting data 100 from the noise removing unit 212 of the preprocessing unit 210 and the operation information from the operation unit 241 of the interface unit 240. The loading arrangement data 112 is output to the loading volume ratio calculation unit 232 and the voxel visualization unit 233 of the result generation unit 230. The loading volume management unit 220 includes a monitoring area designation unit 221, a voxelization unit 222, an array processing unit 223, an array data group storage unit 224, an angle of view occlusion voxel calculation unit 225, and an imaging unit specification data storage unit. 226 and.

The monitoring area designation unit 221 is a monitoring area (for example, a space in the container 11) designated by the user using the operation unit 241 of the interface unit 240 in the shooting data 100 from the noise removal unit 212 of the preprocessing unit 210. ) Is a functional unit that acquires a point cloud (see FIG. 2). Here, the monitoring area means an area for monitoring a change in the loading volume of the luggage 20 (measurement object), and the luggage 20 (measurement object) in the predetermined space among the shooting data obtained by shooting the predetermined space by the photographing unit 300. ) Is the area where it is loaded. The monitoring area designation unit 221 outputs the point cloud in the monitoring area of the acquired shooting data 100 toward the voxelization unit 222.

The voxelization unit 222 is a functional unit that creates array data 111 in which the point cloud in the monitoring area of the shooting data 100 acquired by the monitoring area designation unit 221 is voxelized (processed into a voxel composed of cubes) (). See Figure 2). The voxel-forming unit 222 uses the operation unit 241 of the interface unit 240 to convert the monitoring area of the shooting data 100 acquired by the monitoring area designation unit 221 into a voxel side length (one side of the voxel related to the cube). Create a virtual voxel grid (three-dimensional array of voxels) divided by voxels of length). It should be noted that the voxel side length can be suppressed from an error by appropriately adopting a length sufficiently smaller than the minimum dimension of the luggage 20. Here, the point cloud in the monitoring area of the shooting data 100 acquired by the monitoring area designation unit 221 is mainly only the part on the shooting unit 300 side (front side) of the luggage 20 (the point cloud is dense) due to the problem of resolution. The part that has become blank) is acquired, and the part in the back direction becomes blank (the part where the point cloud is sparse or absent). Therefore, the voxel-forming unit 222 can use voxels in the created voxel grid in which the number of points in the monitoring area of the photographing data 100 is a predetermined number (for example, three) or more (FIG. 8C). Refer to the valid voxels in FIGS. 9 (C) and 10 (C), and invalid voxels with voxels having a point group score less than a predetermined number (see FIGS. 8 (C), 9 (C), and 10 (C)). Create the array data 111 as (see invalid voxels). The effective voxel corresponds to the filled voxel of the “visualization information” in FIG. 7. The array data 111 may be associated with each voxel not only with binary information indicating whether each voxel is valid or invalid in the voxel grid, but also with information related to the score of the point cloud existing in each voxel, and is present in each voxel. Information on the volume according to the score of the point cloud may be associated with each voxel. The voxelization unit 222 outputs the created array data 111 to the array processing unit 223.

The array processing unit 223 is a functional unit that performs voxel array processing based on the array data 111 (see FIG. 2). The array processing unit 223 acquires sequence data 111 from the voxelization unit 222, and acquires blind spot invalid voxel position information from the angle of view occlusion voxel calculation unit 225. Based on the acquired sequence data 111 and blind spot invalid voxel position information, the array processing unit 223 arranges for the interval time at intervals of the interval time specified by the user using the operation unit 241 of the interface unit 240. An array data group 110 is created by arranging the data 111 in chronological order and associating them with each other. The array processing unit 223 stores the created array data group 110 in the array data group storage unit 224.

The array processing unit 223 compares each array data 111 of the array data group 110 stored in the array data group storage unit 224, so that an obstacle (worker, trolley, luggage in the middle of loading, etc.) in front is behind. Estimated sequence data for estimating the effective box cell in a state where the occlusion portion that hides and hides the loaded luggage 20 in the above is created for each sequence data group 110.

Here, in the creation of the estimated sequence data, for example, the positions in the left-right direction (for example, the X-axis direction) and the vertical direction (for example, the Y-axis direction) are compared between the sequence data 111 in one sequence data group 110. If there are effective boxels that are the same and are at different positions in the front-back direction (for example, in the Z-axis direction) or at the same position, the innermost part of those effective boxels (the positions in the left-right direction and the up-down direction are the same and the front-back direction is the same). You can select the effective boxel in the innermost part).

Further, in the creation of estimated sequence data, for example, when the number of invalid voxels in a voxel at a specific position is larger than the number of valid voxels in comparison between the sequence data 111 in one sequence data group 110, the specific position is concerned. If the number of valid voxels in a specific position is larger than the number of invalid voxels, the voxel in the specific position can be regarded as a valid voxel.

Note that the estimation sequence data may be created not only by comparison between the sequence data 111, but also by using labeling, machine learning, or the like. The sequence processing unit 223 may hold the created estimated sequence data.

The array processing unit 223 creates the loading array data 112 based on the retained estimation array data.

Here, in the creation of the loading array data 112, in addition to the conventional array data 111 and the array data group 110 for referring to the past data, the blind spot invalid voxel position information from the angle occlusion voxel calculation unit 225 is also referred to. Estimate the invalid voxels (blind spot invalid voxels) of the occlusion part that becomes the blind spot. The blind spot invalid voxel is estimated based on the condition such as whether the blind spot invalid voxel exists in the voxel grid in the past or present sequence data up to the portion tangent to the occlusion portion which becomes the blind spot. The estimated blind spot invalid voxel is enabled (interpolated) into a valid voxel if a predetermined condition is satisfied. For example, if the part with the valid voxel in the past data is the invalid voxel and the valid voxel does not exist on the depth side in the direction away from the shooting unit 300, the blind spot invalid voxel on the front side that cannot be shot is enabled. do. It also activates all invalid voxels beneath valid voxels that appear to be floating in the air. Furthermore, from the viewpoint of preventing the load from collapsing, the load is usually loaded from the left and right wall sides, so if there is an effective voxel in the front central part and there is a blind spot invalid voxel in the left and right direction, the left and right direction Enable all blind spot disabled voxels.

Further, in the creation of the loading arrangement data 112, the point cloud in the monitoring area of the photographing data 100 acquired by the monitoring area designation unit 221 is mainly only the sensor side (front side) portion of the luggage 20 (the point cloud is dense). Since the part in the back direction is blank (the part where the point cloud is sparse or absent), it is necessary to interpolate the part in the back direction. Therefore, as one method of interpolation of the portion in the back direction, the array processing unit 223 is all located on the back side (the same position in the left-right direction and the up-down direction and the back side in the front-back direction) from the effective voxel in the estimated array data. It is possible to interpolate so that the voxels of are uniformly valid voxels. The interpolation here is based on the premise that the luggage 20 is basically not packed with a gap.

Further, as another method of interpolation of the portion in the back direction, the array processing unit 223 is an invalid voxel in the latest estimated voxel data for the voxel at a specific position, but is a valid voxel in the past estimated voxel data. If so, it can be interpolated so that the voxel at the specific position is a valid voxel. In addition, when the sequence processing unit 223 is a valid voxel in the latest estimated sequence data for the voxel at a specific position but is an invalid voxel in the past estimated sequence data, a new baggage 20 is added. As, a valid voxel can be selected for the voxel at the specific position. Further, when the latest estimated sequence data and the past estimated sequence data are both valid voxels for the voxel at the specific position, the array processing unit 223 assumes that the luggage 20 still exists and at the specific position. You can select valid voxels for voxels. Further, when the latest estimated sequence data and the past estimated sequence data are both invalid voxels for the voxel at the specific position, the array processing unit 223 considers that the baggage 20 does not exist from the beginning and determines that the specific position. You can select invalid voxels for voxels in. It should be noted that another method of interpolating the portion in the back direction may be performed in combination with one method of interpolation of the portion in the back direction.

The arrangement processing unit 223 outputs the created loading arrangement data 112 to the loading volume ratio calculation unit 232 and the voxel visualization unit 233 of the result generation unit 230. The arrangement processing unit 223 may hold the created loading arrangement data 112 and use the held loading arrangement data 112 when creating a new loading arrangement data 112.

The array data group storage unit 224 is a functional unit that stores the array data group 110 created by the array processing unit 223 in chronological order (see FIG. 2). The array data group storage unit 224 writes or reads the array data group 110 according to the operation of the array processing unit 223.

The angle of view occlusion voxel calculation unit 225 is a functional unit that calculates the position of invalid voxels in the occlusion portion caused by the angle of view of the shooting unit 300. The angle of view occlusion voxel calculation unit 225 acquires the imaging unit specification data 113 (imaging unit mounting position information and angle of view information) from the photographing unit specification data storage unit 226. The angle of view occlusion voxel calculation unit 225 acquires the monitoring area and voxel side length specified by the user by using the operation unit 241 of the interface unit 240. The angle of view occlusion voxel calculation unit 225 is an occlusion that becomes a blind spot of the imaging unit 300 in the voxel grid in the array data 111 based on the acquired imaging unit mounting position information, angle of view information, monitoring area, and voxel side length. The position information (blind spot invalid voxel position information) of the invalid voxel (see FIG. 6) whose portion cannot be visualized is calculated. The angle of view occlusion voxel calculation unit 225 outputs the calculated blind spot invalid voxel position information to the array data group storage unit 224.

The photographing unit specification data storage unit 226 is a functional unit that stores the photographing unit specification data 113 (see FIG. 2). The photographing unit specification data storage unit 226 reads out the photographing unit specification data 113 according to the operation of the angle of view occlusion voxel calculation unit 225. The imaging unit specification data 113 is preset data, and includes imaging unit mounting position information and image angle information. The photographing unit mounting position information is information relating to the position where the photographing unit 300 is attached. The photographing unit mounting position information can be, for example, information related to the coordinates to which the photographing unit 300 in the container 11 is attached. The angle of view information is information related to the horizontal angle of view, the vertical angle of view, and the swing angle (angle of the optical axis) of the photographing unit 300.

The result generation unit 230 is a functional unit that generates a result regarding the floor area ratio in chronological order based on the load arrangement data 112 from the arrangement processing unit 223 of the load volume management unit 220 (see FIG. 2). The result generation unit 230 includes a warning value designation unit 231, a load floor area ratio calculation unit 232, and a voxel visualization unit 233.

The warning value specifying unit 231 is a functional unit that designates a warning value specified by the user using the operation unit 241 of the interface unit 240 to the loading floor area ratio calculation unit 232 or the voxel visualization unit 233 (FIG. 2). reference). Here, the warning value is a threshold value of the condition for giving a warning. For example, when the warning value specifying unit 231 warns when the loaded floor area ratio is "90%" or more (for example, contacting a waiting vehicle of a truck berth), the warning value specifying unit 231 calculates the warning value related to "90%". Designated for unit 232. Further, the warning value specifying unit 231 warns when the change in height between the front luggage 20 and the back luggage 20 in the container 11 is "2 m" or more (for example, prevention of cargo collapse in the container 11). If so, the warning value related to "2m" is specified for the voxel visualization unit 233.

The loading floor area ratio calculation unit 232 is a functional unit that calculates the loading volume ratio based on the loading arrangement data 112 from the arrangement processing unit 223 of the loading volume management unit 220 (see FIG. 2). The loaded floor area ratio calculation unit 232 calculates the loaded floor area ratio in the process in which the cargo 20 is loaded in order from the back side of the container 11 or in the process in which the cargo 20 is unloaded in order from the front side. The loading volume ratio can be calculated, for example, by calculating the ratio (percentage) of the number of effective voxels to the total number of voxels (whether valid or invalid voxels) in the loading arrangement data 112. The loaded floor area ratio calculation unit 232 outputs the calculated information related to the loaded floor area ratio to the display unit 242. When the calculated floor area ratio is equal to or greater than the warning value (for example, 90% of the floor area ratio) specified by the warning value designation unit 231, the loading volume ratio calculation unit 232 displays warning display information for the calculated loading. It is output to the display unit 242 together with the information related to the floor area ratio. When it is inappropriate to consider the height inside the container 11 such as when the shape of the luggage 20 is irregular, that is, even if there is a space above the luggage 20, the entire floor surface of the container 11 is covered. If is occupied by the luggage 20, it may be considered that the loading capacity ratio is 100%.

The voxel visualization unit 233 is a functional unit that creates visualization information that visualizes effective voxels based on the loading arrangement data 112 from the arrangement processing unit 223 of the loading volume management unit 220 (see FIG. 2). The voxel visualization unit 233 creates visualization information in which grid lines are drawn on the image of the loading array data 112 in order to visualize effective voxels. This makes it possible to visualize the loading status that cannot be confirmed only by the point cloud. The voxel visualization unit 233 outputs the created visualization information to the display unit 242. When the voxel visualization unit 233 has a portion of the created visualization information that is equal to or greater than the warning value (for example, height change of 2 m) specified by the warning value specification unit 231, the voxel visualization unit 233 displays the warning display information as the created visualization information. And output to the display unit 242.

The interface unit 240 is a functional unit that exchanges information between the user and the loaded floor area ratio measuring device 200 (see FIG. 2). The interface unit 240 has an operation unit 241 and a display unit 242.

The operation unit 241 is a functional unit that receives user operations (see FIG. 2). The user inputs the data (for example, characters, numbers, positions, areas, etc.) input by operating the operation unit 241 (for example, keyboard operation, mouse click operation, touch panel tap operation, etc.) to the loading volume management unit 220. It is output to the monitoring area designation unit 221, the voxelization unit 222, the arrangement processing unit 223, the angle angle occlusion voxel calculation unit 225, or the warning value designation unit 231 of the result generation unit 230. The operation unit 241 designates the monitoring area to the monitoring area designation unit 221 by the operation of the user. The operation unit 241 instructs the voxelization unit 222 of the voxel side length by the user's operation. The operation unit 241 instructs the array processing unit 223 of the interval time by the operation of the user. The operation unit 241 designates the monitoring area and the voxel side length to the angle of view occlusion voxel calculation unit 225 (the same designation as the designation to the monitoring area designation unit 221 and the voxelization unit 222) by the user's operation. The operation unit 241 designates a warning value to the warning value designating unit 231 by the user's operation. The operation unit 241 instructs the start / end of the entire processing in this configuration function by the operation of the user.

The display unit 242 is a functional unit that displays various types of information (see FIG. 2). The display unit 242 displays the shooting data 100 after preprocessing from the noise removing unit 212 of the preprocessing unit 210. The display unit 242 displays the loaded floor area ratio and warning display information from the loaded floor area ratio calculation unit 232 of the result generation unit 230. The display unit 242 displays the visualization information and the warning display information from the voxel visualization unit 233 of the result generation unit 230. As a method of displaying the warning display information, for example, the background color can be changed or blinked. The information displayed on the display unit 242 may be output by voice using a speaker, printed by a printer, or transmitted and output to another terminal.

Next, the interpolation of the occlusion part, which is a blind spot caused by the specifications of the photographing unit, will be explained using drawings. 3A and 3B are image views schematically showing a shooting range of a photographing unit, FIG. 3A is a perspective view, FIG. 3B is a view seen from the arrow D1 of FIG. 3A, and FIG. 3C is a view. It is a figure seen from the arrow view D2 of 3 (A). 4A and 4B are image views schematically showing an example of a situation in which a baggage in a container is photographed by a photographing unit, FIG. 4A is a view seen from arrow D1 of FIG. 3A, and FIG. 4B is a diagram. It is a figure seen from the arrow view D2 of 3 (A). 5A and 5B are image views of an example of a point cloud when the luggage in the container is photographed by the photographing unit, FIG. 5A is a view seen from the arrow D1 of FIG. 3A, and FIG. A is a view seen from the arrow D2 of FIG. 3 (C), and (C) is a view seen from the arrow D3 of FIG. 3 (A). FIG. 6 is an image diagram viewed from the arrow D2 of FIG. 3A, which schematically shows a range that can be visualized from an image image taken by the photographing unit. FIG. 7 is an image diagram schematically showing a change in the visualization information of the baggage that can be visualized by the photographing unit in the process of loading the baggage in the container. 8 to 10 are image diagrams schematically showing a flow from photographing of the floor area ratio measuring device in the floor area ratio measuring system according to the first embodiment to voxel interpolation.

The photographing unit 300 has an angle of view (horizontal angle of view and vertical angle of view) in the horizontal direction and the vertical direction as a specification, and photographs a rectangular or elliptical angle of view within each continuous plane toward the depth direction. (See FIGS. 3A to 3C). It is some LIDAR that the angle of view range is elliptical. The photographing unit 300 measures the distance from the photographing unit 300 to the shield if there is a shield (not shown) between the photographing unit 300 and the measurement target unit (not shown), and is located behind the shield. The object to be measured is not photographed. The angle of view range of the photographing unit 300 becomes narrower as it gets closer to the photographing unit 300. When the shooting unit 300 reaches a certain close distance (several tens of centimeters to 1 m depending on the model), shooting becomes impossible. When the photographing unit 300 is used in a narrow space such as inside the container 11 of a truck, there may be a case where only a narrow range can be photographed depending on the amount of loading (see FIGS. 4A and 4B). In FIG. 4A, the portion close to the side surface of the container 11 on the left and right sides of the photographing unit 300 becomes an occlusion (shielding area), and the portion close to the side surface of the container 11 on the left and right sides as shown in FIGS. 5A and 5C is a point. It will be a point cloudless occlusion. In FIG. 4 (B), the portion near the bottom surface of the container 11 is an occlusion, and as shown in FIGS. 5 (B) and 5 (C), the portion near the bottom surface of the container 11 is an occlusion without a point cloud.

The visible range that can be visualized by voxelizing the image data (point cloud) taken by the photographing unit 300 is as shown in FIG. The valid voxels that can be activated by the voxelization unit (222 in FIG. 2) are within the visible range, and the invalid voxels that are not activated are outside the visible range. In the example of FIG. 6, the vertical angle of view is specialized, but the horizontal angle of view is also taken into consideration when processing. The visible range of FIG. 6 is within the monitoring area designated by the monitoring area designation unit (221 of FIG. 2). Such a visualizeable range includes the mounting position and swing angle (position relative to the monitoring area and the angle of the optical axis) of the photographing unit 300, and the minimum imageable distance peculiar to the photographing unit 300 to be used. It can be calculated from the elements of the angle of view and the blind spot and the non-photographable area in the voxel grid can also be calculated from the same element.

FIG. 7 shows an example of changes in the visualization information of the luggage that can be visualized by the photographing unit in the process of loading the luggage in the container. The luggage in the back of the container, which is not affected by the angle of view of the shooting unit 300, can be estimated and interpolated from the past data, but the closer the luggage is to the front, the more the portion that cannot be photographed (the luggage that cannot be visualized in FIG. 7). (Equivalent to 20b) increases, estimation interpolation cannot be performed from past data, and the deviation from the actual load capacity becomes large.

Therefore, as shown in FIGS. 8 to 10, interpolation is performed to enable invalid voxels that satisfy predetermined conditions. 8 to 10, first, as shown in FIGS. 8 (A), 9 (A), and 10 (A), the image pickup unit 300 photographs the luggage 20 in the container 11, and the photographed data is used as a diagram. The point cloud as shown in 8 (B), 9 (B), and 10 (B) is acquired. Next, sequence data is created with the voxel at the position where the point cloud is located as an effective voxel, and the voxels of the created sequence data are as shown in FIGS. 8 (C), 9 (C), and 10 (C). Create projection array data projected from the front side to the back. Next, in the projection sequence data, the invalid voxels on the same surface as the valid voxels marked with a star (including each x mark in FIG. 10C) directly above the invalid voxels that serve as blind spots are shown in FIG. 8 (D). ), FIG. 9 (D), and FIG. 10 (D). However, if there is an effective voxel under the effective voxel in the back and in front of it as shown in FIGS. 9 (C) and 9 (D), the invalid voxel under the effective voxel in the back is not activated at this stage and is in front. Enables invalid voxels under the previous valid voxels in. In addition, how much the front direction of the invalid voxel in the blind spot is valid is determined by checking the time series data on the plane of the loading progress direction and determining the progress increment in the front direction per unit time.

In summary, from the angle of view on the specifications of the shooting unit 300, the range that cannot be shot is grasped, and invalid voxels that satisfy the predetermined conditions are enabled (interpolated). If the part with cargo (the part where the point cloud can be acquired) disappears in the past data and the point cloud does not exist on the depth side in the direction away from the shooting unit 300, the luggage is placed on the front side where shooting is not possible. Judge as a thing. In addition, it is determined that the luggage does not float in the air (a point cloud can be obtained above), or that the luggage exists in the downward direction of the portion where the luggage is determined to be placed. It is necessary to link the top, bottom, left, and right of the photographing unit 300 with the top, bottom, left, and right of the real world. Further, from the viewpoint of preventing the load from collapsing, the load is usually loaded from the left and right wall sides, so if there is a load in the central portion, it is determined that the load also exists in the left and right directions.

The operation of the floor area ratio measuring device in the floor area ratio measuring system according to the first embodiment will be described with reference to the drawings. FIG. 11 is a flowchart illustrating the operation of the loaded floor area ratio measuring device in the loaded floor area ratio measuring system according to the first embodiment. Please refer to FIGS. 1 and 2 for the components of the loaded floor area ratio measuring system and the loaded floor area ratio measuring device.

First, the operation unit 241 of the interface unit 240 of the load floor area ratio measuring device 200 sets the parameters by the user's operation (step A1). In the parameter settings, the monitoring area is specified in the monitoring area specification unit 221, the voxel side length is specified in the voxelization unit 222, the interval time for associating the array data 111 is specified in the array processing unit 223, and the angle occlusion voxel calculation. The monitoring area and voxel side length are designated to the unit 225, and the warning value (loading volume ratio, change in height of luggage) is specified to the warning value designating unit 231. The monitoring area is specified for the entire area that needs to be measured, for example, according to the internal dimensions of the container. Further, the designation of the monitoring area and the voxel side length to the angle of view occlusion voxel calculation unit 225 is the same as the designation of the monitoring area and the voxel side length to the monitoring area designation unit 221 and the voxelization unit 222.

After step A1 or when the loaded floor area ratio has changed (YES in step A12), the format conversion unit 211 of the preprocessing unit 210 of the loaded floor area ratio measuring device 200 has the imaging data 100 (3) from the imaging unit 300. Dimensional point cloud) is sequentially acquired (step A2). Here, it is assumed that the photographing unit 300 photographs the entire area inside the container 11 of the truck 10.

Next, the format conversion unit 211 converts the format of the shooting data 100 into a common format that can be commonly used in the loaded floor area ratio measuring device 200 (step A3).

Next, the noise removing unit 212 of the preprocessing unit 210 of the loaded floor area ratio measuring device 200 removes noise (point cloud unnecessary for measurement) from the point cloud in the shooting data 100 from the format conversion unit 211 (step). A4).

Next, the monitoring area designation unit 221 of the loading volume management unit 220 of the loading volume ratio measuring device 200 determines a point cloud existing in the monitoring area designated by the operation unit 241 of the photographing data 100 from the noise removing unit 212. Acquire (step A5).

Next, the boxing unit 222 of the loading volume management unit 220 of the loading volume ratio measuring device 200 creates an array data 111 in which the point cloud in the monitoring area of the photographing data 100 acquired by the monitoring area designation unit 221 is voxelized. (Step A6). Here, in the creation of the array data 111, a virtual voxel grid in which the monitoring area is divided by the voxels having the voxel side length specified by the operation unit 241 is created, and the voxels in the created voxel grid are photographed. It is possible to create array data 111 in which voxels having a predetermined number or more of points in the monitoring area of the data 100 are used as valid voxels and voxels having less than a predetermined number of points in the point group are used as invalid voxels.

Next, the angle of view occlusion voxel calculation unit 225 of the loaded floor area ratio measuring device 200 calculates the blind spot invalid voxel position information (position information of the invalid voxel that cannot be visualized in the occlusion portion that becomes the blind spot of the photographing unit 300) (step). A7). Here, in the calculation of the blind spot invalid voxel position information, the monitoring area and voxel side length designated by the operation unit 241 and the imaging unit specification data 113 (photographing unit specification data 113 (photographing unit mounting position) acquired from the photographing unit specification data storage unit 226). Based on the information and the angle of view information), the position information of the blind spot invalid voxel in the voxel grid divided by the voxel of the voxel side length in the management area is calculated.

Next, the arrangement processing unit 223 of the loading volume management unit 220 of the loading volume ratio measuring device 200 has a plurality of time-series array data 111 from the voxelization unit 222 and a blind spot invalid voxel position from the angle occlusion voxel calculation unit 225. Based on the information, the loading array data 112 interpolated so that the invalid voxels of the occlusion portion (including the portion that becomes a blind spot, the shadow of an object, etc.) satisfying a predetermined condition is an effective voxel is created (step A8). The details of step A7 will be described later.

Here, in the creation of the loading array data 112, the array data 111 from the box cell conversion unit 222 is arranged in time series for each interval time interval specified by the operation unit 241. The associated sequence data group 110 is created, the created sequence data group 110 is stored in the sequence data group storage unit 224, and the occlusion part is excluded by comparing each sequence data 111 of the stored sequence data group 110. Estimated sequence data that estimates the valid boxels in the above state is created for each sequence data group 110, and based on the valid boxels of the created estimated sequence data, invalid boxels (blind spots) in the downward direction, the left-right direction, and the back direction are created. The loading array data 112 with (including invalid boxels) enabled (interpolated) is created.

Next, the load volume ratio calculation unit 232 of the result generation unit 230 of the load volume ratio measuring device 200 calculates the load volume ratio based on the load arrangement data 112 from the arrangement processing unit 223 of the load volume management unit 220 (step). A9). When calculating the loaded floor area ratio, it is determined whether or not the calculated loaded floor area ratio is equal to or greater than the warning value specified by the warning value specifying unit 231.

Next, the voxel visualization unit 233 of the result generation unit 230 of the load volume ratio measuring device 200 creates visualization information that visualizes effective voxels based on the load arrangement data 112 from the arrangement processing unit 223 of the load volume management unit 220. (Step A10). When creating the visualization information, it is determined whether or not there is a change in height equal to or higher than the warning value specified by the warning value specifying unit 231 in the created visualization information. Here, in the creation of visualization information, grid lines can be drawn on the image of the loading array data 112. In addition, step A10 may be performed at the same time as step A9, or may be performed before step A9.

Next, the display unit 242 of the interface unit 240 of the load floor area ratio measuring device 200 is the load volume ratio from the load volume ratio calculation unit 232 (warning display information when the determination result is a load volume ratio equal to or higher than the warning value). , And the visualization information from the boxel visualization unit 233 (warning display information when the determination result has a change in height equal to or greater than the warning value) is displayed (step A11). The displayed value of the loaded floor area ratio is stored in the loaded floor area ratio measuring device 200.

Next, the loaded floor area ratio measuring device 200 determines whether or not the value of the loaded floor area ratio displayed this time has changed from the value of the loaded floor area ratio displayed last time (step A12). The change in the value of the loaded floor area ratio can be regarded as no change when there is no change a predetermined number of times (for example, three times). If there is a change in the value of the loaded floor area ratio (YES in step A12), the process returns to step A2. If there is no change in the value of the loaded floor area ratio (NO in step A12), the process ends.

Next, the detailed operation of step A7 of the floor area ratio measuring device in the floor area ratio measuring system according to the first embodiment will be described with reference to the drawings. FIG. 12 is a flowchart illustrating the detailed operation of step A7 of FIG. 11 of the floor area ratio measuring device in the floor area ratio measuring system according to the first embodiment.

After calculating the invalid voxel position information in step A7, the arrangement processing unit 223 of the loading volume management unit 220 of the loading volume ratio measuring device 200 enters the monitoring area in the plurality of time-series array data 111 from the boxing unit 222. It is determined whether or not a valid voxel exists (step B1). If there is no valid voxel in the monitoring area (NO in step B1), the process proceeds to step B9.

When there is an effective voxel in the monitoring area (YES in step B1), the sequence processing unit 223 hides the measurement target behind the obstacle in the foreground based on the plurality of sequence data 111 in the time series. Estimated sequence data that estimates the effective voxels in a state where the obscured occlusion part is excluded is created, and the effective voxels in the created estimated sequence data are projected onto a two-dimensional plane (for example, FIGS. 8 (C) and 9 (for example). C), see FIG. 10 (C)) (step B2).

After step B2, the array processing unit 223 determines whether or not the entire region of the projected two-dimensional plane is an effective voxel (step B3). If the entire area of the two-dimensional plane is a valid voxel (YES in step B3), the process proceeds to step B8.

When a part of the two-dimensional plane is not a valid voxel (NO in step B3), the array processing unit 223 reads the past data of the array data group 110 from the array data group storage unit 224 and refers to it (step B4).

After step B4, the sequence processing unit 223 uses the blind spot invalid voxel position information calculated in step A7 to be within a predetermined distance from the previous valid voxel of the past data, and the front of the three-dimensional voxel in the current sequence data 111. It is determined whether or not there is a blind spot invalid voxel on the same surface as the effective voxel of (step B5). If there is no blind spot invalid voxel within a predetermined distance from the previous valid voxel of the past data (NO in step B5), the process proceeds to step B8.

When the blind spot invalid voxel exists within a predetermined distance from the front valid voxel of the past data (YES in step B5), the sequence processing unit 223 moves the voxel in the left-right direction of the front valid voxel of the three-dimensional box cell in the current sequence data 111. Enable all existing blind spot invalid voxels into valid voxels (step B6).

After step B6, the sequence processing unit 223 validates all the blind spot invalid voxels below the front valid voxels of the 3D voxels in the current sequence data 111 (including the valid voxels activated in step B6). Activate for voxels (step B7).

When all regions are valid voxels (YES in step B3), when there is no blind spot invalid voxel within a predetermined distance from the previous valid voxel in the past data (NO in step B5), or after step B7, sequence processing Part 223 activates all invalid voxels in the depth direction of the front valid voxels (including the valid voxels activated in steps B6 and B7) in the current sequence data 111 to the valid voxels (the valid voxels). Step B8), and then proceed to step A9 in FIG.

When there is no effective voxel in the monitoring area (NO in step B1), the sequence processing unit 223 determines whether or not there is an effective voxel outside the monitoring area in the plurality of time-series sequence data 111 from the box cell conversion unit 222. Is determined (step B9).

If there is an effective voxel outside the monitoring area (YES in step B9), or if there is no loading of luggage (NO in step B12), the array processing unit 223 determines that there is no loading of luggage in the monitoring area. (Step B10), and then proceed to step A9 in FIG.

When there is no effective voxel outside the monitoring area (NO in step B9), the sequence processing unit 223 reads the past data of the sequence data group 110 from the sequence data group storage unit 224 and refers to it (step B11).

After step B11, the array processing unit 223 determines whether or not there is a load in the monitoring area in the past data of the array data group 110 (step B12). If no luggage is loaded (NO in step B12), the process proceeds to step B10.

When there is a load of luggage (YES in step B12), the array processing unit 223 activates the invalid voxels of the three-dimensional voxels in the current array data 111 corresponding to the valid voxels for the past data (step B13).

After step B13, the array processing unit 223 refers to the past data and determines whether or not the state in which no valid voxel exists continues for a predetermined time or longer (step B14).

When the state in which the effective voxels do not exist continues for a predetermined time or longer (YES in step B14), the array processing unit 223 determines that the entire monitoring area has been loaded (step B15), and then, in FIG. 11, FIG. Proceed to step A9.

If the absence of valid voxels has not continued for a predetermined time or longer (NO in step B14), the sequence processing unit 223 determines that the load may be being loaded in the monitoring area (step B16), and then , Proceed to step A9 in FIG.

According to the first embodiment, the point cloud of the monitoring area of the photographing data 100 photographed by the photographing unit 300 is voxelized, and all the blind spot invalid voxels in the left-right direction and below the front effective voxel in the monitoring area are activated. By enabling all invalid voxels in the back direction and calculating the loading volume ratio, even if there is luggage in the immediate vicinity of the shooting unit, the measured loading capacity of the luggage on a volume basis without using multiple imaging units It can contribute to suppressing the discrepancy between the actual load capacity and the actual load capacity. This can contribute to accurately recognizing the situation of the entire luggage while automatically monitoring the loaded floor area ratio. In addition, occlusion caused by the specifications of the photographing unit 300 can be removed, and it can be applied even when a plurality of photographing units 300 are installed and a blind spot occurs, so that the photographing unit 300 can be installed close to the truck. Even if the photographing unit 300 is installed in the container, more appropriate visualization of the loading becomes possible, so that the flexibility (flexibility, flexibility) of the installation location of the photographing unit 300 is enhanced.

Further, according to the first embodiment, the loading volume and the loading volume ratio are quantitatively visualized for the cargo 20 that is sequentially loaded from the back of the container 11 toward the front, or conversely, the cargo 20 that is sequentially unloaded from the front. Can be done. As a result, it is possible to provide data that can be easily determined by the observer in real time, without depending on the individual determination of the personnel who load the cargo 20.

Further, according to the first embodiment, by analyzing the shooting data 100 in chronological order, the influence on the result due to a temporary occlusion factor such as a person who is working generated by using only one shooting unit 300. Can be reduced.

Further, according to the first embodiment, the amount of calculation can be reduced by converting the point cloud of the monitoring area of the photographing data 100 into voxels (converting the data at any time into an array of voxels divided by a unit grid). Interpolation processing of sparse data and estimation processing of missing parts due to occlusion can be simplified, and high-speed calculation can be performed.

[Embodiment 2]
The loaded floor area ratio measuring device according to the second embodiment will be described with reference to the drawings. FIG. 13 is a block diagram schematically showing the configuration of the loaded floor area ratio measuring device according to the second embodiment.

The loaded floor area ratio measuring device 200 is a device that measures the loaded floor area ratio of the object to be measured in the monitoring area. The loading floor area ratio measuring device 200 includes a monitoring area designation unit 221, a voxelization unit 222, an angle of view occlusion voxel calculation unit 225, and an arrangement processing unit 223.

The monitoring area designation unit 221 acquires a point cloud in the designated monitoring area in which the area where the measurement object is loaded in the predetermined space is present in the photographing data 100 obtained by photographing the predetermined space by the photographing unit (not shown). It is configured to do.

The voxelization unit 222 creates array data 111 in which voxels having a predetermined number or more of point clouds among the voxels in the voxel grid obtained by dividing the monitoring area by voxels having a voxel side length specified in advance are used as effective voxels. It is configured to do.

The angle of view occlusion voxel calculation unit 225 is based on the imaging unit mounting position information related to the position where the photographing unit (not shown) is attached, the angle of view information related to the angle of view of the photographing unit, the monitoring area, and the voxel side length. Therefore, in the voxel grid in the array data 111, the blind spot invalid voxel position information 114 related to the position of the invisible blind spot invalid voxel of the occlusion portion which is the blind spot of the photographing unit is calculated.

The array processing unit 223 is in a state of excluding the occlusion portion in which the obstacle in the foreground hides the measurement object in the background and makes it invisible based on the plurality of array data 111 within the interval time specified in advance. It is configured to create estimated sequence data that estimates valid voxels. The sequence processing unit 223 activates all blind spot invalid voxels in the left-right direction or downward direction of the foremost effective voxel in the estimated sequence data as effective voxels, and all after activation, when a predetermined condition is satisfied. It is configured to create loading sequence data 112 in which all invalid voxels in the depth direction of the valid voxels of the above are enabled as valid voxels.

The floor area ratio calculation unit 232 is configured to calculate the floor area ratio, which is the ratio of the volume of the measurement object loaded in the monitoring area to the total volume in the monitoring area, based on the loading arrangement data 112. There is.

According to the second embodiment, the point cloud of the monitoring area of the photographing data 100 photographed by the photographing unit is converted into voxels, and when a predetermined condition is satisfied, all the points in the left-right direction or the downward direction of the frontmost effective voxel in the monitoring area. By enabling the blind spot invalid voxels and calculating the loading volume ratio by enabling all invalid voxels in the back direction, even if there is luggage in the close range of the shooting part, the volume without using multiple shooting parts It can contribute to suppressing the discrepancy between the measured load capacity of the luggage at the base and the actual load capacity.

Note that the loaded floor area ratio measuring device according to the first and second embodiments can be configured by so-called hardware resources (information processing device, computer), and one having the configuration illustrated in FIG. 14 can be used. For example, the hardware resource 1000 includes a processor 1001, a memory 1002, a network interface 1003, and the like, which are interconnected by an internal bus 1004.

Note that the configuration shown in FIG. 14 does not mean to limit the hardware configuration of the hardware resource 1000. The hardware resource 1000 may include hardware (eg, an input / output interface) (not shown). Alternatively, the number of units such as the processor 1001 included in the apparatus is not limited to the example of FIG. 14, and for example, a plurality of processors 1001 may be included in the hardware resource 1000. For the processor 1001, for example, a CPU (Central Processing Unit), an MPU (Micro Processor Unit), a GPU (Graphics Processing Unit), or the like can be used.

For the memory 1002, for example, RAM (RandomAccessMemory), ROM (ReadOnlyMemory), HDD (HardDiskDrive), SSD (SolidStateDrive) and the like can be used.

For the network interface 1003, for example, a LAN (Local Area Network) card, a network adapter, a network interface card, or the like can be used.

The function of the hardware resource 1000 is realized by the above-mentioned processing module. The processing module is realized, for example, by the processor 1001 executing a program stored in the memory 1002. In addition, the program can be downloaded via a network or updated using a storage medium in which the program is stored. Further, the processing module may be realized by a semiconductor chip. That is, the function performed by the processing module may be realized by executing the software on some hardware.

A part or all of the above embodiment may be described as in the following appendix, but is not limited to the following.

[Appendix 1]
A monitoring area designation unit configured to acquire a point cloud in which a region in which a measurement object is loaded in the predetermined space is located in a designated monitoring region among shooting data captured by the imaging unit in a predetermined space.
Among the voxels in the voxel grid obtained by dividing the monitoring area by voxels having a predetermined voxel side length, the voxels having a predetermined number of points in the point cloud are used as effective voxels to create array data. With the voxelization department,
The voxel grid in the array data based on the imaging unit mounting position information related to the position where the imaging unit is attached, the angle of view information related to the angle of view of the imaging unit, the monitoring area, and the voxel side length. In the angle of view occlusion voxel calculation unit configured to calculate the blind spot invalid voxel position information related to the position of the invisible blind spot invalid voxel of the occlusion portion which is the blind spot of the photographing unit.
Estimated effective voxels excluding the occlusion part where the obstacle in the foreground hides the measurement object behind and makes it invisible, based on the plurality of said sequence data within the predetermined interval time. When sequence data is created and certain conditions are met, all the blind spot invalid voxels located in the left-right direction or downward direction of the front valid voxel in the estimated sequence data are enabled as valid voxels, and after activation. An array processing unit configured to create loading array data with all invalid voxels in the depths of all the valid voxels enabled for valid voxels.
Based on the loading arrangement data, the loading volume ratio calculation unit configured to calculate the loading volume ratio, which is the ratio of the volume of the measurement object loaded in the monitoring area to the total volume in the monitoring area. When,
A loaded floor area ratio measuring device.
[Appendix 2]
The sequence processing unit is configured to create the estimated sequence data when a valid voxel exists in the monitored region in the plurality of sequence data.
The loaded floor area ratio measuring device according to Appendix 1.
[Appendix 3]
The sequence processing unit projects the effective voxels in the estimated sequence data onto a two-dimensional plane, and under the predetermined condition, when the valid voxels do not exist in a part of the two-dimensional plane, the estimated sequence data All the blind spot invalid voxels located in the left-right direction or downward direction of the foremost effective voxel are configured to be activated as effective voxels.
The loaded floor area ratio measuring device according to Appendix 1 or 2.
[Appendix 4]
When the array processing unit creates the loaded array data, as the predetermined condition, when the blind spot invalid voxel exists within a predetermined distance from the position of the foremost effective voxel in the past data, the array processing unit is the foremost in the estimated array data. All the blind spot invalid voxels in the left-right direction of the effective voxel of the above are enabled to the effective voxels, and all the blind spot invalid voxels in the downward direction of the previous effective voxel after activation are activated to the effective voxels. Is configured in,
The loaded floor area ratio measuring device according to any one of Supplementary note 1 to 3.
[Appendix 5]
The sequence processing unit activates all the downward blind spot invalid voxels into effective voxels, and then activates all the invalid voxels in the depth direction of all the activated effective voxels into effective voxels. Is configured to
The loaded floor area ratio measuring device according to Appendix 4.
[Appendix 6]
When the array processing unit creates the loading array data, as the predetermined condition, when there is no blind spot invalid voxel within a predetermined distance from the position of the previous effective voxel in the past data, the array processing unit of all the effective voxels. It is configured to create loading array data with all invalid voxels in the back direction enabled for valid voxels,
The loaded floor area ratio measuring device according to any one of Supplementary note 1 to 5.
[Appendix 7]
As the predetermined condition, when the effective voxels are present in the entire region of the two-dimensional plane, the array processing unit enables all the invalid voxels in the depth direction of all the effective voxels to be effective voxels. It is configured to create loading array data,
The loaded floor area ratio measuring device according to Appendix 3.
[Appendix 8]
A shooting unit configured to output shooting data shot in a predetermined space,
The loaded floor area ratio measuring device according to any one of Supplementary note 1 to 7.
A loading floor area ratio measurement system equipped with.
[Appendix 9]
It is a loading floor area ratio measurement method that measures the floor area ratio using hardware resources.
A step of acquiring a point cloud in the designated monitoring area where the area where the measurement object is loaded in the predetermined space is among the shooting data taken by the photographing unit in the predetermined space.
Among the voxels in the voxel grid obtained by dividing the monitoring area by voxels having a predetermined voxel side length, a step of creating array data using voxels having a predetermined number or more of points in the point cloud as effective voxels.
The voxel grid in the array data based on the imaging unit mounting position information related to the position where the imaging unit is attached, the angle of view information related to the angle of view of the imaging unit, the monitoring area, and the voxel side length. In the step of calculating the blind spot invalid voxel position information related to the position of the blind spot invalid voxel that cannot be visualized in the occlusion portion that becomes the blind spot of the photographing unit.
Estimated effective voxels excluding the occlusion part where the obstacle in the foreground hides the measurement object behind and makes it invisible, based on the plurality of said sequence data within the predetermined interval time. When sequence data is created and certain conditions are met, all the blind spot invalid voxels located in the left-right direction or downward direction of the front valid voxel in the estimated sequence data are enabled as valid voxels, and after activation. Steps to create loading array data with all invalid voxels in the depths of all the valid voxels enabled for valid voxels,
Based on the loading arrangement data, a step of calculating the loading volume ratio, which is the ratio of the volume of the measurement object loaded in the monitoring area to the total volume in the monitoring area, and
Load floor area ratio measurement method including.
[Appendix 10]
It is a program that causes hardware resources to execute the process of measuring the loaded floor area ratio.
Processing to acquire a point cloud in the designated monitoring area where the area where the measurement object is loaded in the predetermined space is among the shooting data taken by the shooting unit in the predetermined space.
Among the voxels in the voxel grid obtained by dividing the monitoring area by voxels having a predetermined voxel side length, a process of creating array data using a voxel having a predetermined number or more of points in the point cloud as an effective voxel.
The voxel grid in the array data based on the imaging unit mounting position information related to the position where the imaging unit is attached, the angle of view information related to the angle of view of the imaging unit, the monitoring area, and the voxel side length. In the process of calculating the blind spot invalid voxel position information related to the position of the blind spot invalid voxel that cannot be visualized in the occlusion portion that becomes the blind spot of the photographing unit.
Estimated effective voxels excluding the occlusion part where the obstacle in the foreground hides the measurement object behind and makes it invisible, based on the plurality of said sequence data within the predetermined interval time. When sequence data is created and certain conditions are met, all the blind spot invalid voxels located in the left-right direction or downward direction of the front valid voxel in the estimated sequence data are enabled as valid voxels, and after activation. The process of creating loading array data with all invalid voxels in the depth direction of all the valid voxels enabled for valid voxels, and
Based on the loading arrangement data, a process of calculating the loading volume ratio, which is the ratio of the volume of the measurement object loaded in the monitoring area to the total volume in the monitoring area,
To let the hardware resource execute
program.

It should be noted that each disclosure of the above patent documents shall be renormalized and described in this document, and may be used as the basis or a part of the present invention as necessary. Within the framework of all disclosures (including claims and drawings) of the present invention, embodiments or examples can be changed or adjusted based on the basic technical idea thereof. Further, various combinations or selections (necessary) of various disclosure elements (including each element of each claim, each element of each embodiment or embodiment, each element of each drawing, etc.) within the framework of all disclosure of the present invention. (Not selected) is possible. That is, it goes without saying that the present invention includes all disclosures including claims and drawings, and various modifications and modifications that can be made by those skilled in the art in accordance with the technical idea. In addition, regarding the numerical values and numerical ranges described in the present application, it is considered that arbitrary intermediate values, lower numerical values, and small ranges are described even if they are not specified. Further, each of the disclosed matters of the above-cited documents may be used in combination with the matters described in this document in part or in whole as a part of the disclosure of the present invention, if necessary, in accordance with the purpose of the present invention. It is considered to be included in (belonging to) the matters disclosed in the present application.

1 Loading capacity ratio measurement system 10 Truck 11 Truck container 20 Luggage 20a Visualizable luggage 20b Non-visualable luggage 100 Shooting data 110 Sequence data group 111 Sequence data 112 Loading sequence data 113 Shooting section specification data 114 Blind spot invalid boxel position Information 200 Load capacity ratio measuring device 210 Preprocessing unit 211 Format conversion unit 212 Noise removal unit 220 Load capacity management unit 221 Monitoring area designation unit 222 Box cell conversion unit 223 Sequence processing unit 224 Array data group storage unit 225 Angle occlusion box cell calculation unit 226 Imaging unit Specifications Data storage unit 230 Result generation unit 231 Warning value specification unit 232 Loading volume ratio calculation unit 233 Box cell visualization unit 240 Interface unit 241 Operation unit 242 Display unit 300 Imaging unit 400 Network 1000 Hardware resources 1001 Processor 1002 Memory 1003 Network interface 1004 Internal bus

Claims

A monitoring area designation unit configured to acquire a point cloud in which a region in which a measurement object is loaded in the predetermined space is located in a designated monitoring region among shooting data captured by the imaging unit in a predetermined space.
Among the voxels in the voxel grid obtained by dividing the monitoring area by voxels having a predetermined voxel side length, the voxels having a predetermined number of points in the point cloud are used as effective voxels to create array data. With the voxelization department,
The voxel grid in the array data based on the imaging unit mounting position information related to the position where the imaging unit is attached, the angle of view information related to the angle of view of the imaging unit, the monitoring area, and the voxel side length. In the angle of view occlusion voxel calculation unit configured to calculate the blind spot invalid voxel position information related to the position of the invisible blind spot invalid voxel of the occlusion portion which is the blind spot of the photographing unit.
Estimated effective voxels excluding the occlusion part where the obstacle in the foreground hides the measurement object behind and makes it invisible, based on the plurality of said sequence data within the predetermined interval time. When sequence data is created and certain conditions are met, all the blind spot invalid voxels located in the left-right direction or downward direction of the front valid voxel in the estimated sequence data are enabled as valid voxels, and after activation. An array processing unit configured to create loading array data with all invalid voxels in the depths of all the valid voxels enabled for valid voxels.
Based on the loading arrangement data, the loading volume ratio calculation unit configured to calculate the loading volume ratio, which is the ratio of the volume of the measurement object loaded in the monitoring area to the total volume in the monitoring area. When,
A loaded floor area ratio measuring device.
The sequence processing unit is configured to create the estimated sequence data when a valid voxel exists in the monitored region in the plurality of sequence data.
The loaded floor area ratio measuring device according to claim 1.
The sequence processing unit projects the effective voxels in the estimated sequence data onto a two-dimensional plane, and under the predetermined condition, when the valid voxels do not exist in a part of the two-dimensional plane, the estimated sequence data All the blind spot invalid voxels located in the left-right direction or downward direction of the foremost effective voxel are configured to be activated as effective voxels.
The loaded floor area ratio measuring device according to claim 1 or 2.
When the array processing unit creates the loaded array data, as the predetermined condition, when the blind spot invalid voxel exists within a predetermined distance from the position of the foremost effective voxel in the past data, the array processing unit is the foremost in the estimated array data. All the blind spot invalid voxels in the left-right direction of the effective voxel of the above are enabled to the effective voxels, and all the blind spot invalid voxels in the downward direction of the previous effective voxel after activation are activated to the effective voxels. Is configured in,
The loaded floor area ratio measuring device according to any one of claims 1 to 3.
The sequence processing unit activates all the downward blind spot invalid voxels into effective voxels, and then activates all the invalid voxels in the depth direction of all the activated effective voxels into effective voxels. Is configured to
The loaded floor area ratio measuring device according to claim 4.
When the array processing unit creates the loading array data, as the predetermined condition, when there is no blind spot invalid voxel within a predetermined distance from the position of the previous effective voxel in the past data, the array processing unit of all the effective voxels. It is configured to create loading array data with all invalid voxels in the back direction enabled for valid voxels,
The loaded floor area ratio measuring device according to any one of claims 1 to 5.
As the predetermined condition, when the effective voxels are present in the entire region of the two-dimensional plane, the array processing unit enables all the invalid voxels in the depth direction of all the effective voxels to be effective voxels. It is configured to create loading array data,
The loaded floor area ratio measuring device according to claim 3.
A shooting unit configured to output shooting data shot in a predetermined space,
The loaded floor area ratio measuring device according to any one of claims 1 to 7.
A loading floor area ratio measurement system equipped with.
It is a loading floor area ratio measurement method that measures the floor area ratio using hardware resources.
A step of acquiring a point cloud in the designated monitoring area where the area where the measurement object is loaded in the predetermined space is among the shooting data taken by the photographing unit in the predetermined space.
Among the voxels in the voxel grid obtained by dividing the monitoring area by voxels having a predetermined voxel side length, a step of creating array data using voxels having a predetermined number or more of points in the point cloud as effective voxels.
The voxel grid in the array data based on the imaging unit mounting position information related to the position where the imaging unit is attached, the angle of view information related to the angle of view of the imaging unit, the monitoring area, and the voxel side length. In the step of calculating the blind spot invalid voxel position information related to the position of the blind spot invalid voxel that cannot be visualized in the occlusion portion that becomes the blind spot of the photographing unit.
Estimated effective voxels excluding the occlusion part where the obstacle in the foreground hides the measurement object behind and makes it invisible, based on the plurality of said sequence data within the predetermined interval time. When sequence data is created and certain conditions are met, all the blind spot invalid voxels located in the left-right direction or downward direction of the front valid voxel in the estimated sequence data are enabled as valid voxels, and after activation. Steps to create loading array data with all invalid voxels in the depths of all the valid voxels enabled for valid voxels,
Based on the loading arrangement data, a step of calculating the loading volume ratio, which is the ratio of the volume of the measurement object loaded in the monitoring area to the total volume in the monitoring area, and
Load floor area ratio measurement method including.
It is a program that causes hardware resources to execute the process of measuring the loaded floor area ratio.
Processing to acquire a point cloud in the designated monitoring area where the area where the measurement object is loaded in the predetermined space is among the shooting data taken by the shooting unit in the predetermined space.
Among the voxels in the voxel grid obtained by dividing the monitoring area by voxels having a predetermined voxel side length, a process of creating array data using voxels having a predetermined number or more of points in the point cloud as effective voxels.
The voxel grid in the array data based on the imaging unit mounting position information related to the position where the imaging unit is attached, the angle of view information related to the angle of view of the imaging unit, the monitoring area, and the voxel side length. In the process of calculating the blind spot invalid voxel position information related to the position of the blind spot invalid voxel that cannot be visualized in the occlusion portion that becomes the blind spot of the photographing unit.
Estimated effective voxels excluding the occlusion part where the obstacle in the foreground hides the measurement object behind and makes it invisible, based on the plurality of said sequence data within the predetermined interval time. When sequence data is created and certain conditions are met, all the blind spot invalid voxels located in the left-right direction or downward direction of the front valid voxel in the estimated sequence data are enabled as valid voxels, and after activation. The process of creating loading array data with all invalid voxels in the depth direction of all the valid voxels enabled for valid voxels, and
Based on the loading arrangement data, a process of calculating the loading volume ratio, which is the ratio of the volume of the measurement object loaded in the monitoring area to the total volume in the monitoring area,
To let the hardware resource execute
program.