WO2022181753A1

WO2022181753A1 - Loading space recognition device, system, method, and program

Info

Publication number: WO2022181753A1
Application number: PCT/JP2022/007817
Authority: WO
Inventors: 悟己上野; 教之青木; 真則高岡; 研二河野; ゆり安達
Original assignee: 日本電気通信システム株式会社
Priority date: 2021-02-26
Filing date: 2022-02-25
Publication date: 2022-09-01
Also published as: JPWO2022181753A1

Abstract

Provided is a loading space recognition device, etc. that can contribute to determining the possibility of the occurrence of a cargo collapse due to the movement of baggage. The loading space recognition device comprises an entire baggage image estimation unit, a voxelization unit, and a determination unit. The entire baggage image estimation unit is configured so as to estimate, on the basis of three-dimensional data in which images of a loading space for baggage are captured from a predetermined direction, an entire image of baggage loaded in the loading space and output the entire image as estimation result data. The voxelization unit is configured so as to voxelize the estimation result data and output the estimation result data as voxel data. The determination unit is configured so as to estimate a fluctuated volume amount or movement amount of the baggage by comparing the voxel data of an arbitrary reference time with the voxel data after the passage of a predetermined or arbitrary amount of time from the reference time and determine the presence or absence of the possibility of a cargo collapse by comparing the estimated fluctuated volume amount or movement amount with a threshold value.

Description

Load space recognition device, system, method, and program

[Description of related applications]
The present invention is based on the priority claim of Japanese Patent Application: Japanese Patent Application No. 2021-030741 (filed on February 26, 2021), and the entire description of the application is incorporated herein by reference. shall be
The present invention relates to a loading space recognition device, system, method, and program.

In the logistics industry, cargo loaded in truck containers is damaged due to collapse of cargo such as falling during transportation, resulting in a decline in transportation quality. Therefore, techniques have been proposed for generating information for arranging cargo so as to prevent cargo from collapsing, and for promptly discovering collapse of cargo when it actually occurs (for example, see Patent Document 1). 1 to 6)

Patent Documents 1 and 2 disclose three-dimensional information of a first region on the surface of a plurality of articles obtained by imaging or scanning the plurality of articles stacked from a first point as a technique for suppressing collapse of cargo. and a second information acquisition unit for acquiring three-dimensional information of a second region of the surfaces of the plurality of articles obtained by imaging or scanning the plurality of articles from a second point. and, based on the three-dimensional information of the first region acquired by the first information acquisition unit and the three-dimensional information of the second region acquired by the second information acquisition unit, the surfaces of the plurality of articles a synthesizing unit that generates information indicating at least a part of the three-dimensional shape, wherein the positions of the first point and the second point are different from each other, and the synthesizing unit generates the three-dimensional information of the first region and the A shape information generation device is disclosed that complements one of the three-dimensional information of the second region with the other to generate information indicating the three-dimensional shape of at least part of the surfaces of the plurality of articles.

Patent Document 3 discloses a stowage method for stacking a plurality of objects to be stowed in a predetermined packing style and with a predetermined weight as a technique for preventing collapse of cargo, wherein each of the objects to be stowed a step of measuring the weight and packing style of each of the stowage items, a step of calculating the density of each stowage item from the weight and packing style of each of these stowage items, and a step of accumulating the density information of each of these stowage items. and calculating the stowage position of each of these items to be stowed.

Patent Document 4 discloses a technology that allows the driver to check the state of luggage in the luggage compartment at any time (whether or not the luggage has collapsed) on the upper surface of the luggage compartment at an intermediate position in the vehicle width direction. An information processing device for inputting image data of the luggage compartment from the surveillance camera, and an image data of the luggage compartment input from the information processing device and displayed. Equipped with a monitor and a communication device for inputting image data of the luggage compartment from the information processing device and transmitting the image data to a base station, the vehicle is constructed so that the availability of the loading space can be grasped from the image of the surveillance camera. A cargo room monitor is disclosed.

In Patent Document 5, a sensor installed in the loading platform of a transportation vehicle is used as a technology that enables drivers and management centers to easily know that cargo has collapsed in a transportation vehicle and to respond quickly to the collapse of cargo. an in-vehicle terminal that monitors collapse of cargo and, when collapse of cargo is detected, issues a warning to a driver indicating that collapse of cargo has occurred; a management center for receiving a load collapse monitoring signal indicating that the cargo collapse has occurred, which is transmitted from the A system is disclosed.

In Patent Document 6, as a technology capable of detecting collapse of cargo that occurs during and after transfer work, each time an individual item is transferred, a group of items stacked on a pallet before and after the transfer is displayed upward. to acquire a first image before the transfer of the article and a second image after the transfer, compare the first image and the second image, and determine the area other than the area where the transferred article existed Disclosed is an article collapse detection method for determining whether or not a collapse of cargo has occurred based on the degree of change in the article area.

Japanese Patent No. 6511681 Japanese Patent No. 6577687 JP-A-2002-29631 JP 2018-199489 A JP 2005-018472 A Japanese Unexamined Patent Application Publication No. 2007-179301

The following analysis is given by the inventor of this application.

However, in the technologies for suppressing and preventing collapse of cargo described in Patent Documents 1 to 3, collapse of cargo cannot be completely eliminated by simply loading cargo so that collapse of cargo does not occur, acceleration, braking, vibration of the vehicle, etc. Since there are movements of cargo in all directions that lead to collapse of cargo due to movement of cargo, it is impossible to determine the possibility of occurrence of collapse of cargo due to movement of cargo.

In addition, in the technology for checking the luggage described in Patent Document 4, the driver checks the status of the luggage in the luggage compartment with a monitor and knows that the luggage has collapsed. It is difficult from a safety point of view to determine the possibility of collapse of cargo.

In addition, in the technology for detecting the collapse of cargo described in Patent Document 5, the collapse of cargo is determined based on the presence or absence of reception of signals such as infrared rays and ultrasonic waves at the sensor. movement of cargo cannot be detected, and the possibility of cargo collapse due to movement of cargo cannot be determined.

In addition, in the technology of Patent Document 6 that can detect collapse of cargo that occurs during transfer and after transfer work is completed, a first image taken before transfer and a first image after transfer are taken. An edge image is generated for each of the two images, the two images are compared, and the presence or absence of collapse of cargo is determined based on the degree of change of the goods other than the transferred goods. However, it is not possible to detect the movement of cargo without movement at the edge position (movement of cargo in the far and near direction with respect to the imaging position), and it may not be possible to determine the possibility of cargo collapse due to cargo movement. .

The main object of the present invention is to provide a loading space recognition device, system, method, and program that can contribute to determining the possibility of cargo collapse due to movement of cargo.

A loading space recognizing device according to a first viewpoint estimates an overall image of cargo loaded in the loading space based on three-dimensional data obtained by imaging the loading space of the cargo from a predetermined direction, and outputs it as estimation result data. a voxelization unit configured to voxelize the estimation result data and output as voxel data; the voxel data at an arbitrary reference time; and the reference estimating the amount of change in volume or amount of movement of the load by comparing the voxel data after a predetermined or arbitrary time has passed from time to time, and comparing the estimated amount of change in volume or amount of movement with a threshold value; and a determination unit configured to determine whether or not there is a possibility of cargo collapse.

A load space recognition system according to a second viewpoint includes a sensor that senses the surface of a load in the load space and outputs imaged three-dimensional data, and a load space recognition device according to the first viewpoint. .

A loading space recognition method according to a third viewpoint is a loading space recognition method for recognizing a cargo loading space using hardware resources, and is based on three-dimensional data obtained by imaging the loading space from a predetermined direction, a step of estimating an overall image of the cargo loaded in the loading space and outputting it as estimation result data; a step of voxelizing the estimation result data and outputting it as voxel data; the voxel data at an arbitrary reference time; estimating the amount of change in volume or amount of movement of the cargo by comparing the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and comparing the estimated amount of change in volume or amount of movement with a threshold; and determining whether or not there is a possibility of cargo collapse by comparing.

A program according to a fourth aspect is a program that causes a hardware resource to execute a process of recognizing a cargo loading space, and is based on three-dimensional data obtained by imaging the cargo space from a predetermined direction. A process of estimating the overall image of the loaded luggage and outputting it as estimation result data, a process of voxelizing the estimation result data and outputting it as voxel data, the voxel data at an arbitrary reference time, and the voxel data from the reference time By comparing the voxel data after the elapse of a predetermined or arbitrary period of time with the voxel data, and by comparing the estimated amount of change in volume or movement with a threshold value. and a process of determining whether or not there is a possibility of cargo collapse.

The program can be recorded on a computer-readable storage medium. The storage medium can be non-transient such as semiconductor memory, hard disk, magnetic recording medium, optical recording medium, and the like. The present disclosure may also be embodied as a computer program product. A program is input to a computer device via an input device or an external communication interface, is stored in a storage device, drives a processor in accordance with predetermined steps or processes, and stages the results of processing including intermediate states as necessary. can be displayed via a display device, or can be communicated with the outside via a communication interface. A computer device for this purpose, as an example, typically includes a processor, a storage device, an input device, a communication interface, and optionally a display device, all of which are connectable to each other by a bus.

According to the first to fourth viewpoints, it is possible to contribute to determining the possibility of cargo collapse due to cargo movement.

1 is an image diagram schematically showing an example of the configuration and usage of a loading space recognition system according to Embodiment 1. FIG. 2 is a block diagram schematically showing the configuration of a loading space recognition device in the loading space recognition system according to Embodiment 1; FIG. 4 is a flowchart schematically showing the operation of the loading space recognition device in the loading space recognition system according to Embodiment 1; FIG. 10 is an image diagram schematically showing some examples in which there is a varying volume amount when extracting the difference between the reference voxel and the comparison voxel. FIG. 10 is an image diagram schematically showing several examples in which there is no volume variation and there is movement when the difference between the reference voxel and the comparison voxel is extracted. FIG. 6 is an image diagram schematically showing a transition from coherent estimation to longest movement amount estimation in example 2-1 of FIG. 5; FIG. 6 is an image diagram schematically showing a transition from collective estimation to longest movement amount estimation in Example 2-2 of FIG. 5; FIG. 6 is an image diagram schematically showing a transition from collective estimation to longest movement amount estimation in Example 2-3 of FIG. 5 ; FIG. 6 is an image diagram schematically showing a transition from collective estimation to longest movement amount estimation in Example 2-4 of FIG. 5; FIG. 5 is an image diagram schematically showing a modification of the configuration and usage of the loading space recognition system according to the first embodiment; FIG. 7 is a block diagram schematically showing the configuration of a loading space recognition device according to Embodiment 2; 3 is a block diagram schematically showing the configuration of hardware resources; FIG.

Hereinafter, embodiments will be described with reference to the drawings. It should be noted that when reference numerals are attached to the drawings in this application, they are solely for the purpose of helping understanding, and are not intended to limit the embodiments shown in the drawings. Moreover, the following embodiments are only examples, and do not limit the present invention. Also, connection lines between blocks in drawings and the like referred to in the following description include both bidirectional and unidirectional connections. The unidirectional arrows schematically show the flow of main signals (data) and do not exclude bidirectionality. Furthermore, in the circuit diagrams, block diagrams, internal configuration diagrams, connection diagrams, etc. disclosed in the present application, an input port and an output port exist at the input end and the output end of each connection line, respectively, although not explicitly shown. The input/output interface is the same. The program is executed via a computer device, and the computer device includes, for example, a processor, a storage device, an input device, a communication interface, and optionally a display device. It is configured to be able to communicate with external devices (including computers), whether wired or wireless.

[Embodiment 1]
A loading space recognition system according to the first embodiment will be described with reference to the drawings. FIG. 1 is an image diagram schematically showing an example of the configuration and usage of the loading space recognition system according to the first embodiment. FIG. 2 is a block diagram schematically showing the configuration of the loading space recognition device in the loading space recognition system according to the first embodiment. Here, the cargo of containers on a truck will be described as an example.

The loading space recognition system 1 uses a sensor 10 to detect changes in an object (volume variation, It is a system that recognizes distance fluctuations (see Fig. 1). In the loading space recognition system 1, the sensor 10 and the loading space recognition device 200 are connected so as to be communicable (wired communication or wireless communication). The loading space recognition system 1 can be mounted on the truck 2 . In the loading space recognition system 1, the loading space recognition device 200 recognizes the change of the object in the loading space 5 based on the photographed data (100 in FIG. 2) photographed by the sensor 10. FIG. Here, the photographing data 100 is three-dimensional data photographed by the sensor 10 (data obtained by photographing three-dimensional elements such as point cloud data and depth data, data reconstructed into a three-dimensional space from a plurality of images, etc.). ).

The sensor 10 is a sensor that senses and photographs (images) the surface of the load 4 in the loading space 5, which is the photographing area (see FIG. 1). The sensor 10 is communicably connected to the loading space recognition device 200 . The sensor 10 outputs photographed data ( 100 in FIG. 2 ) created from three-dimensional data obtained by photographing the loading space 5 to the loading space recognition device 200 . The sensor 10 can be selected according to the photographing conditions such as the photographing distance, the angle of view, and the size of the container 3 necessary for detecting the package 4 and the customer's request. Products sold by various manufacturers can be used for the sensor 10 . As the sensor 10, for example, a stereo camera, ToF (Time of Flight) camera, 3D-lidar (Three Dimensions - light detection and ranging), 2D-lidar (Two Dimensions - light detection and ranging), LIDAR (Laser Imaging Detection And Ranging ) etc. can be used. The sensor 10 can be installed at a position where the loading space 5, which is an imaging area, can be photographed, for example, near the upper part of the container 3 on the loading entrance side. The sensor 10 can photograph the cargo 4 in the loading space 5 from above the cargo entrance side of the container 3 . At least one sensor 10 may be provided in the loading space 5, but a plurality of sensors may be provided. Further, a plurality of sensors 10 may be used even when the loading space 5 is so large that it is difficult to photograph with one sensor. In addition, when there are a plurality of sensors 10 in the loading space 5, the types and manufacturers may differ. Also, if there are a plurality of sensors 10 in the loading space 5 , the loading space recognizing device 200 synthesizes each photographed data captured by each sensor 10 .

The loading space recognizing device 200 is a device that recognizes fluctuations (volume fluctuations, distance fluctuations) of the cargo 4 in the loading space 5 in which the cargo 4 is loaded, based on the photographed data 100 from the sensor 10 (FIGS. 1 and 2). 2). As the loading space recognition device 200, a device (computer device) having functional units (eg, a processor, a storage device, an input device, a communication interface, and a display device) constituting a computer can be used. Computers, smart phones, tablet terminals, etc. can be used. The loading space recognition device 200 has a function of determining whether or not there is a possibility that the cargo 4 will collapse. The loading space recognizing device 200 can also be employed when a certain amount of volumetric change is expected in the event of collapse of cargo under conditions such as pallet stacking. Loading space recognizing device 200 is implemented by preprocessing unit 210, packing appearance grasping unit 220, determining unit 230, and user interface unit 240 by executing a predetermined program.

The preprocessing unit 210 is a functional unit that performs preprocessing for carrying out loading space recognition processing on the photographed data 100 (see FIG. 2). The preprocessing unit 210 outputs preprocessed data 101 obtained by preprocessing the photographed data 100 to the packing style grasping unit 220 and the user interface unit 240 . The preprocessing section 210 includes a format conversion section 211 and a noise removal section 212 .

The format conversion unit 211 is a functional unit that converts the format of the photographed data 100 into a common format that can be commonly used in the loading space recognition device 200 as preprocessing (see FIG. 2). The format conversion unit 211 outputs the common format photographed data 100 to the noise elimination unit 212 . Note that if the format of the photographed data 100 is originally a common format, the process of the format conversion unit 211 can be omitted (skipped).

The noise removal unit 212 is a functional unit that removes noise (for example, point groups unnecessary for loading space recognition) from the photographed data 100 from the format conversion unit 211 as preprocessing (see FIG. 2). The noise removal unit 212 outputs the preprocessed data 101 from which the noise in the photographed data 100 has been removed to the package overall image estimation unit 222 of the package grasping unit 220, and if necessary, the display unit of the user interface unit 240. 241 output. Noise removal methods include, for example, smoothing processing, filtering (eg, moving average filter processing, median filter processing, etc.), outlier removal processing (eg, outlier removal processing by chi-square test), and the like. Note that the processing of the noise removal unit 212 may be omitted (skipped) if there is almost no noise.

The preprocessed data 101 is output to the display unit 241 and can be viewed by the user. In addition, since the photographed data 100 and the preprocessed data 101 are the basis of all processing, it is possible to ensure that they can be saved by making them storable.

The packing style grasping unit 220 is a functional block that grasps the current packing style from the preprocessed data 101 (see FIG. 2). The packing style grasping unit 220 outputs the voxel data 102 related to the grasped current packing style to the determining unit 230 . The packing style grasping unit 220 includes an area designating unit 221 , a whole package image estimating unit 222 , and a voxelizing unit 223 . The packing form grasping unit 220 measures information about the shape of the loaded baggage or empty space from the preprocessed data 101 and provides quantitative information.

The area designation unit 221 is a functional unit that designates an area (loading area) in which the cargo 4 can be loaded within the container 3 (see FIG. 2). For example, if there is an area in the container 3 in which the cargo 4 cannot be loaded, the area designating unit 221 designates the area by excluding that area. The area specifying unit 221 allows the user to specify the stacking area via the operation unit 242, and can automatically specify the area in combination with a system that automatically acquires the edge of the area.

Also, the area specifying unit 221 may specify a determination exclusion area to be excluded from determination by the determination unit 230 from the user via the operation unit 242 . As a result, it is possible to prevent excessive change in packing appearance from being judged by not judging the change in packing appearance of the package 4 that cannot maintain a constant shape. The change determination exclusion area may be specified not only in units of areas but also in units of voxels, or in units of single/individual packages (in units of objects).

Based on the preprocessed data 101 from the noise remover 212 of the preprocessed data 101 and the loading area specified by the area designating unit 221, the package overall image estimation unit 222 estimates the entire package 4 loaded in the loading area. This is the functional part for estimating the image (see FIG. 2). The package overall image estimation unit 222 creates estimation result data relating to the overall image of the loaded package 4 . The package overall image estimation unit 222 outputs the generated estimation result data to the voxelization unit 223 . The package overall image estimation unit 222 detects not only the packages loaded on the surface of the package and the frontmost package, but also the packages (if necessary gap) is also estimated. For estimating the load loaded in the occlusion area, it is possible to supplement with past data accumulated in chronological order or use prior knowledge of the size and number of loaded loads. Preprocessed data 101 stored in chronological order may be used, or machine learning may be used, in addition to a simple algorithm that assumes that a coordinate point indicating a package exists on an extension of .

An example of a machine learning teacher data collection method is a method of associating preprocessed data 101 with the actual loading status and recording it in a database. For example, distance sensors are provided at predetermined intervals on the ceiling surface of the container, and the distance to the loaded cargo is measured by the distance sensors. As a result, it is possible to ascertain the loading status of the cargo, such as whether the cargo directly below the sensor is piled up to the ceiling, is piled up to about half the height, or is completely empty. By providing multiple sensors on the ceiling surface, it is possible to grasp the loading status of the entire cargo inside the container. In this manner, machine learning can be performed by associating the preprocessed data 101 with the actual loading status. The machine learning method is not limited to this, and other methods may be used. Also, when machine learning is used, a function of estimating information/attributes other than shape such as "weight" and "strictly no stacking/strictly prohibiting stacking below" may be provided.

The voxelization unit 223 is a functional unit that voxels the estimation result data of the package overall image estimation unit 222 (see FIG. 2). The voxelization unit 223 creates voxel data 102 relating to the overall image of the package 4 based on the estimation result data of the package overall image estimation unit 222 . The voxelization unit 223 creates the voxel data 102 by treating the portion where the loaded baggage 4 does not exist as an empty space. The voxelization unit 223 outputs the created voxel data 102 to the reference determination unit 231 and the difference extraction unit 232 of the determination unit 230 . In the voxelization process, it is possible to estimate the hidden area that cannot be seen from the 3D sensor, or to complement it with data accumulated in chronological order, or to use prior knowledge such as the size and number of cargo to be loaded. .

Here, the voxel data 102 is data representing the overall image of the package 4 by combining a plurality of voxels (cubes) of a predetermined size. The voxel data 102 includes information on the dimensions of each voxel and the position of each plane. The voxel data 102 may be stored each time it is created. The voxel data 102 may also include information such as the number of packages in one voxel, information about which multiple voxels one package is located across, and information such as weight and shape.

The determination unit 230 compares the voxel data 102 (reference voxel data) at a certain reference point in time with the voxel data 102 (comparison voxel data) at a point in time after a predetermined or arbitrary period of time has passed, thereby determining changes in the load. It is a functional unit that determines the possibility of collapse of cargo by estimating the amount of volume or amount of movement and comparing the estimated amount of fluctuating volume or amount of movement with a threshold value (see FIG. 2). Thereby, the load 4 can be prevented from falling. In addition, the determination unit 230 not only compares the reference voxel data and the comparison voxel data, but also compares the comparison voxel data with another comparison voxel data created immediately before to determine the possibility of cargo collapse. You may make it detect the presence or absence. As a result, when the amount of change between the comparison voxel data per unit time is smaller than the amount of change between the reference voxel data and the comparison voxel data, it is possible not to determine that there is a possibility of collapse of cargo. can. Furthermore, the truck 2 (a structure having a loading space) is provided with a detection unit 20 for detecting vibration and sound (see FIG. 10), and the determination unit 230 detects vibration and sound exceeding a certain level in the detection unit 20. In some cases, comparison voxel data may be obtained from the voxelization unit 223 to detect the possibility of collapse of cargo. Furthermore, in order to supplement and verify the certainty of movement estimation, information obtained from another system, such as the fact that there are many heavy packages and many light packages, may be used. The determination unit 230 includes a reference determination unit 231, a difference extraction unit 232, a volume fluctuation estimation unit 233, a unity extraction unit 234, a combination estimation unit 235, a movement amount estimation unit 236, and a load collapse determination unit 237. , provided.

The reference determination unit 231 is a functional unit that determines the voxel data 102 at a certain reference point in time (reference time) from the voxelization unit 223 as a change reference point (see FIG. 2). The reference determination unit 231 stores the voxel data 102 at the reference time point (the indicated time point) from the voxelization unit 223 according to an instruction from the operation unit 242, and holds it as reference data for positional variation. The reference determination unit 231 outputs the voxel data 102 at the reference time to the difference extraction unit 232 .

The difference extraction unit 232 compares the voxel data 102 at an arbitrary reference time (reference voxel data) with the voxel data 102 at a point in time when a predetermined or arbitrary time has passed from the reference time (comparison voxel data). is a functional unit that extracts the difference (for example, the difference in the position in the depth direction) between voxels corresponding to the surface of the cargo 4 viewed from a predetermined position (for example, the rear of the truck 2, the position of the sensor 10). (See Figure 2). The difference extraction unit 232 acquires reference voxel data from the reference determination unit 231 and acquires comparison voxel data from the voxelization unit 223 in the difference extraction process. In the difference extraction process, for example, when the position of the surface of the voxel when viewing the load 4 from behind the truck 2 changes to the front side of the track 2 in the depth direction (for example, when the load 4 at the position of the target voxel moves to the left, right, or bottom, and the cargo 4 on the far side appears; extracts the difference so as to indicate that the volume has "decreased", and similarly when the position of the voxel surface changes to the rear side of the track 2 (for example, the load 4 at the target voxel position moves to the front side If the target voxel moves, if another load 4 moves in front of the load 4 at the position of the target voxel, if the occlusion part increases), the difference is extracted so as to indicate that the volume "increased". be able to. In addition, in the difference extraction process, for example, the difference is extracted so as to express the degree of volume decrease or increase stepwise according to the magnitude of the distance by which the surface position of the voxel changes forward or backward on the track 2. can be done. The difference extraction unit 232 outputs the difference data extracted by the difference extraction process to the variation volume estimation unit 233 and the unity extraction unit 234 .

The fluctuating volume estimation unit 233 is a functional unit that estimates the fluctuating volume (fluctuation volume) of the overall image of the package 4 based on the difference data from the difference extraction unit 232 (see FIG. 2). The fluctuating volume can be represented, for example, by the sum of the differences in the depth direction positions of voxels at corresponding positions on the surface of the cargo 4 viewed from the rear of the truck 2 . When a certain volume increases at a certain position and the same volume decreases at another position, there is no change in volume (0) (see movement examples 2-1 to 2-4 in FIG. 5). Also, if a certain amount of volume increases at a certain position and a greater amount of volume decreases than the increased volume amount at another position, the fluctuating volume amount will be a negative value. In addition, if a certain volume increases at a certain position and a volume that is smaller than the increased volume decreases at another position, the fluctuating volume becomes a positive value (movement example 1-1 in FIG. 4 ~ see 1-3). Also, if a certain volume increases at a certain position and there is no volume decrease at another position, the fluctuating volume has a positive value. Furthermore, if at one location there is a decrease in one volume and there is no increase in another location, the variable volume will be a negative value. The fluctuating volume estimation unit 233 outputs the estimated fluctuating volume estimation data to the cargo collapse determination unit 237 .

Based on the difference data from the difference extraction unit 232, the unity extraction unit 234 extracts voxels with increased volume (increased volume voxels) and voxels with decreased volume (decreased volume voxels) at the same horizontal position. ), and extracts a group of voxels with no increase or decrease in volume (see FIG. 2). The unity extraction unit 234 can be based on the premise that the volume of the packages 4 does not change. When the grouping extracting section 234 can extract the grouping, the grouping extracting section 234 outputs difference data including the extracted grouping data to the combination estimating section 235 . The grouping extracting section 234 outputs the difference data from the difference extracting section 232 to the combination estimating section 235 when the grouping cannot be extracted.

The combination estimating unit 235 is a functional unit that estimates a combination of increased volume voxels and decreased volume voxels based on the difference data from the unity extraction unit 234 (including unity data if a unity can be extracted). (See Figure 2). The combination estimating unit 235 can assume that the volume of the package 4 does not change. In the combination estimation process, for example, when a plurality of combinations are possible, it is possible to preferentially combine increased volume voxels and decreased volume voxels at the same horizontal position. Also, when a plurality of combinations are possible, it is possible to preferentially combine the increased volume voxel and the decreased volume voxel that are farthest apart from each other. Also, when a plurality of combinations are possible, it is possible to preferentially combine increased volume voxels and decreased volume voxels so as to maximize the total distance. Also, a volume-decreasing voxel can be prevented from being combined with a volume-increasing voxel that is one level above it, since it is impossible for the cargo 4 to move upward if the cargo 4 falls. . A volume-increasing voxel may not be combined with a volume-decreasing voxel one level below it, since the load 4 cannot move up if it falls. If the combination can be estimated, the combination estimating section 235 outputs difference data including the extracted combination data to the movement amount estimating section 236 . The combination estimation unit 235 outputs the difference data from the unity extraction unit 234 to the movement amount estimation unit 236 when the combination cannot be estimated.

Based on the difference data (which may include grouped data and combination data) from the combination estimation unit 235, the movement amount estimation unit 236 calculates the number of packages generated between the reference time and the time when a predetermined or arbitrary time has passed. 4 (see FIG. 2). The moving amount estimator 236 can assume that the volume of the load 4 does not change. When the difference data includes grouped data and combination data, the movement amount estimation unit 236 calculates the distance from the volume reduction voxel to the volume increase voxel related to the combination data, and calculates the distance from the volume amount increase voxel related to the combined data. is calculated, and a value obtained by subtracting the calculated length from the calculated distance is estimated as the amount of movement of the load 4 . If the difference data does not include combined data and includes combination data, the movement amount estimation unit 236 calculates the distance from the volume reduction voxel to the volume increase voxel related to the combination data, and calculates the calculated distance. is estimated as the amount of movement of the load 4 . If the difference data does not include combination data (regardless of the presence or absence of grouped data), the movement of the cargo 4 is accompanied by a change in volume. It is possible to preferentially perform the estimation of the volume variation in the volume estimation unit 233 . In addition, when estimating the amount of movement, correction may be made so that different weights are assigned to horizontal movement and vertical movement (falling). may be corrected to be smaller than the estimated movement amount, and correction may be made to be larger than the estimated movement amount so that even a small movement is warned when a vertical or oblique movement occurs. . When the difference data includes a plurality of combination data, the movement amount estimator 236 estimates the movement amount for each combination data.

Also, the movement amount estimation unit 236 selects the longest movement amount from the estimated movement amounts, and estimates the selected movement amount as the longest movement amount. In addition, when the estimated movement amount is one, the movement amount is estimated as the longest movement amount. Movement amount estimation section 236 outputs the estimated longest movement amount estimation data to cargo collapse determination section 237 .

Examples of operations of the grouping extraction unit 234, the combination estimation unit 235, and the movement amount estimation unit 236 will be described later.

The load collapse determination unit 237 determines the estimated volume fluctuation data from the volume fluctuation estimation unit 233 or the estimated longest movement amount data from the movement amount estimation unit 236 and a preset threshold (the first value for the volume fluctuation). This is a functional unit that determines whether or not there is a possibility that the load 4 will collapse by comparing the threshold value (or the second threshold value for the longest movement amount) (see FIG. 2). The cargo collapse determination unit 237 determines that there is a possibility that the cargo 4 collapses when the estimated volume fluctuation data is greater than the first threshold value (even when the first threshold value or more is possible). The cargo collapse determination unit 237 determines that there is no possibility of cargo collapse of the cargo 4 when the estimated volume fluctuation data is equal to or less than the first threshold value (or less than the first threshold value). The cargo collapse determination unit 237 determines that there is a possibility that the cargo 4 collapses when the longest movement amount is larger than the second threshold value (even when the second threshold value or more is possible). The cargo collapse determination unit 237 determines that there is no possibility of cargo collapse of the cargo 4 when the longest movement amount is equal to or less than the second threshold value (even when less than the second threshold value is possible). It is arbitrary for the cargo collapse determination unit 237 to determine whether or not there is a possibility of collapse of the cargo 4 by preferentially using either the variable volume amount or the longest movement amount. The determination can be made by preferentially using the variable volume amount with a relatively small load. When it is determined that the cargo 4 may collapse, the cargo collapse determination unit 237 outputs warning output instruction information to the warning output unit 243 to warn the user of the occurrence of an abnormality.

The user interface unit 240 is a functional unit that has a user interface (function for exchanging information with the user) (see FIG. 2). The user interface unit 240 provides an interface between the user and the loading space recognition device 200 so that each process can be operated and the result of each process can be confirmed. The user interface section 240 includes a display section 241 , an operation section 242 and a warning output section 243 .

The display unit 241 is a functional unit that displays the preprocessed data 101 and the like from the noise removal unit 212 (see FIG. 2). As the display unit 241, for example, a liquid crystal display, an organic EL (Electroluminescence) display, an AR (Augmented Reality) glass, or the like can be used.

The operation unit 242 is a functional unit that performs area designation and confirmation instructions to the area designation unit 221 and the reference determination unit 231 based on user operations (see FIG. 2). As the operation unit 242, for example, a touch panel, a mouse, a camera and software for recognizing gestures and eye movements can be used.

The warning output unit 243 is a functional unit that outputs a warning to the user based on the warning output instruction information from the cargo collapse determination unit 237 (see FIG. 2). As the warning output unit 243, for example, a display that displays characters and images related to the warning, a speaker that outputs an alarm sound, a lamp that lights up the alarm, a communication unit that transmits warning output instruction information to another system, etc. can be used. .

Next, the operation of the loading space recognition device in the loading space recognition system according to Embodiment 1 will be described with reference to the drawings. FIG. 3 is a flow chart schematically showing the operation of the loading space recognition device in the loading space recognition system according to the first embodiment. 1 and 2 and their descriptions should be referred to for the configuration and details of the loading space recognition device.

First, the format conversion unit 211 of the preprocessing unit 210 acquires from the sensor 10 reference photographing data 100 (reference photographing data; three-dimensional data) obtained by photographing the cargo 4 in the loading space 5 serving as the photographing area ( Step A1).

Next, the format conversion section 211 of the preprocessing section 210 converts the format of the reference photographing data 100 into a common format (step A2).

Next, the noise removing unit 212 of the preprocessing unit 210 removes noise from the reference photographing data 100 converted into the common format to create reference preprocessing data 101 (step A3).

Next, based on the reference preprocessed data 101 and the loading area designated by the area designating section 221, the overall baggage image estimation section 222 of the packing style grasping section 220 determines the loaded baggage 4 on the loading area. is estimated (step A4).

Next, the voxelization unit 223 of the packing style grasping unit 220 creates reference voxel data 102 (reference voxel data) relating to the overall image of the package 4 based on the estimation result data estimated in step A4 ( Step A5).

Next, the reference determination unit 231 of the determination unit 230 saves the reference voxel data created by the voxelization unit 223 as a reference time value for the load collapse determination process by the operation of the operation unit 242 by the user ( Step A6). The operation of the operation unit 242 by the user can be performed, for example, after the loading of the cargo 4 into the container 3 is completed and before delivery is started.

Next, the format conversion unit 211 of the preprocessing unit 210 converts the image data 100 (reference image data) from the sensor 10 into the image pickup area when a predetermined or arbitrary time has passed since the reference image data 100 (reference image data) was acquired. Photographed data 100 for comparison (comparative photographed data; three-dimensional data) obtained by photographing the package 4 in the space 5 is acquired (step A7).

Next, the format conversion section 211 of the preprocessing section 210 converts the format of the photographing data 100 for comparison into a common format (step A8).

Next, the noise removal unit 212 of the preprocessing unit 210 removes noise from the comparison imaging data 100 converted into the common format to create comparison preprocessing data 101 (step A9).

Next, based on the preprocessed data 101 for comparison and the loading area designated by the area designating section 221, the overall baggage image estimation section 222 of the packing style grasping section 220 determines the loaded baggage 4 on the loading area. is estimated (step A10).

Next, the voxelization unit 223 of the packing style grasping unit 220 creates comparison voxel data 102 (comparison voxel data) relating to the overall image of the package 4 based on the estimation result data estimated in step A10 ( Step A11).

Next, the difference extraction unit 232 of the determination unit 230 compares the reference voxel data stored in the reference determination unit 231 with the comparison voxel data created by the voxelization unit 223 to obtain a predetermined position (for example, , the rear of the truck 2, the position of the sensor 10), and the difference between the voxels at the corresponding positions on the surface of the cargo 4 (for example, the difference in the position in the depth direction) is extracted (step A12).

Next, the fluctuating volume estimation unit 233 of the determination unit 230 estimates the fluctuating volume of the overall image of the package 4 (fluctuation volume) based on the difference data from the difference extraction unit 232 (step A13).

Next, the cargo collapse determination unit 237 of the determination unit 230 determines whether or not the fluctuation volume estimation data from the fluctuation volume estimation unit 233 is greater than a preset first threshold value for the fluctuation volume. (Step A14). If the estimated volume change is greater than the first threshold (YES in step A14), it is determined that there is a possibility that the cargo 4 may collapse, output warning output instruction information to the warning output unit 243, and proceed to step A20. .

If the estimated fluctuation volume is equal to or less than the first threshold (NO in step A14), the unity extraction unit 234 of the determination unit 230 extracts the volume at the same horizontal position based on the difference data from the difference extraction unit 232. A set of voxels with no increase or decrease in volume between the increased voxel and the decreased volume voxel is extracted (step A15). Skip when no grouping can be extracted.

Next, the combination estimation unit 235 of the determination unit 230 combines the increased volume voxels and decreased volume voxels based on the difference data from the unity extraction unit 234 (including the unity data if the unity can be extracted). is estimated (step A16).

Next, the movement amount estimator 236 of the determination unit 230, based on the difference data (which may include grouped data and combination data) from the combination estimator 235, determines when a predetermined or arbitrary time has elapsed from the reference time. Estimate the amount of movement of the cargo 4 that occurred during the period up to (step A17).

Here, in estimating the amount of movement, when the difference data includes grouped data and combined data, the distance from the volume reduction voxel to the volume increase voxel related to the combined data is calculated, and the volume increase or decrease related to the combined data is calculated. The length of the non-existent voxel is calculated, and a value obtained by subtracting the calculated length from the calculated distance is estimated as the movement amount of the load 4 . Further, in estimating the amount of movement, when the difference data does not include combined data and includes combination data, the distance from the volume decrease voxel to the volume increase voxel related to the combination data is calculated, and the calculated The distance value is estimated as the amount of movement of the load 4 . Further, in estimating the movement amount, if the difference data includes a plurality of combination data, the movement amount is estimated for each combination data.

Next, the movement amount estimation unit 236 of the determination unit 230 selects the longest movement amount from the estimated movement amounts, and estimates the selected movement amount as the longest movement amount (step A18).

Next, the load collapse determination unit 237 of the determination unit 230 determines whether or not the longest movement amount estimation data from the movement amount estimation unit 236 is larger than a preset second threshold value for the longest movement amount ( Step A19). If the estimated longest movement amount data is greater than the second threshold value (YES in step A19), it is determined that there is a possibility that the load 4 may collapse, output warning output instruction information to the warning output unit 243, and proceed to step A20. . If the estimated longest movement amount data is equal to or less than the second threshold value (NO in step A19), it is determined that there is no possibility of collapse of the cargo 4, and one cycle is terminated until the user instructs termination. Repeat steps A7 to A20.

If the estimated volume change is greater than the first threshold (YES in step A14), or if the estimated maximum movement amount data is greater than the second threshold (YES in step A19), the warning output unit 243 of the user interface unit 240 outputs a warning to the user based on the warning output instruction information from the cargo collapse determination unit 237 (step A20). After that, one cycle is finished, and steps A7 to A20 are repeated until the user instructs to finish.

Next, the operations of the difference extraction unit, the volume fluctuation estimation unit, and the collapse determination unit of the loading space recognition device in the loading space recognition system according to the first embodiment will be described using several examples and drawings. FIG. 4 is an image diagram schematically showing some examples in which there is a varying volume amount when extracting the difference between the reference voxel and the comparison voxel.

When the reference voxel data when the whole image of the package (4 in FIG. 1) is viewed from the rear side of the truck (2 in FIG. 1) is in a state like the reference voxel data in FIG. 4, the reference voxel data in FIG. 4, the difference extraction data extracted by the difference extraction unit (232 in FIG. 2) is changed from the comparison voxel data according to the movement example 1-1 to the difference extraction data according to the movement example 1-1 in FIG. Become. In the difference extraction data according to Movement Example 1-1, the volume amounts of only the upper left four voxels increase by one step, and the other voxels do not increase or decrease. That is, since there are only four increases in volume for one stage, the reference voxel data and comparison voxel data differ in total volume. In this case, it is thought that the occlusion part (gap or space) has increased on the back side of the voxel whose volume has increased. The volume estimator (233 in FIG. 2) estimates the volume fluctuation, and the collapse determination unit (237 in FIG. 2) performs threshold determination of the volume fluctuation.

When the reference voxel data in FIG. 4 changes to the comparison voxel data according to the movement example 1-2, the difference extraction data extracted by the difference extraction unit (232 in FIG. 2) becomes the difference extraction data according to the movement example 1-2 in FIG. It becomes like differential extraction data. In the differential extraction data according to Movement Example 1-2, the volumes of four voxels each decrease by one level, the volumes of other eight voxels increase by one level, and the other voxels increase or decrease no change in In other words, there is an increase and a decrease in volume, but even if the increase and decrease in volume are offset, there are four increases in volume by one stage, so the reference voxel data and the comparison voxel data Different total volume. In this case, since the load has moved as a whole, it is impossible to accurately calculate the amount of movement. Also, in this case, there is a movement of the load 4 between the increase of 4 volumes and the decrease of 4 volumes, and the surface position of the comparison voxel data increased from the surface position of the reference voxel data to the remaining 4 increases of the volume. Assuming that the distance to the position has moved, the fluctuating volume estimation unit (233 in FIG. 2) estimates the fluctuating volume, and the collapse determination unit (237 in FIG. 2) performs threshold determination of the fluctuating volume.

When the reference voxel data in FIG. 4 changes to the comparison voxel data according to the movement example 1-3, the difference extraction data extracted by the difference extraction unit (232 in FIG. 2) is the difference extraction data according to the movement example 1-3 in FIG. It becomes like differential extraction data. In the differential extraction data according to Movement Example 1-3, the volumes of 8 voxels decrease by one level, the volumes of another 12 voxels increase by one level, and the volumes of another four voxels increase by one level. The volume increases by two steps, and there is no increase or decrease in other voxels. In other words, there is an increase and a decrease in volume, but even if the increase and decrease in volume are offset, there are 4 more increases in volume by 1 step, and the increase in volume is 4 by 2 steps. Since there are many, the reference voxel data and comparison voxel data have different total volumes. In this case, there is movement of the cargo 4 between the volume increase of 8 units and the volume decrease of 8 units. Assuming that the distance from the surface position of the voxel data to the increased surface position of the comparison voxel data has moved, the fluctuating volume estimation unit (233 in FIG. 2) estimates the fluctuating volume, and the cargo collapse determination unit (237 in FIG. 2) ) to determine the threshold value of the volume fluctuation.

Next, some examples and drawings of the operation of the difference extraction unit, unity extraction unit, combination estimation unit, movement amount estimation unit, and load collapse determination unit of the loading space recognition device in the loading space recognition system according to the first embodiment will be described. will be used for explanation. FIG. 5 is an image diagram schematically showing several examples in which there is no volume variation and there is movement when the difference between the reference voxel and the comparison voxel is extracted. FIG. 6 is an image diagram schematically showing the transition from collective estimation to longest movement amount estimation in example 2-1 of FIG. FIG. 7 is an image diagram schematically showing the transition from coherent estimation to longest movement amount estimation in example 2-2 of FIG. FIG. 8 is an image diagram schematically showing the transition from coherent estimation to longest movement amount estimation in Example 2-3 of FIG. FIG. 9 is an image diagram schematically showing the transition from coherent estimation to longest movement amount estimation in Example 2-4 of FIG.

If the reference voxel data when the entire image of the package (4 in FIG. 1) is viewed from the rear side of the truck (2 in FIG. 1) is in a state like the reference voxel data in FIG. 5, the difference extraction data extracted by the difference extraction unit (232 in FIG. 2) is changed from the comparison voxel data according to the movement example 2-1 to the difference extraction data according to the movement example 2-1 in FIG. Become. In the difference extraction data according to Movement Example 2-1, the volume of each of the six voxels increases by one step, the volume of another six voxels decreases by one step, and the other voxels undergo an increase or decrease change. There is no In other words, there is an increase and a decrease in volume, and the increase and decrease in volume are offset by the same number and number of stages, so the total volume of the reference voxel data and comparison voxel data is the same. In this case, there is no increase or decrease in the volume existing between the volume increase voxel and the volume decrease voxel located at the same horizontal position as in FIG. Two groups of voxels are extracted, and two combinations of a group of increased volume voxels and a group of decreased volume voxels linked as shown in FIG. Then, the movement amount estimating unit (236 in FIG. 2) calculates the distance (see the two arrows in FIG. 6C) from the volume decrease voxel to the volume increase voxel related to the combination as shown in FIG. Then, the length (not shown) of the voxel with no volume increase or decrease related to the aggregated data was calculated, and the calculated length was subtracted from the distance calculated as shown in FIG. 6(D). The value is estimated as the movement amount of the load 4 (see two arrows in FIG. 6(D)), and the movement amount estimator (236 in FIG. 2) selects the longest movement amount from among the movement amounts as the longest movement amount (FIG. 6 (See the circled arrow in (D)), and the load collapse determination unit (237 in FIG. 2) determines the maximum amount of movement with a threshold value.

When the reference voxel data in FIG. 5 changes to the comparison voxel data according to the movement example 2-2, the difference extraction data extracted by the difference extraction unit (232 in FIG. 2) is the difference extraction data according to the movement example 2-2 in FIG. It becomes like differential extraction data. In the differential extraction data according to Movement Example 2-2, the volumes of eight voxels increase by one level, the volumes of other eight voxels decrease by one level, and the other voxels increase or decrease. There is no In other words, there is an increase and a decrease in volume, and the increase and decrease in volume are offset by the same number and number of stages, so the total volume of the reference voxel data and comparison voxel data is the same. In this case, there is no increase or decrease in the volume existing between the volume increase voxel and the volume decrease voxel located at the same horizontal position as in FIG. One group of voxels is extracted, and two combinations of a group of increased volume voxels and a group of decreased volume voxels linked as shown in FIG. Then, the movement amount estimation unit (236 in FIG. 2) calculates the distance (see the two arrows in FIG. 7C) from the volume decrease voxel related to the combination to the volume increase voxel as shown in FIG. Calculate the length of voxels (not shown; one) with no increase or decrease in the volume amount related to the combined data, and from the distance calculated as shown in FIG. 7(D), the calculated length is estimated as the amount of movement of the cargo 4 (see the two arrows in FIG. 7(D); one is deducted, the other is not deducted), and the movement amount estimation unit (236 in FIG. 2) The longest amount of movement from among them is estimated as the longest amount of movement (see the circled arrow in FIG. 7(D); in this case, there are two, but either can be used), and the cargo collapse determination unit (Fig. 237) performs a threshold determination of the maximum movement amount.

When the reference voxel data in FIG. 5 changes to the comparison voxel data according to the movement example 2-3, the difference extraction data extracted by the difference extraction unit (232 in FIG. 2) is the difference extraction data according to the movement example 2-3 in FIG. It becomes like differential extraction data. In the differential extraction data according to Movement Example 2-3, the volume of each of the eight voxels increases by one step, the volume of another eight voxels decreases by one step, and the other voxels undergo an increase or decrease change. There is no In other words, there is an increase and a decrease in volume, and the increase and decrease in volume are offset by the same number and number of stages, so the total volume of the reference voxel data and comparison voxel data is the same. In this case, in the unity extraction unit (234 in FIG. 2), there is no increase or decrease in volume existing between the volume increase voxel and the volume decrease voxel located at the same horizontal position as shown in FIG. 8A. Since the group of voxels cannot be extracted, it is skipped, and the group of voxels with increased volume and the group of voxels with decreased volume that are linked as shown in FIG. Two combinations are estimated, and the displacement estimation unit (236 in FIG. 2) calculates the distance from the volume decrease voxel to the volume increase voxel (two 8(D)), and the distance calculated as shown in FIG. 8(D) is estimated as the amount of movement of the load 4 (see two arrows in FIG. 236), the longest movement amount among the movement amounts is estimated as the longest movement amount (see the circled arrow in FIG. 8(D)), and the cargo collapse determination unit (237 in FIG. 2) determines the longest movement amount. Perform threshold judgment.

When the reference voxel data in FIG. 5 changes to the comparison voxel data according to the movement example 2-4, the difference extraction data extracted by the difference extraction unit (232 in FIG. 2) is the difference extraction data according to the movement example 2-4 in FIG. It becomes like differential extraction data. In the differential extraction data according to Movement Example 2-4, the volumes of 12 voxels increase by one step, the volumes of other 12 voxels decrease by one step, and the other voxels increase or decrease. There is no In other words, there is an increase and a decrease in volume, and the increase and decrease in volume are offset by the same number and number of stages, so the total volume of the reference voxel data and comparison voxel data is the same. In this case, in the unity extraction unit (234 in FIG. 2), there is no increase or decrease in volume existing between the volume increase voxel and the volume decrease voxel located at the same horizontal position as shown in FIG. 9A. Since the group of voxels cannot be extracted, it is skipped, and the group of increased volume voxels and the group of decreased volume voxels linked as shown in FIG. Three combinations are estimated, and the displacement estimation unit (236 in FIG. 2) calculates the distance from the volume decrease voxel to the volume increase voxel (three distances in FIG. 9C) as shown in FIG. (see arrows)), and the distance calculated as shown in FIG. 236), the longest movement amount out of the movement amounts is estimated as the longest movement amount (see the circled arrow in FIG. 9(D)), and the cargo collapse determination unit (237 in FIG. 2) determines the longest movement amount. Perform threshold judgment.

Note that, in the movement examples 1-1 to 1-3 and 2-1 to 2-4 above, the unity extraction processing criteria of the unity extraction unit 234, the combination estimation processing criteria of the combination estimation unit 235, and the movement amount estimation unit See the detailed description of FIG. 2 for the criteria of the H.236 displacement estimation process.

According to the first embodiment, the difference between the voxels at the corresponding positions between the reference voxel data and the comparison voxel data is extracted to estimate the amount of change in volume or the maximum amount of movement of the overall image of the package 4, and perform threshold determination. Therefore, it is possible to contribute to determining the possibility of collapse of cargo due to movement of cargo.

Further, according to the first embodiment, even if an occlusion portion hidden by the cargo 4 occurs, the occlusion portion is estimated and the packing appearance of the cargo 4 is grasped. Therefore, it is possible to consider the change of the cargo 4 in the occlusion portion. It is possible.

Furthermore, according to the first embodiment, the reference voxel data of the packing appearance at a specific time, such as before the start of movement of the truck 2, is held, and the comparative voxel data of the packing appearance after a predetermined or arbitrary time has passed from the reference time. By comparing, it is possible to detect the possibility of collapse of cargo. Drivers can be notified. As a result, the possibility of collapse of the cargo 4 can be detected at an early stage, the damage of the cargo 4 can be prevented, and the transportation quality can be prevented from being deteriorated.

[Embodiment 2]
A loading space recognition device according to the second embodiment will be described with reference to the drawings. FIG. 11 is a block diagram schematically showing the configuration of the loading space recognition device according to the second embodiment.

The loading space recognizing device 200 is a device that recognizes variations (volumetric variations, distance variations) of the cargo in the loading space where the cargo is loaded, based on the photographed data. Loading space recognizing device 200 includes package overall image estimating unit 222 , voxelizing unit 223 , and determining unit 230 .

The luggage overall image estimating unit 222 is configured to estimate the overall image of the luggage loaded in the loading space based on the three-dimensional data obtained by imaging the loading space of the luggage from a predetermined direction, and output it as estimation result data. ing.

The voxelization unit 223 is configured to voxelize the estimation result data and output it as voxel data.

The determination unit 230 compares voxel data at an arbitrary reference time with voxel data after a predetermined or arbitrary time has elapsed from the reference time, thereby estimating the amount of change in volume or the amount of movement of the load. It is configured to determine the presence or absence of the possibility of collapse of cargo by comparing the amount of change in volume or the amount of movement with a threshold value.

According to the second embodiment, the difference between the voxel data at the corresponding position between the voxel data at the reference time and the voxel data at the time other than the reference time is extracted, and the amount of change in volume or the maximum movement amount of the overall image of the package is estimated, and threshold determination is performed. is carried out, it is possible to contribute to determining the possibility of collapse of cargo due to movement of cargo.

It should be noted that the loading space recognition device according to the first and second embodiments can be configured by so-called hardware resources (information processing device, computer), and can use one having the configuration illustrated in FIG. 12 . For example, hardware resource 1000 includes processor 1001 , memory 1002 , network interface 1003 , etc., which are interconnected by internal bus 1004 .

It should be noted that the configuration shown in FIG. 12 is not intended to limit the hardware configuration of the hardware resource 1000 . The hardware resource 1000 may include hardware not shown (for example, an input/output interface). Alternatively, the number of units such as the processors 1001 included in the device is not limited to the illustration in FIG. For the processor 1001, for example, a CPU (Central Processing Unit), MPU (Micro Processor Unit), GPU (Graphics Processing Unit), etc. can be used.

For the memory 1002, for example, RAM (Random Access Memory), ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc. can be used.

For the network interface 1003, for example, a LAN (Local Area Network) card, network adapter, network interface card, etc. can be used.

The functions of the hardware resource 1000 are realized by the processing modules described above. The processing module is implemented by the processor 1001 executing a program stored in the memory 1002, for example. Also, the program can be downloaded via a network or updated using a storage medium storing the program. Furthermore, the processing module may be realized by a semiconductor chip. In other words, the functions performed by the above processing modules may be realized by executing software in some kind of hardware.

Some or all of the above embodiments can be described as the following supplementary notes, but are not limited to the following.

[Appendix 1]
An overall luggage image estimation unit configured to estimate an overall image of the luggage loaded in the loading space based on three-dimensional data obtained by imaging the loading space of the luggage from a predetermined direction, and to output estimation result data. When,
a voxelization unit configured to voxelize the estimation result data and output as voxel data;
By comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, the amount of change in volume or the amount of movement of the cargo is estimated, and the estimated a determination unit configured to determine the presence or absence of the possibility of cargo collapse by comparing the amount of change in volume or the amount of movement with a threshold value;
A loading space recognition device.
[Appendix 2]
The determination unit is
a reference determination unit configured to determine the voxel data at the reference time from the voxelization unit as a change reference point;
By comparing the voxel data at the reference time with the voxel data at the time when a predetermined or arbitrary time has passed since the reference time, voxels at corresponding positions on the surface of the package viewed from a predetermined position a difference extraction unit configured to extract the difference between and output as difference data;
a volume variation estimating unit configured to estimate a volume variation of the overall image of the package based on the difference data and output as volume variation estimation data;
a cargo collapse determination unit configured to determine the possibility of cargo collapse by comparing the estimated data of the volume fluctuation and the threshold for the volume fluctuation as the threshold;
The load space recognition device according to appendix 1, comprising:
[Appendix 3]
The determination unit is
Based on the difference data, a group of voxels without an increase or decrease in volume existing between an increased volume voxel with an increased volume and a decreased volume voxel with a decreased volume at the same horizontal position a unity extraction unit configured to extract and output as unity data;
a combination estimation unit configured to estimate a combination of the volume increase voxel and the volume decrease voxel based on the difference data and output the combined data;
estimating at least one amount of movement of the cargo during a period from the reference time to a time when a predetermined or arbitrary time has elapsed based on the difference data, the grouped data, and the combination data; a movement amount estimator configured to select the longest movement amount from among the movement amounts, estimate the selected movement amount as the longest movement amount, and output the longest movement amount estimation data;
further comprising
The cargo collapse determination unit compares the estimated data of the volume fluctuation with the threshold for the volume fluctuation as the threshold to determine the possibility of cargo collapse. When it is determined that there is a possibility of cargo collapse, the presence or absence of the possibility of collapse of cargo is determined by comparing the estimated longest movement amount data with the threshold value for the longest movement amount as the threshold value. , the loading space recognition device according to Supplementary Note 2.
[Appendix 4]
The determination unit is
a reference determination unit configured to determine the voxel data at the reference time from the voxelization unit as a change reference point;
By comparing the voxel data at the reference time with the voxel data at the time when a predetermined or arbitrary time has passed since the reference time, voxels at corresponding positions on the surface of the package viewed from a predetermined position a difference extraction unit configured to extract the difference between and output as difference data;
Based on the difference data, a group of voxels without an increase or decrease in volume existing between an increased volume voxel with an increased volume and a decreased volume voxel with a decreased volume at the same horizontal position a unity extraction unit configured to extract and output as unity data;
a combination estimation unit configured to estimate a combination of the volume increase voxel and the volume decrease voxel based on the difference data and output the combined data;
estimating at least one amount of movement of the cargo during a period from the reference time to a time when a predetermined or arbitrary time has elapsed based on the difference data, the grouped data, and the combination data; a movement amount estimator configured to select the longest movement amount from among the movement amounts, estimate the selected movement amount as the longest movement amount, and output the longest movement amount estimation data;
a cargo collapse determination unit configured to determine the possibility of cargo collapse by comparing the longest movement amount estimation data with the threshold for the maximum movement amount as the threshold;
The load space recognition device according to appendix 1, comprising:
[Appendix 5]
further comprising an area specifying unit configured to specify a cargo loading area in the loading space by a user's operation;
The luggage overall image estimation unit is configured to estimate the overall image of the luggage loaded in the loading area.
5. The loading space recognition device according to any one of Appendices 1 to 4.
[Appendix 6]
The region specifying unit is configured to specify a determination exclusion region to be excluded from determination by the determination unit by user operation,
The luggage overall image estimating unit is configured to exclude the luggage loaded in the determination exclusion area and estimate the overall image of the luggage loaded in the loading area.
The loading space recognition device according to appendix 5.
[Appendix 7]
Further comprising a detection unit attached to the structure having the loading space and detecting vibration or sound,
The determination unit acquires the voxel data from the voxelization unit when the detection unit detects shaking or sound of a certain level or more, and compares the voxel data at the reference time and the acquired voxel data. configured to estimate the varying volume or displacement of the load by comparing
The loading space recognition device according to any one of Appendices 1 to 6.
[Appendix 8]
The movement amount estimator corrects the movement amount to be smaller than the estimated movement amount when the movement direction of the baggage is the horizontal direction, or the estimated movement amount when the movement direction of the baggage is the vertical direction or the oblique direction. corrected so as to be larger than the corrected movement amount, and selecting the longest movement amount from the corrected movement amount.
The loading space recognition device according to appendix 3 or 4.
[Appendix 9]
The determination unit further includes: a change amount between the voxel data other than the reference time and the other voxel data immediately before the voxel data; It is configured to determine the presence or absence of the possibility of cargo collapse by comparing the amount of change between the data,
The loading space recognition device according to any one of appendices 1 to 8.
[Appendix 10]
The determination unit is configured to output warning output instruction information when it is determined that there is a possibility of cargo collapse,
The loading space recognition device further includes a warning output unit configured to output a warning based on the warning output instruction information.
The loading space recognition device according to any one of Appendices 1 to 9.
[Appendix 11]
a sensor that senses the surface of the cargo in the loading space and outputs imaged three-dimensional data;
a loading space recognition device according to any one of appendices 1 to 10;
A loading space recognition system.
[Appendix 12]
A loading space recognition method for recognizing a cargo loading space using hardware resources,
a step of estimating an overall image of the cargo loaded in the loading space based on three-dimensional data obtained by imaging the loading space from a predetermined direction and outputting the estimated image as estimation result data;
a step of voxelizing the estimation result data and outputting it as voxel data;
By comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, the amount of change in volume or the amount of movement of the cargo is estimated, and the estimated a step of determining whether or not there is a possibility of cargo collapse by comparing the amount of change in volume or the amount of movement with a threshold value;
A load space recognition method, comprising:
[Appendix 13]
A program for causing a hardware resource to execute a process for recognizing a loading space for cargo,
a process of estimating an overall image of the cargo loaded in the loading space based on three-dimensional data obtained by imaging the loading space from a predetermined direction and outputting the estimation result data;
a process of voxelizing the estimation result data and outputting it as voxel data;
By comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, the amount of change in volume or the amount of movement of the cargo is estimated, and the estimated A process of determining the presence or absence of the possibility of collapse of cargo by comparing the amount of change in volume or the amount of movement with a threshold;
to the hardware resource.

It should be noted that the disclosures of the above patent documents are incorporated herein by reference, and can be used as the basis or part of the present invention as necessary. Within the framework of the entire disclosure of the present invention (including claims and drawings), modifications and adjustments of the embodiments and examples are possible based on the basic technical concept thereof. Also, various combinations or selections of various disclosure elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) within the framework of the full disclosure of the present invention (if necessary not selected) is possible. That is, the present invention naturally includes various variations and modifications that can be made by those skilled in the art according to the entire disclosure including the claims and drawings and the technical idea. Also, with regard to numerical values and numerical ranges described in this application, it is assumed that any intermediate values, sub-numerical values and sub-ranges thereof are described even if not specified. Furthermore, each disclosure matter of the above-cited documents can be used in combination with the descriptions of this document as part of the disclosure of the present invention in accordance with the spirit of the present invention, if necessary. are considered to be included in (belong to) the disclosure of the present application.

1 Loading space recognition system 2 Truck 3 Container 4 Cargo 5 Loading space 10 Sensor 20 Detecting unit 100 Photographed data 101 Preprocessed data 102 Voxel data 200 Loading space recognition device 210 Preprocessing unit 211 Format conversion unit 212 Noise removal unit 220 Packing grasp Section 221 Region Designating Section 222 Parcel Whole Image Estimating Section 223 Voxelizing Section 230 Judging Section 231 Reference Establishing Section 232 Difference Extracting Section 233 Fluctuating Volume Estimating Section 234 Grouping Extracting Section 235 Combination Estimating Section 236 Moving Amount Estimating Section 237 Load Collapse Judging Section 240 User interface unit 241 Display unit 242 Operation unit 243 Warning output unit 1000 Hardware resource 1001 Processor 1002 Memory 1003 Network interface 1004 Internal bus

Claims

An overall luggage image estimation unit configured to estimate an overall image of the luggage loaded in the loading space based on three-dimensional data obtained by imaging the loading space of the luggage from a predetermined direction, and to output estimation result data. When,
a voxelization unit configured to voxelize the estimation result data and output as voxel data;
By comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, the amount of change in volume or the amount of movement of the cargo is estimated, and the estimated a determination unit configured to determine the presence or absence of the possibility of cargo collapse by comparing the amount of change in volume or the amount of movement with a threshold value;
A loading space recognition device.
The determination unit is
a reference determination unit configured to determine the voxel data at the reference time from the voxelization unit as a change reference point;
By comparing the voxel data at the reference time with the voxel data at the time when a predetermined or arbitrary time has passed since the reference time, voxels at corresponding positions on the surface of the package viewed from a predetermined position a difference extraction unit configured to extract the difference between and output as difference data;
a volume variation estimating unit configured to estimate a volume variation of the overall image of the package based on the difference data and output as volume variation estimation data;
a cargo collapse determination unit configured to determine the possibility of cargo collapse by comparing the estimated data of the volume fluctuation and the threshold for the volume fluctuation as the threshold;
The load space perceiving device of claim 1, comprising:
The determination unit is
Based on the difference data, a group of voxels without an increase or decrease in volume existing between an increased volume voxel with an increased volume and a decreased volume voxel with a decreased volume at the same horizontal position a unity extraction unit configured to extract and output as unity data;
a combination estimation unit configured to estimate a combination of the volume increase voxel and the volume decrease voxel based on the difference data and output the combined data;
estimating at least one amount of movement of the cargo during a period from the reference time to a time when a predetermined or arbitrary time has elapsed based on the difference data, the grouped data, and the combination data; a movement amount estimator configured to select the longest movement amount from among the movement amounts, estimate the selected movement amount as the longest movement amount, and output the longest movement amount estimation data;
further comprising
The cargo collapse determination unit compares the estimated data of the volume fluctuation with the threshold for the volume fluctuation as the threshold to determine the possibility of cargo collapse. When it is determined that there is a possibility of cargo collapse, the presence or absence of the possibility of collapse of cargo is determined by comparing the estimated longest movement amount data with the threshold value for the longest movement amount as the threshold value. 3. The loading space recognition device according to claim 2.
The determination unit is
a reference determination unit configured to determine the voxel data at the reference time from the voxelization unit as a change reference point;
By comparing the voxel data at the reference time with the voxel data at the time when a predetermined or arbitrary time has passed since the reference time, voxels at corresponding positions on the surface of the package viewed from a predetermined position a difference extraction unit configured to extract the difference between and output as difference data;
Based on the difference data, a group of voxels without an increase or decrease in volume existing between an increased volume voxel with an increased volume and a decreased volume voxel with a decreased volume at the same horizontal position a unity extraction unit configured to extract and output as unity data;
a combination estimation unit configured to estimate a combination of the volume increase voxel and the volume decrease voxel based on the difference data and output the combined data;
estimating at least one amount of movement of the cargo during a period from the reference time to a time when a predetermined or arbitrary time has elapsed based on the difference data, the grouped data, and the combination data; a movement amount estimator configured to select the longest movement amount from among the movement amounts, estimate the selected movement amount as the longest movement amount, and output the longest movement amount estimation data;
a cargo collapse determination unit configured to determine the possibility of cargo collapse by comparing the longest movement amount estimation data with the threshold for the maximum movement amount as the threshold;
The load space perceiving device of claim 1, comprising:
further comprising an area specifying unit configured to specify a cargo loading area in the loading space by a user's operation;
The luggage overall image estimation unit is configured to estimate the overall image of the luggage loaded in the loading area.
The loading space recognition device according to any one of claims 1 to 4.
The region specifying unit is configured to specify a determination exclusion region to be excluded from determination by the determination unit by user operation,
The luggage overall image estimating unit is configured to exclude the luggage loaded in the determination exclusion area and estimate the overall image of the luggage loaded in the loading area.
The loading space recognition device according to claim 5.
Further comprising a detection unit attached to the structure having the loading space and detecting vibration or sound,
The determination unit acquires the voxel data from the voxelization unit when the detection unit detects shaking or sound of a certain level or more, and compares the voxel data at the reference time and the acquired voxel data. configured to estimate the varying volume or displacement of the load by comparing
The loading space recognition device according to any one of claims 1 to 6.
The movement amount estimator corrects the movement amount to be smaller than the estimated movement amount when the movement direction of the baggage is the horizontal direction, or the estimated movement amount when the movement direction of the baggage is the vertical direction or the oblique direction. corrected so as to be larger than the corrected movement amount, and selecting the longest movement amount from the corrected movement amount.
The loading space recognition device according to claim 3 or 4.
The determination unit further includes: a change amount between the voxel data other than the reference time and the other voxel data immediately before the voxel data; It is configured to determine the presence or absence of the possibility of cargo collapse by comparing the amount of change between the data,
The loading space recognition device according to any one of claims 1 to 8.
The determination unit is configured to output warning output instruction information when it is determined that there is a possibility of cargo collapse,
The loading space recognition device further includes a warning output unit configured to output a warning based on the warning output instruction information.
The loading space recognition device according to any one of claims 1 to 9.
a sensor that senses the surface of the cargo in the loading space and outputs imaged three-dimensional data;
a loading space recognition device according to any one of claims 1 to 10;
A loading space recognition system.
A loading space recognition method for recognizing a cargo loading space using hardware resources,
a step of estimating an overall image of the cargo loaded in the loading space based on three-dimensional data obtained by imaging the loading space from a predetermined direction and outputting the estimated image as estimation result data;
a step of voxelizing the estimation result data and outputting it as voxel data;
By comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, the amount of change in volume or the amount of movement of the cargo is estimated, and the estimated a step of determining whether or not there is a possibility of cargo collapse by comparing the amount of change in volume or the amount of movement with a threshold value;
A load space recognition method, comprising:
A program for causing a hardware resource to execute processing for recognizing a loading space for cargo,
a process of estimating an overall image of the cargo loaded in the loading space based on three-dimensional data obtained by imaging the loading space from a predetermined direction and outputting the estimation result data;
a process of voxelizing the estimation result data and outputting it as voxel data;
By comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, the amount of change in volume or the amount of movement of the cargo is estimated, and the estimated A process of determining the presence or absence of the possibility of collapse of cargo by comparing the amount of change in volume or the amount of movement with a threshold;
to the hardware resource.