WO2020215772A1 - 一种智能叉车以及容器位姿偏移检测方法 - Google Patents

一种智能叉车以及容器位姿偏移检测方法 Download PDF

Info

Publication number
WO2020215772A1
WO2020215772A1 PCT/CN2019/128048 CN2019128048W WO2020215772A1 WO 2020215772 A1 WO2020215772 A1 WO 2020215772A1 CN 2019128048 W CN2019128048 W CN 2019128048W WO 2020215772 A1 WO2020215772 A1 WO 2020215772A1
Authority
WO
WIPO (PCT)
Prior art keywords
container
transported
image
feature
depth
Prior art date
Application number
PCT/CN2019/128048
Other languages
English (en)
French (fr)
Inventor
郭晓丽
杨宁宁
Original Assignee
北京极智嘉科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京极智嘉科技有限公司 filed Critical 北京极智嘉科技有限公司
Priority to KR1020217035759A priority Critical patent/KR102461759B1/ko
Priority to JP2021563272A priority patent/JP7206421B2/ja
Priority to EP19926162.9A priority patent/EP3960692A4/en
Publication of WO2020215772A1 publication Critical patent/WO2020215772A1/zh
Priority to US17/508,784 priority patent/US11625854B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B66—HOISTING; LIFTING; HAULING
    • B66F—HOISTING, LIFTING, HAULING OR PUSHING, NOT OTHERWISE PROVIDED FOR, e.g. DEVICES WHICH APPLY A LIFTING OR PUSHING FORCE DIRECTLY TO THE SURFACE OF A LOAD
    • B66F9/00—Devices for lifting or lowering bulky or heavy goods for loading or unloading purposes
    • B66F9/06—Devices for lifting or lowering bulky or heavy goods for loading or unloading purposes movable, with their loads, on wheels or the like, e.g. fork-lift trucks
    • B66F9/075—Constructional features or details
    • B66F9/0755—Position control; Position detectors
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B66—HOISTING; LIFTING; HAULING
    • B66F—HOISTING, LIFTING, HAULING OR PUSHING, NOT OTHERWISE PROVIDED FOR, e.g. DEVICES WHICH APPLY A LIFTING OR PUSHING FORCE DIRECTLY TO THE SURFACE OF A LOAD
    • B66F17/00—Safety devices, e.g. for limiting or indicating lifting force
    • B66F17/003—Safety devices, e.g. for limiting or indicating lifting force for fork-lift trucks
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
    • G01C11/04—Interpretation of pictures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds

Definitions

  • the embodiments of the present application relate to the technical field of smart storage, for example, to a smart forklift and a method for detecting the displacement of a container.
  • Forklifts are playing an increasingly important role in the warehousing field. Forklifts carry inventory containers such as pallets to carry and store items.
  • the forklift can be manually driven to move to the storage position of the inventory container to be transported, and the inventory container to be transported can be lifted by the fork of the forklift to pick up the inventory container to be transported, so that during the transportation process, the staff can pass the time Pay attention to the position of the inventory container to be moved on the fork to realize safe and stable transportation and storage of items to the designated storage location.
  • the smart storage environment it is also possible to preset a moving route for the smart forklift so that the smart forklift moves to the storage location of the inventory container to be transported according to the preset route, and lifts it by identifying the bottom surface of the inventory container. To-be-moved inventory containers and carry out operations such as handling and storing the to-be-moved inventory containers.
  • the inventory container on the Internet performs intelligent posture detection, which reduces the efficiency and safety of forklift handling inventory containers.
  • the embodiments of the present application provide an intelligent forklift and a container posture deviation detection method, which can perform intelligent posture detection of the inventory container on the fork, and improve the efficiency and safety of the forklift in handling the inventory container.
  • the embodiment of the present application provides an intelligent forklift, which includes: a working state monitoring module, a container image acquisition module, and a processing module;
  • the container image acquisition module is electrically connected to the working state monitoring module and the processing module; wherein:
  • the working state monitoring module is configured to monitor the working state of the smart forklift carrying inventory containers to be transported, and according to the working state, control the container image acquisition module to perform image acquisition;
  • the container image acquisition module is configured to respond to a trigger of the working status monitoring module, acquire an RGBD image frame including the inventory container to be transported, and transmit the RGBD image frame to the processing module;
  • the processing module is configured to receive the RGBD image frame fed back by the container image acquisition module, and detect the degree of pose deviation of the inventory container to be transported according to the RGBD image frame; and according to the detection result of the degree of pose deviation An alarm is issued to adjust the position of the inventory container to be transported on the smart forklift.
  • the embodiment of the present application provides a method for detecting a container pose shift, including:
  • RGBD image frames including the inventory container to be transported; wherein the RGBD image frames include RGB images and depth images;
  • the preset depth feature and area feature of the template to be compared, and the preset depth feature threshold and the preset area feature threshold, the current position of the inventory container to be transported is detected The degree of attitude shift.
  • the embodiment of the present application provides a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium.
  • the program is executed by a processor, the container pose shift detection method described in any embodiment of the present application is implemented.
  • FIG. 1 is an example diagram of a smart forklift carrying inventory containers according to an embodiment of the application
  • FIG. 2 is a schematic diagram of the structure of a storage container provided by an embodiment of the application.
  • FIG. 3 is a schematic structural diagram of a smart forklift provided by an embodiment of the application.
  • FIG. 4 is a structural block diagram of another smart forklift provided by an embodiment of the application.
  • FIG. 5 is a flowchart of a method for detecting a container pose shift provided by an embodiment of the application
  • FIG. 6 is a flowchart of another method for detecting the position and orientation deviation of a container provided by an embodiment of the application.
  • FIG. 1 Please refer to Figure 1 for an example of a smart forklift handling inventory containers.
  • the smart forklift 110 lifts the inventory container 120 through the fork so that the inventory container 120 is located on the fork of the smart forklift 110 to carry out the handling operation of the inventory container 120.
  • the sidewall of the inventory container 120 has a fork hole structure corresponding to the fork of the smart forklift 110, such as a pallet, and the smart forklift 110 inserts the fork into the fork hole structure to carry the inventory container 120 and provide a good response to the inventory container. 120 for transportation.
  • An article 130 may be placed on the storage container 120.
  • the inventory container 120 may be located in a storage area, for example, on a shelf compartment in the storage area, and may also be located in a sorting area such as a workstation. Therefore, the smart forklift 110 carries out the handling operation of the inventory container 120 in the storage environment according to requirements.
  • Figure 2 is a schematic structural diagram of a storage container provided by an embodiment of the application.
  • the storage container 120 may be a regular or irregular rectangular parallelepiped.
  • the storage container 120 has four side walls, and at least two opposite side walls of the storage container 120 have a fork hole structure 200.
  • the fork hole structure 200 includes the fork hole sub-structure 210 and the frame sub-structure 220 of the storage container 120 itself.
  • the item 130 can be placed directly on the inventory container 130, and the item 130 can be placed on the inventory container 120 in any suitable manner, for example, a bin storing the item 130 is placed on the inventory container 120.
  • the material box may be separated from the inventory container 120, or may be an integrated structure with the inventory container 120, and one or more items 130 may be placed in the material box.
  • Fig. 3 is a schematic structural diagram of a smart forklift provided by an embodiment of the application. 1 and 3, in an example, the smart forklift 110 may include a fork 111 and a sliding rail 112, through the sliding rail 112, the fork 111 of the smart forklift 110 can be raised and lowered within a certain height range.
  • the smart forklift 110 can move to the storage location of the inventory area such as the exit of the shelf, and insert the fork 111 into the fork hole structure 200 of the inventory container 120 to carry the inventory container 120, and transport the inventory container 120 to a designated location, such as being allocated Connection location or workstation, etc. Workers (or automated equipment such as robotic arms) at the picking workstation pick out items from the inventory container 120.
  • the embodiment of the present application is not limited to transporting items to a designated location such as a docking location or a workstation, and any transport destination can be applied to this embodiment.
  • the smart forklift 110 includes at least a container image acquisition module 113, such as a depth camera, and the image acquisition module 113 is capable of checking the fork 111, the inventory container 120 on the fork 111, and the items on the inventory container 120. 130 performs image collection to obtain an image that completely contains the inventory container 120 and the article 130.
  • the smart forklift 110 further includes a processing module 114, which is configured to process the images collected by the container image acquisition module 113, so as to realize the detection of the displacement degree of the inventory container 120.
  • the smart forklift 110 triggers the container image acquisition module 113 to collect the image of the inventory container 120 on the fork 111 when it detects that the fork 111 completes the lifting action of the inventory container 120.
  • the processing module 114 collects Through the identification of the inventory container 120, the current position and attitude deviation degree of the inventory container 120 is determined, so that according to the detection result of the position and attitude deviation degree, the brake and alarm prompts are performed to adjust the inventory container 130 in stock After the position on the fork 111, the handling operation is continued.
  • the forklift can be manually driven to move to the storage location of the inventory container to be transported, and the inventory container to be transported can be lifted by the fork of the forklift to pick up the inventory container to be transported, so as to work during the transportation process.
  • the personnel can safely and stably move and store the items to the designated storage location.
  • it is also possible to preset a moving route for the smart forklift so that the smart forklift moves to the storage location of the inventory container to be transported according to the preset route, and lifts it by identifying the bottom surface of the inventory container.
  • To-be-moved inventory containers and carry out operations such as handling and storing the to-be-moved inventory containers.
  • it is difficult for humans to accurately and effectively observe the posture of the inventory container lifted on the fork which increases the difficulty of operation and the cost of manual intervention, and the intelligent forklift cannot realize the intelligent posture detection of the inventory container on the fork. , Which reduces the efficiency and safety of forklifts for handling inventory containers.
  • FIG. 4 is a structural block diagram of a smart forklift provided by an embodiment of the application. This embodiment is applicable to a situation where a smart forklift carries inventory containers.
  • the smart forklift 400 includes: a working status monitoring module 410, a container image acquisition module 420, and processing Module 430;
  • the container image acquisition module 420 is electrically connected to the working state monitoring module 410 and the processing module 430.
  • the working status monitoring module 410 is configured to monitor the working status of the smart forklift carrying inventory containers to be transported, and according to the working status, control the container image acquisition module 420 to perform image collection; the container image acquisition module 420 is configured to respond The working status monitoring module is triggered to collect the RGBD image frame including the inventory container to be transported, and transmit the RGBD image frame to the processing module 430; the processing module 430 is configured to receive the RGBD image frame fed back from the container image acquisition module 420, and the basis The RGBD image frame detects the degree of pose deviation of the inventory container to be transported; and gives an alarm based on the detection result of the degree of pose deviation to adjust the position of the inventory container to be transported on the smart forklift.
  • the working status monitoring module 410 has the function of monitoring the working status of multiple mechanical structures of the smart forklift, and can be installed in the smart forklift in the form of hardware.
  • the installation position of the working status monitoring module 410 is not limited, or can be The form of software is integrated in the software processing program of the intelligent forklift.
  • the working status monitoring module 410 can monitor the working status of the smart forklift carrying inventory containers based on data such as the position of the smart forklift, the lift height of the fork, the weight of the fork, and the progress of the smart forklift path. The state is being lifted, the storage container is being moved, and the storage container has been put down.
  • the working state of the container image acquisition module 420 is controlled, for example, the container image acquisition module 420 is triggered to perform image acquisition, or the image acquisition is terminated.
  • the container image acquisition module 420 has an image acquisition function, is installed on the smart forklift, and is configured to perform image acquisition on the inventory container on the fork of the intelligent forklift and the carried items on the inventory container in real time. After the container image acquisition module 420 is triggered to start image acquisition, it performs continuous image acquisition of multiple frames. There may be one or more container image collection modules 420 installed on the smart forklift. For example, a container image collection module 420 is installed at each height position that the fork of the smart forklift can pass, or a container image collection module 420 is installed above the fork, that is, at the top of the smart forklift.
  • the processing module 430 may have an image recognition function, a data processing function, a container pose detection function, and an alarm function.
  • the processing module 430 is installed in the smart forklift, and the installation position of the processing module 430 is not limited.
  • the processing module 430 may receive the image frames fed back from the container image acquisition module 420 in real time, perform inventory container recognition on the image frames, and determine the characteristics of the inventory container based on the recognition results, and finally perform the degree of pose shift based on the characteristics of the inventory container Detection etc.
  • connection mode between the container image acquisition module 420 and the working state monitoring module 410 and the connection mode between the container image acquisition module 420 and the processing module 430 may be electrical connection or communication connection. Electric connection or communication connection for data or command transmission.
  • the inventory container to be transported refers to the inventory container that can be transported by a forklift in a storage environment, such as a pallet.
  • this type of inventory container to be transported has a fork hole structure that matches the fork of the forklift, and is used for the insertion of the fork to lift the inventory container to be transported and carry the inventory container to be transported.
  • This type of inventory container to be moved can be located anywhere in the storage environment, for example, in a storage area. In an embodiment, the inventory container to be transported may be located on the compartment of the shelf in the storage area.
  • the fork of the smart forklift when the fork of the smart forklift is raised to the height of the inventory container to be transported, when the fork is aligned with the fork hole structure of the inventory container to be transported, the fork is inserted into the fork hole structure of the inventory container to be transported, thereby
  • the intelligent forklift carries the inventory containers to be moved, and realizes the operations of picking up and moving the inventory containers to be moved.
  • the container image collection module 420 may be an RGBD image sensor such as Kinect, and is configured to collect RGBD (RGB+Depth Map) images.
  • RGBD RGB+Depth Map
  • Each pixel in the RGBD image includes R (Red), G (Green, green), B (Blue, blue) color pixel information and corresponding depth information.
  • the RGB color information of each pixel in the RGBD image constitutes an RGB image
  • the depth information of each pixel in the RGBD image constitutes the two-dimensional pixel matrix of the scene, that is, the depth image.
  • the value of each pixel in the depth image represents the distance between the object corresponding to that pixel and the plane where the container image acquisition module 420 is located.
  • the position of each pixel corresponds to the position of the object corresponding to the pixel in the scene. Corresponds to the position in the associated RGB image.
  • the smart forklift when the inventory container to be transported is transported, the smart forklift follows a predetermined route to the storage location of the inventory container to be transported, such as a storage area or a shelf area where pallets with a fork hole structure are stored. Under the control of the intelligent forklift, the forklift is lifted to the height of the storage position of the inventory container to be transported.
  • the working status monitoring module 410 can monitor the height of the fork, the load-bearing weight, or the operation of the fork. If it is detected that the fork lifts to the height of the inventory container to be transported, and the inventory container to be transported is completed The lifting action of the container triggers the container image acquisition module 420 to continuously acquire RGBD image frames.
  • the container image acquisition module 420 responds to the trigger of the working status monitoring module, acquires RGBD image frames including the inventory container to be transported, and transmits the RGBD image frames to the processing module 430.
  • the processing module 430 receives the RGBD image frame fed back by the container image acquisition module 420, and can identify the inventory container to be transported in the RGBD image frame based on the predetermined container recognition model, and determine the RGB image to be transported from the RGB image of the current RGBD image frame.
  • the position of the storage container in the RGB image is intercepted.
  • the target region of interest with depth information is used to determine the depth feature and area feature of the target region of interest, and the current pose deviation degree of the inventory container to be transported is detected.
  • the detection of the degree of pose deviation may include the detection of the relative deviation between the inventory container to be transported and the fork, and may also include the relative deviation between the items on the inventory container to be transported and the inventory container to be transported. Degree of detection.
  • the smart forklift of the embodiment of the present application always detects the position deviation degree of the inventory container to be transported on the fork and the items placed on the inventory container to be transported, so as to determine the fork
  • brake and alarm will be performed to remind the staff or other robots to pose the inventory container and/or item to be moved on the fork of the smart forklift Adjustment.
  • the transportation is continued until the smart forklift transports the inventory container to be transported to the destination, and puts down the inventory container to be transported, and terminates the image collection and detection.
  • the technical solution of this embodiment uses the container image acquisition module installed in the smart forklift to perform RGBD image acquisition on the storage container to be transported on the fork during the process of transporting the inventory container, so that the RGBD image frame is collected at all times.
  • the storage container to be transported on the fork detects the degree of posture deviation, and an alarm is given according to the detection result of the degree of posture deviation to adjust the posture of the storage container to be transported on the smart forklift.
  • the embodiment of the present application collects and recognizes the RGBD image containing the storage container to be transported on the fork, and realizes the effective detection of the posture deviation of the storage container to be transported on the fork, avoiding the Due to external forces or emergency brakes of the forklift, the position of the inventory container to be transported on the fork changes and cannot be transported safely, thereby improving the efficiency and safety of the forklift for transporting the inventory container.
  • This embodiment provides an implementation of a smart forklift on the basis of the first embodiment above, which can detect the degree of displacement of the inventory container to be transported by identifying and determining the characteristics of the inventory container to be transported.
  • the container image acquisition module 420 includes at least one camera, and the camera is installed above the fork of the smart forklift.
  • the RGBD image frame includes an RGB image and a depth image, and the image used to represent the inventory container to be transported in the RGBD image frame is located in a designated area in the RGBD image frame.
  • the working status monitoring module 410 is configured to control the container image acquisition module 420 to perform image collection according to the working status in the following manner: when it is detected that the smart forklift completes the lifting action of the inventory container to be transported, trigger The container image acquisition module 420 performs image acquisition; when it is monitored that the smart forklift completes the placing action of the inventory container to be transported, the image acquisition action of the container image acquisition module 420 is terminated.
  • the container image acquisition module 420 may be a device capable of acquiring RGBD images, such as a depth camera or a Kinect sensor.
  • the container image acquisition module 420 includes at least one camera, and the camera may be installed on each height gear that the fork can pass, or installed above the fork, that is, the top of the smart forklift. Furthermore, after the fork completes the lifting action of the inventory container to be transported, the camera can project and collect the RGBD image frame that completely contains the inventory container to be transported and the items on the inventory container to be transported.
  • the collected images may include invalid surrounding environment images such as the ground.
  • the position of the inventory container in the image can be determined according to the height of the fork and the inherent relative position relationship between the fork and the camera at this height, so that when the image is obtained, the image can represent the inventory container to be transported.
  • the image is usually located in a designated area in the image.
  • the working status monitoring module 410 can monitor the height of the fork, the load-bearing weight, or the operation of the fork. If it is monitored that the fork lifts to the height of the inventory container to be transported, and completes the lifting of the inventory container to be transported When the action is started, the container image acquisition module 420 is triggered to continuously acquire RGBD image frames. If it is detected that the fork lifts to the height of the target storage position and the lowering action of the inventory container to be transported is completed, the container image acquisition module 420 is controlled to stop the image acquisition action.
  • the processing module 430 is configured to detect the pose shift degree of the inventory container to be transported according to the RGBD image frame in the following manner: according to a predetermined container recognition model, determine from the RGB image of the current RGBD image frame The position of the storage container to be transported in the RGB image; according to the position of the storage container to be transported in the RGB image, from the depth image of the current RGBD image frame, intercept the target region of interest containing the storage container to be transported; determine the sense of object The depth feature and area feature of the region of interest, and based on the depth feature and area feature of the target region of interest, and the preset depth feature and area feature of the template to be compared, determine the depth feature difference and the area feature difference; according to the depth feature The difference value and the area characteristic difference value, as well as the preset depth characteristic threshold value and the preset area characteristic threshold value, detect the current pose deviation degree of the inventory container to be transported.
  • the processing module 430 is configured to determine the depth feature and the area feature of the target region of interest in the following manner: perform down-sampling processing on the target region of interest according to a preset number of layers to obtain the down-sampling feeling Interest region image; sum the depth values of all pixels in the downsampled region of interest image to obtain the depth feature of the target region of interest; determine the area of the downsampled region of interest image to obtain the target region of interest Area characteristics.
  • the preset template to be compared includes at least a standard container template and a previous RGBD image frame of the current RGBD image frame.
  • the standard container template is calculated based on the mean value of the depth feature and the mean value of the area feature of a preset number of RGBD image frames initially collected.
  • the processing module 430 is configured to detect the current pose deviation of the inventory container to be transported according to the depth feature difference value and the area feature difference value, and the preset depth feature threshold value and the preset area feature threshold value in the following manner Degree: According to the depth feature difference and area feature difference between the current RGBD image frame and the standard container template, the depth feature difference and area feature difference between the current RGBD image frame and the previous RGBD image frame of the current RGBD image frame The difference, and the preset standard depth feature threshold, standard area feature threshold, adjacent feature threshold, and adjacent area feature threshold. When it is detected that at least one feature difference is greater than the corresponding feature threshold, determine the current moment, The position of the inventory container to be moved on the smart forklift has shifted.
  • the container identification model refers to a model for identifying the inventory container to be transported on the fork, which can be obtained through pre-training.
  • multiple scenarios can be preset, and a large number of sample RGB images during the process of the smart forklift moving the inventory container in multiple scenarios are collected.
  • the sample RGB image may contain complete, partial, or excluding inventory containers and items on the inventory containers.
  • the sample RGB image is classified, and the RGB image containing the inventory container is taken as the positive sample, and the RGB image not containing the inventory container is taken as the negative sample.
  • a support vector machine (Support Vector Machine, SVM) can be used to train the container recognition model to obtain the container recognition model.
  • SVM Support Vector Machine
  • the trained SVM model is used to identify the RGB image of the current RGBD image frame, add the smallest bounding rectangle to the identified inventory container to be transported, and record at least two pairs of the smallest bounding rectangle in the RGB image
  • the corner vertex coordinates are used as the position of the storage container to be transported in the RGB image, such as the upper left corner vertex coordinates and the lower right corner vertex coordinates, through which the position and size of the minimum bounding rectangle can be expressed.
  • the camera of the container image acquisition module 420 performs image acquisition during the movement of the smart forklift. During this period, the relative position of the camera and the smart forklift is fixed. Under normal working conditions, the positions of the inventory containers and items to be transported are different. change.
  • a filter can be used to denoise the depth image.
  • a Gaussian low-pass filter can be used to smooth the depth image to remove noise caused by camera shake or laser reception interference in the depth image, so as to obtain accurate distance information for each pixel.
  • the size of the Gaussian filter template used is 3 ⁇ 3 or 5 ⁇ 5, and the standard deviation is 1.
  • the denoising algorithm of the depth image in this embodiment is not limited to the Gaussian low-pass filter, and any algorithm that can realize the denoising of the depth image can be applied in this embodiment.
  • the target region of interest includes the storage container to be transported and contains depth information of the storage container to be transported.
  • the depth feature and the area feature are used as the key information of the target region of interest, and the inventory container's pose deviation degree is detected based on the key information.
  • the depth feature refers to the sum of the depth values of all pixels in the depth image
  • the area feature refers to the total area of the depth image. Since the pixel resolution of the original image is relatively high, in order to improve the detection efficiency, in this embodiment, after obtaining the target region of interest, a pyramid image may be constructed on the target region of interest first to perform down-sampling processing on the target region of interest. In an embodiment, the number of layers of the pyramid image may be determined according to the camera focal length of the RGBD sensor. Therefore, for the down-sampling image of the region of interest, the sum of the depth values of all pixels in the region of interest image is calculated as the depth feature, and the total area of the region of interest image is calculated as the area feature.
  • the preset template to be compared may refer to the depth image of the inventory container with a safe pose state, or may refer to the depth image of the image frame before the current RGBD image frame.
  • the depth image average value of a preset number of RGBD image frames initially collected can be used as the preliminary Set the template to be compared; in addition, because when the current RGBD image frame is detected, the image frames before the current RGBD image frame are all detected without pose offset or within the safe offset range, so the current The depth image of the previous RGBD image of the RGBD image frame is used as the preset template to be compared.
  • the depth feature and area feature of the preset template to be compared can be determined.
  • the depth feature and area feature of the target region of interest are respectively compared with the depth feature and area feature of the preset template to be compared to obtain the depth feature difference and the area feature difference.
  • the depth feature threshold and the area feature threshold are determined in advance according to the degree of dangerous deviation in different scenarios. That is, the degree of pose deviation is determined based on the depth feature difference value and the area feature difference value. If the pose deviation limit defined by the depth feature threshold value and/or the area feature threshold value is exceeded, it is determined that the inventory container to be transported currently has a large deviation. There are potential safety hazards, which can easily lead to inability to safely or normally carry, or affect the normal handling of other forklifts or robots in the storage environment.
  • the processing module 430 controls the braking of the smart forklift, and sends an alarm prompt so that the staff or other robots can perform inspections on the inventory containers and/or items to be moved on the forks of the smart forklift.
  • Adjustment of posture After the posture is adjusted, the transportation is continued until the smart forklift transports the inventory container to be transported to the destination and puts down the inventory container to be transported. At this time, image acquisition and detection are stopped.
  • the smallest bounding rectangle of the inventory container to be transported in the image is determined, and the smallest bounding rectangle coordinate information (X LeftTop , Y LeftTop ) and (X RightBottem , Y RightBottem ), and use the coordinate information as the position of the storage container to be transported in the RGB image.
  • Denoising is obtained by performing Gaussian low-pass filtering on the depth image in the current RGBD image frame to obtain the denoised depth image D filter . According to the positions (X LeftTop , Y LeftTop ) and (X RightBottem , Y RightBottem ), the corresponding rectangular area is intercepted on the D filter as the target region of interest R D1 .
  • the image R D11 is obtained after the first downsampling, and R D11 is 1/2 of the size of R D1 ; the image R D12 is obtained after the second downsampling, and R D12 is 1/4 of the size of R D1 ; the third downsampling is Then the image R D13 is obtained, and R D13 is 1/8 of the size of R D1 .
  • the depth values of all pixels in the image R D13 are summed to obtain the depth feature d, and the area of the image R D13 is taken as the area feature a. That is, the key information of the current RGBD image frame can be [R D13 ,d,a].
  • the average value of the first three frames triggered by the container image acquisition module 420 is used as the preset template to be compared.
  • the average value of the key information of the first three frames is used as the depth feature and image feature of the preset template to be compared, namely Src[R D13 ,d,a].
  • the previous RGBD image frame of the current RGBD image frame is used as the preset template to be compared, and the key information composed of the depth feature and image feature of the previous RGBD image frame is determined according to the calculation process of the above-mentioned depth feature and image feature Is Current[R D13 ,d,a].
  • the depth feature threshold between the current RGBD image frame and the initial three RGBD image frames is D CurrSrc and the area feature threshold S CurrSrc are predetermined according to the degree of dangerous shift in different scenes
  • the current RGBD image frame and the current RGBD image are predetermined
  • the depth feature threshold between the previous RGBD image frames of the frame is D CurrLast
  • the area feature threshold S CurrLast is the area feature threshold between the previous RGBD image frames of the frame.
  • adjacent frames are continuously changing, so the area of two adjacent image frames is similar to the number of pixels under normal operating conditions.
  • the technical solution of this embodiment triggers the container image collection module to perform RGBD image collection on the storage container to be transported on the fork when it is detected that the smart forklift lifts the storage container.
  • the container image collection module By identifying the inventory container of the RGB image of the current RGBD image frame, the position of the inventory container to be transported in the RGB image is determined, and the target region of interest of the depth image of the current RGBD image frame is intercepted according to the position.
  • the depth feature and area feature of the target region of interest are determined, the feature is compared with the feature threshold according to the preset template to be compared, and the posture shift of the storage container to be carried on the fork is performed according to the result of the feature comparison
  • an alarm will be given according to the detection result of the position and attitude deviation degree.
  • the embodiment of the present application collects and recognizes the RGBD image containing the storage container to be transported on the fork, and realizes the effective detection of the posture deviation of the storage container to be transported on the fork, avoiding the Due to external forces or emergency brakes of the forklift, the position of the inventory container to be transported on the fork changes and cannot be transported safely, thereby improving the efficiency and safety of the forklift for transporting the inventory container.
  • FIG. 5 is a flowchart of a method for detecting the displacement of a container according to an embodiment of the application. This embodiment can be applied to the case of a smart forklift carrying inventory containers.
  • the method can be executed by a device for detecting the displacement of a container.
  • the device can be implemented in software and/or hardware.
  • the device is configured in a smart forklift. The method includes the following steps:
  • Step 510 When it is monitored that the smart forklift completes the lifting action of the inventory container to be transported, collect RGBD image frames including the inventory container to be transported.
  • the inventory container to be transported refers to the inventory container that can be transported by a forklift in a storage environment, such as a pallet.
  • this type of inventory container to be transported has a fork hole structure that matches the fork of the forklift, and is used for the insertion of the fork to lift the inventory container to be transported and carry the inventory container to be transported.
  • This type of inventory container to be moved can be located anywhere in the storage environment, for example, in a storage area. In an embodiment, the inventory container to be transported may be located on the compartment of the shelf in the storage area.
  • the fork of the smart forklift when the fork of the smart forklift is raised to the height of the inventory container to be transported, when the fork is aligned with the fork hole structure of the inventory container to be transported, the fork is inserted into the fork hole structure of the inventory container to be transported, thereby
  • the intelligent forklift carries the inventory containers to be moved, and realizes the operations of picking up and moving the inventory containers to be moved.
  • RGBD image sensors such as Kinect can be used for image collection.
  • the collected images are RGBD images (RGB+Depth Map).
  • Each pixel in the RGBD image includes R (Red, red), G (Green, Green), B (Blue, blue) color pixel information and corresponding depth information.
  • the RGB color information of each pixel in the RGBD image constitutes an RGB image
  • the depth information of each pixel in the RGBD image constitutes the two-dimensional pixel matrix of the scene, that is, the depth image.
  • Each pixel value in the depth image represents the distance between the object corresponding to the pixel and the plane where the RGBD image sensor is located.
  • the position of each pixel corresponds to the position of the object corresponding to the pixel in the scene, and is associated with the RGB image Corresponds to the position in.
  • the smart forklift when the inventory container to be transported is transported, the smart forklift follows a predetermined route to the storage location of the inventory container to be transported, such as a storage area or a shelf area where pallets with a fork hole structure are stored. Under the control of the intelligent forklift, the forklift is lifted to the height of the storage position of the inventory container to be transported. In one embodiment, the height of the fork, the load-bearing weight, or the operation of the fork can be monitored. If it is monitored that the fork lifts to the height of the inventory container to be transported, and the lifting action of the inventory container to be transported is completed, Then trigger continuous acquisition of RGBD image frames.
  • Step 520 Determine the position of the storage container to be transported in the RGB image from the RGB image of the current RGBD image frame according to the predetermined container recognition model.
  • the storage container to be transported in the RGBD image frame can be identified based on a predetermined container identification model, and the position of the storage container to be transported in the RGB image is determined from the RGB image of the current RGBD image frame.
  • the container recognition model refers to the model used to identify the inventory container to be moved on the fork, which can be obtained through pre-training.
  • multiple scenarios can be preset, and a large number of sample RGB images during the process of the smart forklift moving the inventory container in multiple scenarios are collected.
  • the sample RGB image may contain complete, partial, or excluding inventory containers and items on the inventory containers.
  • SVM can be used to train the container recognition model to obtain the container recognition model.
  • the trained SVM model is used to identify the RGB image of the current RGBD image frame, add the smallest bounding rectangle to the identified inventory container to be transported, and record at least two pairs of the smallest bounding rectangle in the RGB image
  • the corner vertex coordinates are used as the position of the storage container to be transported in the RGB image, such as the upper left corner vertex coordinates and the lower right corner vertex coordinates, through which the position and size of the minimum bounding rectangle can be expressed.
  • Step 530 According to the position of the storage container to be transported in the RGB image, intercept the target region of interest containing the storage container to be transported from the depth image of the current RGBD image frame.
  • the RGB image of the current RGBD image frame and each pixel in the depth image have an associated corresponding relationship, they represent different image information in the same scene. Therefore, according to the position of the storage container to be transported in the RGB image, the same rectangular area can be intercepted as the target area of interest from the depth image of the current RGBD image frame.
  • the target region of interest includes the storage container to be transported and contains depth information of the storage container to be transported.
  • Step 540 According to the depth feature and area feature of the target region of interest, preset depth feature and area feature of the template to be compared, and preset depth feature threshold and preset area feature threshold, detect the current pose of the inventory container to be transported The degree of deviation.
  • the depth feature and the area feature are used as the key information of the target region of interest, and the degree of shift of the inventory container is detected based on the key information.
  • the depth feature refers to the sum of the depth values of all pixels in the depth image
  • the area feature refers to the total area of the depth image.
  • the preset template to be compared may refer to the depth image of the inventory container with a safe pose state, or may refer to the depth image of the image frame before the current RGBD image frame.
  • the depth image average value of the preset number of RGBD image frames initially collected can be used as the preset The template to be compared; in addition, because when the current RGBD image frame is detected, the image frames before the current RGBD image frame are all detected without pose offset or within the safe offset range, so the current RGBD The depth image of the previous RGBD image frame of the image frame is used as the preset template to be compared.
  • the depth feature and area feature of the preset template to be compared can be determined.
  • the depth feature and area feature of the target region of interest are respectively compared with the depth feature and area feature of the preset template to be compared to obtain the depth feature difference and the area feature difference.
  • the depth feature threshold and the area feature threshold are determined in advance according to the degree of dangerous deviation in different scenarios. That is, the degree of pose deviation is determined based on the depth feature difference value and the area feature difference value. If the pose deviation limit defined by the depth feature threshold value and/or the area feature threshold value is exceeded, it is determined that the inventory container to be transported currently has a large deviation. There are potential safety hazards, which can easily lead to inability to safely or normally carry, or affect the normal handling of other forklifts or robots in the storage environment.
  • the technical solution of this embodiment triggers the image collection function when monitoring that the smart forklift completes the lifting action of the inventory container to be transported, so that the storage container to be transported on the fork is continuously processed during the process of transporting the inventory container.
  • RGBD image collection based on the RGBD image frames collected at all times, the detection of the position and attitude deviation of the storage container to be carried on the fork, and an alarm prompt based on the detection result of the position and position deviation to adjust the storage container to be carried The pose on the smart forklift.
  • the embodiment of the present application collects and recognizes the RGBD image containing the storage container to be transported on the fork, and realizes the effective detection of the posture deviation of the storage container to be transported on the fork, avoiding the Due to external forces or emergency brakes of the forklift, the position of the inventory container to be transported on the fork changes and cannot be transported safely, thereby improving the efficiency and safety of the forklift for transporting the inventory container.
  • this embodiment provides an implementation of the method for detecting the position and orientation of the container, which can detect the position and orientation of the inventory container to be transported by identifying and determining the characteristics of the inventory container to be transported. degree.
  • Fig. 6 is a flowchart of another method for detecting a container pose shift provided by an embodiment of the application. As shown in Fig. 6, the method includes the following steps:
  • Step 610 When it is detected that the smart forklift completes the lifting action of the inventory container to be transported, collect RGBD image frames including the inventory container to be transported.
  • Step 620 Determine the position of the storage container to be transported in the RGB image from the RGB image of the current RGBD image frame according to the predetermined container recognition model.
  • the trained SVM model is used to identify the RGB image of the current RGBD image frame, add the smallest bounding rectangle to the identified inventory container to be transported, and record at least two of the smallest bounding rectangles in the RGB image
  • the coordinates of the diagonal vertices are used as the position of the storage container to be transported in the RGB image, and the position and size of the smallest bounding rectangle can be expressed by this position.
  • the smallest bounding rectangle of the inventory container to be transported in the image is determined, and the smallest bounding rectangle coordinate information (X LeftTop , Y LeftTop ) and (X RightBottem , Y RightBottem ), and use the coordinate information as the position of the storage container to be transported in the RGB image.
  • Step 630 According to the position of the storage container to be transported in the RGB image, intercept the target region of interest containing the storage container to be transported from the depth image of the current RGBD image frame.
  • the image sensor such as the depth camera collects images during the movement of the smart forklift. During this period, the image sensor and the smart forklift position are fixed relative to each other. Under normal working conditions, the position of the inventory containers and items to be transported constant.
  • a filter can be used to denoise the depth image.
  • a Gaussian low-pass filter can be used to smooth the depth image to remove noise in the depth image caused by image sensor vibration or laser reception interference, so as to obtain accurate distance information for each pixel.
  • the size of the Gaussian filter template used is 3 ⁇ 3 or 5 ⁇ 5, and the standard deviation is 1.
  • the denoising algorithm of the depth image in this embodiment is not limited to the Gaussian low-pass filter, and any algorithm that can realize the denoising of the depth image can be applied in this embodiment.
  • the target region of interest includes the storage container to be transported and contains depth information of the storage container to be transported.
  • the depth image in the current RGBD image frame is denoised by Gaussian low-pass filtering to obtain the denoised depth image D filter .
  • the position (X LeftTop , Y LeftTop ) and (X RightBottem , Y RightBottem ) intercept the corresponding rectangular area on the D filter as the target area of interest.
  • Step 640 Determine the depth feature and area feature of the target region of interest.
  • this embodiment can first construct a pyramid image of the target region of interest to perform the calculation of the target region of interest. Downsampling processing.
  • the number of layers of the pyramid image may be determined according to the camera focal length of the RGBD sensor. Therefore, for the down-sampling image of the region of interest, the sum of the depth values of all pixels in the region of interest image is calculated as the depth feature, and the total area in the region of interest image is calculated as the area feature.
  • the target region of interest is down-sampled according to a preset number of layers to obtain a down-sampled region-of-interest image; the depth values of each pixel of the down-sampled region-of-interest image are summed To obtain the depth feature of the target region of interest; determine the area of the down-sampled image of the region of interest to obtain the area feature of the target region of interest.
  • the number of pyramid image layers is 3, down-sampling processing is performed on the target region of interest R D1 . That is, the image R D11 is obtained after the first downsampling, and R D11 is 1/2 of the size of R D1 ; the image R D12 is obtained after the second downsampling, and R D12 is 1/4 of the size of R D1 ; the third downsampling is Then the image R D13 is obtained, and R D13 is 1/8 of the size of R D1 .
  • the depth values of all pixels in the image R D13 are summed to obtain the depth feature d, and the area of the image R D13 is taken as the area feature a. That is, the key information of the current RGBD image frame can be expressed as [R D13 ,d,a].
  • Step 650 Determine the depth feature difference and the area feature difference according to the depth feature and the area feature of the target region of interest, and the preset depth feature and area feature of the template to be compared.
  • the preset template to be compared may refer to the depth image of the inventory container with a safe pose state, or may refer to the depth image of the image frame before the current frame.
  • the preset template to be compared may at least include a standard container template and the previous RGBD image frame.
  • the standard container template is calculated based on the mean value of the depth feature and the area feature in the preset number of RGBD image frames initially collected.
  • the depth feature and area feature of the preset template to be compared can be determined.
  • the depth feature and area feature of the target region of interest are respectively compared with the depth feature and area feature of the preset template to be compared to obtain the depth feature difference and the area feature difference.
  • the average value of the initial three frames before the acquisition is triggered as the preset template to be compared.
  • the average value of the key information of the first three RGBD image frames is used as the depth feature and image feature of the preset template to be compared, that is, Src[R D13 ,d,a].
  • the previous RGBD image frame of the current RGBD image frame is used as the preset template to be compared, and the key information composed of the depth feature and image feature of the previous RGBD image frame is determined according to the calculation process of the above-mentioned depth feature and image feature Expressed as Current[R D13 ,d,a].
  • Step 660 Detect the current pose deviation degree of the inventory container to be transported according to the depth feature difference value and the area feature difference value, and the preset depth feature threshold value and the preset area feature threshold value.
  • the depth feature threshold and the area feature threshold are determined in advance according to the degree of risk deviation in different scenarios. That is, the degree of pose deviation is determined based on the depth feature difference value and the area feature difference value. If the pose deviation limit defined by the depth feature threshold value and/or the area feature threshold value is exceeded, it is determined that the inventory container to be transported currently has a large deviation. There are potential safety hazards, which can easily lead to inability to safely or normally carry, or affect the normal handling of other forklifts or robots in the storage environment.
  • the depth feature difference between the current RGBD image frame and the previous RGBD image frame of the current RGBD image frame Value and area feature difference as well as the preset standard depth feature threshold, standard area feature threshold, adjacent feature threshold, and adjacent area feature threshold.
  • the depth feature difference between the current RGBD image frame and the standard container template is compared with the standard depth feature threshold; the area feature difference between the current RGBD image frame and the standard container template is compared with the standard area Compare feature thresholds; compare the depth feature difference between the current RGBD image frame and the previous RGBD image frame with the preset adjacent feature threshold; compare the current RGBD image frame with the previous RGBD image frame The area feature difference value of is compared with the adjacent area feature threshold. Therefore, in the above four comparisons, if there is at least one feature difference greater than the corresponding feature threshold, it is determined that the position of the inventory container to be transported on the smart forklift has shifted at the current moment.
  • the depth feature threshold between the current RGBD image frame and the initial three RGBD image frames is D CurrSrc
  • the area feature threshold S CurrSrc is pre-determined according to the degree of dangerous shift in different scenes.
  • Step 670 When it is determined that the position of the inventory container to be transported on the smart forklift has shifted, an alarm is issued to adjust the position of the inventory container to be transported on the smart forklift.
  • the smart forklift brakes and alarms when a large deviation is detected according to the uploaded detection data, so that the staff or other robots can handle the inventory container and/or the inventory container to be moved on the fork of the smart forklift.
  • the posture of the article is adjusted and the alarm is cancelled.
  • the smart forklift continues to perform the transportation task until the inventory container to be transported is transported to the destination and the inventory container to be transported is put down, and then the image collection and detection are stopped.
  • the technical solution of this embodiment triggers the RGBD image collection of the storage container to be transported on the fork when it is monitored that the smart forklift lifts the storage container.
  • the position of the inventory container to be transported in the RGB image is determined, and the target region of interest of the depth image of the current RGBD image frame is intercepted according to the position.
  • the depth feature and area feature of the target area of interest are determined, the feature comparison is performed according to the preset template to be compared, and the posture shift of the storage container to be transported on the fork is performed based on the feature comparison result and the feature threshold.
  • the embodiment of the application collects and recognizes the RGBD image containing the storage container to be transported on the fork, and realizes the effective detection of the posture deviation of the inventory container to be transported on the fork, avoiding the Due to external forces or emergency brakes of the forklift, the position of the inventory container to be transported on the fork changes and cannot be transported safely, thereby improving the efficiency and safety of the forklift for transporting the inventory container.
  • the embodiment of the present application also provides a computer-readable storage medium.
  • the computer-readable storage medium stores a computer program (or referred to as computer-executable instructions).
  • the program When the program is executed by a processor, it is used to execute a container pose deviation.
  • Moving detection method the method includes: in the case of monitoring that the smart forklift completes the lifting action of the inventory container to be transported, collecting RGBD image frames including the inventory container to be transported; wherein, the RGBD image frames include RGB images and Depth image; according to a predetermined container recognition model, from the RGB image of the current RGBD image frame, determine the position of the storage container to be transported in the RGB image; according to the storage container to be transported in the RGB image From the depth image of the current RGBD image frame, intercept the target region of interest containing the storage container to be transported; preset the depth characteristics of the template to be compared according to the depth feature and area feature of the target region of interest And the area feature, as well as the preset depth feature threshold and the preset area feature threshold, to
  • a computer-readable storage medium provided by an embodiment of the present application.
  • the computer-executable instructions stored on the computer-readable storage medium are not limited to the method operations described above, and can also execute the container pose provided in any embodiment of the present application. Related operations in the offset detection method.
  • the embodiments of the present application can be implemented by software and general-purpose hardware, and of course can also be implemented by hardware.
  • the technical solutions of the embodiments of the present application can be embodied in the form of a software product.
  • the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (Read-Only Memory, ROM). ), Random Access Memory (RAM), Flash memory (FLASH), hard disk or CD-ROM, etc., including multiple instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute this Apply the method described in any embodiment.
  • the multiple units and modules included are only divided according to the functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the names of multiple functional units are also It is just for the convenience of distinguishing each other, and is not used to limit the protection scope of this application.

Landscapes

  • Engineering & Computer Science (AREA)
  • Structural Engineering (AREA)
  • Physics & Mathematics (AREA)
  • Transportation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Geology (AREA)
  • Mechanical Engineering (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Civil Engineering (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Geometry (AREA)
  • Forklifts And Lifting Vehicles (AREA)
  • Image Analysis (AREA)

Abstract

一种智能叉车（400）以及容器位姿偏移检测方法，该智能叉车（400）包括：工作状态监测模块（410），设置为监测智能叉车（400）搬运待搬运库存容器（120）的工作状态，并依据工作状态，控制容器图像采集模块（420）进行图像采集；容器图像采集模块（420），设置为响应工作状态监测模块（410）的触发，采集包括待搬运库存容器（120）的RGBD图像帧，并将该RGBD图像帧传输至处理模块（430）；处理模块（430），设置为接收容器图像采集模块（420）反馈的RGBD图像帧，以及依据RGBD图像帧检测待搬运库存容器（120）的位姿偏移程度，并依据位姿偏移程度检测结果进行报警提示，以调整待搬运库存容器在该智能叉车（400）上的位姿。

Description

一种智能叉车以及容器位姿偏移检测方法
本申请要求在2019年04月25日提交中国专利局、申请号为201910337791.X的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及智能仓储技术领域，例如涉及一种智能叉车以及容器位姿偏移检测方法。
背景技术
随着现代仓库存储技术的快速发展，叉车在仓储领域中发挥着越来越重要的作用，叉车承载着托盘等库存容器实现物品的搬运和存储。
相关技术中，可以通过人工开动叉车移动至待搬运库存容器的存储位置处，利用叉车的货叉将待搬运库存容器举起以拾取待搬运库存容器，从而在搬运的过程中，工作人员通过时刻注意待搬运库存容器在货叉上的位置，实现安全稳定地将物品搬运和存储至指定存储位置。此外，在智能仓储环境中，还可以通过为智能叉车预设移动路线，以使智能叉车按照预设路线移动至待搬运库存容器的存储位置处，通过对库存容器底面标识的识别，来举起待搬运库存容器并进行搬运、存储待搬运库存容器等操作流程。
然而，相关技术在搬运库存容器的过程中，人工难以对货叉上举起的库存容器位姿进行准确有效且及时地观察，增加了操作难度以及人工干预成本，且智能叉车无法实现对货叉上的库存容器进行智能化位姿检测，降低了叉车搬运库存容器的效率和安全性。
发明内容
本申请实施例提供了一种智能叉车以及容器位姿偏移检测方法，能够对货叉上的库存容器进行智能化位姿检测，提高了叉车搬运库存容器的效率和安全性。
本申请实施例提供了一种智能叉车，所述智能叉车包括：工作状态监测模块、容器图像采集模块和处理模块；
所述容器图像采集模块与所述工作状态监测模块以及所述处理模块电连接；其中：
所述工作状态监测模块，设置为监测所述智能叉车搬运待搬运库存容器的 工作状态，并依据所述工作状态，控制所述容器图像采集模块进行图像采集；
所述容器图像采集模块，设置为响应所述工作状态监测模块的触发，采集包括所述待搬运库存容器的RGBD图像帧，并将该RGBD图像帧传输至所述处理模块；
所述处理模块，设置为接收所述容器图像采集模块反馈的RGBD图像帧，以及依据所述RGBD图像帧检测所述待搬运库存容器的位姿偏移程度；并依据位姿偏移程度检测结果进行报警提示，以调整所述待搬运库存容器在所述智能叉车上的位姿。
本申请实施例提供了一种容器位姿偏移检测方法，包括：
在监测智能叉车完成待搬运库存容器的举起动作的情况下，采集包括所述待搬运库存容器的RGBD图像帧；其中，所述RGBD图像帧包括RGB图像和深度图像；
依据预先确定的容器识别模型，从当前RGBD图像帧的RGB图像中，确定所述待搬运存储容器在所述RGB图像中的位置；
依据所述待搬运存储容器在所述RGB图像中的位置，从所述当前RGBD图像帧的深度图像中，截取包含所述待搬运存储容器的目标感兴趣区域；
依据所述目标感兴趣区域的深度特征和面积特征、预设待比对模板的深度特征和面积特征，以及预设深度特征阈值和预设面积特征阈值，检测所述待搬运库存容器当前的位姿偏移程度。
本申请实施例提供了一种计算机可读存储介质，计算机可读存储介质上存储有计算机程序，该程序被处理器执行时实现本申请任意实施例所述的容器位姿偏移检测方法。
附图说明
图1为本申请实施例提供的智能叉车搬运库存容器示例图；
图2为本申请实施例提供的库存容器的结构示意图；
图3为本申请实施例提供的一种智能叉车的结构示意图；
图4为本申请实施例提供的另一种智能叉车的结构框图；
图5为本申请实施例提供的一种容器位姿偏移检测方法的流程图；
图6为本申请实施例提供的另一种容器位姿偏移检测方法的流程图。
具体实施方式
下面结合附图和实施例对本申请实施例进行说明。可以理解的是，此处所描述的实施例仅仅用于解释本申请实施例，而非对本申请的限定。另外，为了便于描述，附图中仅示出了与本申请实施例相关的部分而非全部结构。
请参阅图1所示的智能叉车搬运库存容器示例图。如图1所示，智能叉车110通过货叉举起库存容器120，以使库存容器120位于智能叉车110的货叉上，对库存容器120进行搬运操作。本实施例中，库存容器120侧壁具有与智能叉车110的货叉对应的叉孔结构，例如托盘，智能叉车110将货叉插入至叉孔结构中，以承载库存容器120，并对库存容器120进行搬运。库存容器120上可以放置有物品130。在仓储环境中，库存容器120可以位于存储区域中，例如存储区域中货架隔层上，还可以位于工作站等分拣区域。从而智能叉车110依据需求，在仓储环境中对库存容器120进行搬运操作。
图2为本申请实施例提供的库存容器的结构示意图。如图2所示，库存容器120可以为规则或者不规则的长方体，库存容器120具有四个侧壁，库存容器120的至少两个相对的侧壁具有叉孔结构200。叉孔结构200中包括叉孔子结构210以及库存容器120本身的框架子结构220。库存容器130上可以直接放置物品130，物品130在库存容器120上能够以任何适当的方式放置在库存容器120上，例如存放有物品130的料箱放置在库存容器120上。在一实施例中，料箱可以与库存容器120分离，也可以与库存容器120为一体结构，料箱中可以放置一个或多个物品130。
图3为本申请实施例提供的智能叉车的结构示意图。结合图1和图3所示，在一个示例中，智能叉车110可以包括货叉111和滑轨112，通过该滑轨112，智能叉车110的货叉111能够在一定高度范围内升降。智能叉车110可以运动到货架的出口处等库存区的存储位置，利用货叉111插入库存容器120的叉孔结构200内而承载库存容器120，并将库存容器120搬运到指定地点，例如被分配的接驳位置或工作站等。在拣选工作站处工作人员(或机械臂等自动化设备)从库存容器120上拣选出物品。本申请实施例不限于将物品搬运至接驳位置或工作站等指定位置，任何搬运目的地都可以应用于本实施例中。
除此之外，在一个示例中，智能叉车110至少包括容器图像采集模块113，例如深度相机，图像采集模块113能够对货叉111、货叉111上的库存容器120以及库存容器120上的物品130进行图像采集，获得完整包含库存容器120和物品130的图像。相应的，智能叉车110还包括处理模块114，处理模块114设置为对容器图像采集模块113采集的图像进行处理，实现对库存容器120位姿偏移程度的检测。
智能叉车110在搬运库存容器120的过程中，当监测到货叉111完成库存 容器120的举起动作时，触发容器图像采集模块113采集货叉111上库存容器120的图像，处理模块114依据采集的图像，通过对库存容器120进行识别，确定库存容器120当前时刻的位姿偏移程度，从而依据位姿偏移程度检测结果，进行制动和报警提示，以在调整运库存容器130在货叉111上的位置之后，继续进行搬运操作。
在传统的仓储系统中，可以通过人工开动叉车移动至待搬运库存容器的存储位置处，利用叉车的货叉将待搬运库存容器举起以拾取待搬运库存容器，从而在搬运的过程中，工作人员通过时刻注意待搬运库存容器在货叉上的位置，实现安全稳定的将物品搬运和存储至指定存储位置。此外，在智能仓储环境中，还可以通过为智能叉车预设移动路线，以使智能叉车按照预设路线移动至待搬运库存容器的存储位置处，通过对库存容器底面标识的识别，来举起待搬运库存容器并进行搬运、存储待搬运库存容器等操作流程。然而，人工难以对货叉上举起的库存容器位姿进行准确有效且及时地观察，增加了操作难度以及人工干预成本，且智能叉车无法实现对货叉上的库存容器进行智能化位姿检测，降低了叉车搬运库存容器的效率和安全性。
下面针对本申请实施例中提供的智能叉车，通过多个实施例进行阐述。
实施例一
图4为本申请实施例提供的一种智能叉车的结构框图，本实施例可适用于智能叉车搬运库存容器的情况，该智能叉车400包括：工作状态监测模块410、容器图像采集模块420和处理模块430；容器图像采集模块420与工作状态监测模块410以及处理模块430电连接。
在本实施例中，工作状态监测模块410，设置为监测智能叉车搬运待搬运库存容器的工作状态，并依据工作状态，控制容器图像采集模块420进行图像采集；容器图像采集模块420，设置为响应工作状态监测模块的触发，采集包括待搬运库存容器的RGBD图像帧，并将该RGBD图像帧传输至处理模块430；处理模块430，设置为接收容器图像采集模块420反馈的RGBD图像帧，以及依据RGBD图像帧检测待搬运库存容器的位姿偏移程度；并依据位姿偏移程度检测结果进行报警提示，以调整待搬运库存容器在智能叉车上的位姿。
在本实施例中，工作状态监测模块410具有监测智能叉车多个机械结构工作状态的功能，可以以硬件的形式安装于智能叉车中，工作状态监测模块410的安装位置不被限定，或者可以以软件的形式集成于智能叉车软件处理程序中。在一实施例中，工作状态监测模块410可以根据智能叉车位置、货叉升降高度、 货叉承载重量以及智能叉车路径行进进度等数据，监测智能叉车搬运库存容器的工作状态，例如，库存容器已被举起、库存容器被搬运中以及库存容器已被放下等状态。从而依据监测到的工作状态，控制容器图像采集模块420的工作状态，例如触发容器图像采集模块420进行图像采集，或终止图像采集。
容器图像采集模块420具有图像采集功能，安装于智能叉车上，设置为实时地对智能叉车的货叉上的库存容器以及库存容器上的承载物品进行图像采集。容器图像采集模块420在被触发启动图像采集后，进行连续多帧的图像采集。智能叉车上安装的容器图像采集模块420可以为一个，也可以为多个。例如，在智能叉车的货叉所能够经过的每个高度档位上均安装一个容器图像采集模块420，或者在货叉上方即智能叉车的顶端安装一个容器图像采集模块420。
处理模块430可以具有图像识别功能、数据处理功能、容器位姿检测功能以及报警等功能。处理模块430安装于智能叉车中，处理模块430的安装位置不被限定。在一实施例中，处理模块430可以接收容器图像采集模块420实时反馈的图像帧，对图像帧进行库存容器识别，并依据识别结果确定库存容器特征，最终依据库存容器特征进行位姿偏移程度检测等。
本实施例中，容器图像采集模块420与工作状态监测模块410之间的连接方式，以及容器图像采集模块420与处理模块430之间的连接方式，可以为电连接，也可以为通信连接，通过电连接或通信连接进行数据或指令的传输等。
本实施例中，待搬运库存容器是指仓储环境中叉车所能搬运的库存容器，例如托盘等。相应的，该类待搬运库存容器具有与叉车的货叉相匹配的叉孔结构，用于货叉的插入，以举起待搬运库存容器，对待搬运库存容器进行搬运。该类待搬运库存容器可以位于仓储环境中的任意位置，例如位于存储区域。在一实施例中，待搬运库存容器可以位于存储区域中货架的隔层上。进而在智能叉车的货叉升降至待搬运库存容器所在高度的情况下，当货叉与待搬运库存容器的叉孔结构对齐时，将货叉插入至待搬运库存容器的叉孔结构中，从而智能叉车承载起待搬运库存容器，实现对待搬运库存容器的拾取和搬运等操作。
在一实施例中，容器图像采集模块420可以为Kinect等RGBD图像传感器，设置为采集RGBD(RGB+Depth Map)图像。RGBD图像中每个像素点包括R(Red，红色)、G(Green，绿色)、B(Blue，蓝色)彩色像素信息以及对应的深度信息。RGBD图像中每个像素点的RGB彩色信息构成RGB图像，RGBD图像中每个像素点的深度信息构成场景的二维像素矩阵，即深度图像。深度图像中每个像素点的值代表的是该像素点对应的物体与容器图像采集模块420所在平面的距离，每个像素点的位置与该像素点对应的物体在场景中的位置对应，与关联的RGB图像中的位置对应。
在一实施例中，在对待搬运库存容器进行搬运时，智能叉车按照既定路线行驶至待搬运库存容器所在存储位置，例如存储区域或存储具有叉孔结构的托盘的货架区等。智能叉车在控制下，将货叉升降至待搬运库存容器的存储位置所在高度。在一实施例中，工作状态监测模块410可以对货叉所在高度、承载重量或货叉作业动作等进行监测，若监测到货叉升降至与待搬运库存容器所在高度，且完成对待搬运库存容器的举起动作，则触发容器图像采集模块420进行RGBD图像帧连续采集。容器图像采集模块420响应工作状态监测模块的触发，采集包括待搬运库存容器的RGBD图像帧，并将该RGBD图像帧传输至处理模块430。
从而处理模块430接收容器图像采集模块420反馈的RGBD图像帧，可以基于预先确定的容器识别模型，对RGBD图像帧中待搬运库存容器进行识别，从当前RGBD图像帧的RGB图像中，确定待搬运存储容器在RGB图像中的位置。并依据待搬运存储容器在RGB图像中的位置，从当前RGBD图像帧的深度图像中，截取包含待搬运存储容器的目标感兴趣区域。进而利用具有深度信息的目标感兴趣区域，确定目标感兴趣区域的深度特征和面积特征，对待搬运库存容器当前的位姿偏移程度进行检测。本实施例中，位姿偏移程度的检测可以包括对待搬运库存容器与货叉之间相对偏移程度的检测，还可以包括待搬运库存容器上的物品与待搬运库存容器之间相对偏移程度的检测。
因此，本申请实施例的智能叉车在搬运待搬运库存容器的过程中，时刻对货叉上的待搬运库存容器及待搬运库存容器上放置的物品进行位置偏移程度检测，从而当确定货叉上的待搬运库存容器或物品存在较大位置偏移时，进行制动和报警提示，以提示工作人员或其他机器人对该智能叉车货叉上的待搬运库存容器和/或物品进行位姿的调整。在位姿调整后继续进行搬运，直至智能叉车将待搬运库存容器搬运至目的地，并放下待搬运库存容器为止，终止图像的采集和检测。从而避免由于货叉上的待搬运库存容器和/或物品的位姿发生变化，而无法安全或正常进行搬运的问题，保证了在待搬运库存容器的整个搬运过程中，待搬运库存容器和/或物品稳定地放置在货叉上。
本实施例的技术方案，通过安装于智能叉车的容器图像采集模块，在搬运库存容器的过程中，对货叉上的待搬运存储容器进行RGBD图像采集，从而依据时刻采集的RGBD图像帧，对货叉上的待搬运存储容器进行位姿偏移程度的检测，并依据位姿偏移程度检测结果进行报警提示，以调整待搬运库存容器在智能叉车上的位姿。本申请实施例通过对包含货叉上待搬运存储容器的RGBD图像进行采集和容器识别，实现了对货叉上的待搬运库存容器的位姿偏移进行有效检测，避免了搬运过程中由于待搬运库存容器受到外力或叉车紧急刹车等原因，导致待搬运库存容器在货叉上的位置发生变化而无法安全搬运的问题， 从而提高了叉车搬运库存容器的效率和安全性。
实施例二
本实施例在上述实施例一的基础上，提供了智能叉车的一个实施方式，能够通过对待搬运库存容器进行识别以及特征的确定，来检测待搬运库存容器的位姿偏移程度。
在一实施例中，容器图像采集模块420包括至少一个摄像头，摄像头安装于智能叉车的货叉上方。
在一实施例中，RGBD图像帧包括RGB图像和深度图像，RGBD图像帧中用于表征待搬运库存容器的图像位于RGBD图像帧中的指定区域内。
在一实施例中，工作状态监测模块410是设置为通过如下方式依据工作状态，控制容器图像采集模块420进行图像采集：在监测到智能叉车完成待搬运库存容器的举起动作的情况下，触发容器图像采集模块420进行图像采集；在监测到智能叉车完成待搬运库存容器的放置动作的情况下，终止容器图像采集模块420的图像采集动作。
在本实施例中，容器图像采集模块420可以为深度相机或Kinect传感器等能够采集RGBD图像的设备。在一实施例中，容器图像采集模块420包括至少一个摄像头，摄像头可以安装于货叉所能够经过的每个高度档位上，或者安装在货叉上方，即智能叉车的顶端。进而在货叉完成对待搬运库存容器的托举动作后，摄像头可以投射采集完整包含待搬运库存容器及待搬运库存容器上物品的RGBD图像帧。
本实施例中，在容器图像采集模块420的摄像头距离托盘的距离或相对高度有所变化的情况下，采集的图像中可能包括地面等无效的周围环境图像。为了提高待搬运库存容器位姿的检测效率，只对包含待搬运库存容器及待搬运库存容器上物品的区域进行检测即可。因此可以根据货叉的高度，以及该高度下货叉与摄像头之间固有的相对位置关系，确定图像中库存容器所在图像中的位置，从而在获得图像时，图像中可以表征待搬运库存容器的图像通常位于图像中的指定区域内。
本实施例中，工作状态监测模块410可以对货叉所在高度、承载重量或货叉作业动作等进行监测，若监测到货叉升降至待搬运库存容器所在高度，且完成对待搬运库存容器的举起动作，则触发容器图像采集模块420进行RGBD图像帧连续采集。若监测到货叉升降至与目标存储位置所在高度，且完成对待搬运库存容器的放下动作，则控制容器图像采集模块420停止图像采集动作。
在一实施例中，处理模块430是设置为通过如下方式依据RGBD图像帧检测待搬运库存容器的位姿偏移程度：依据预先确定的容器识别模型，从当前RGBD图像帧的RGB图像中，确定待搬运存储容器在RGB图像中的位置；依据待搬运存储容器在所述RGB图像中的位置，从当前RGBD图像帧的深度图像中，截取包含待搬运存储容器的目标感兴趣区域；确定目标感兴趣区域的深度特征和面积特征，并依据目标感兴趣区域的深度特征和面积特征，以及预设待比对模板的深度特征和面积特征，确定深度特征差值和面积特征差值；依据深度特征差值和面积特征差值，以及预设深度特征阈值和预设面积特征阈值，检测待搬运库存容器当前的位姿偏移程度。
在一实施例中，处理模块430是设置为通过如下方式确定目标感兴趣区域的深度特征和面积特征：依据预设层数对所述目标感兴趣区域进行降采样处理，得到降采样后的感兴趣区域图像；将降采样后的感兴趣区域图像中所有像素点的深度值求和，得到目标感兴趣区域的深度特征；确定降采样后的感兴趣区域图像的面积，得到目标感兴趣区域的面积特征。
在一实施例中，预设待比对模板至少包括标准容器模板和当前RGBD图像帧的上一帧RGBD图像帧。
在一实施例中，标准容器模板依据初始采集的预设数量的RGBD图像帧的深度特征均值和面积特征均值计算获得。
在一实施例中，处理模块430是设置为通过如下方式依据深度特征差值和面积特征差值，以及预设深度特征阈值和预设面积特征阈值，检测待搬运库存容器当前的位姿偏移程度：依据当前RGBD图像帧与标准容器模板之间的深度特征差值和面积特征差值，当前RGBD图像帧与当前RGBD图像帧的上一帧RGBD图像帧之间的深度特征差值和面积特征差值，以及预设的标准深度特征阈值、标准面积特征阈值、相邻特征阈值和相邻面积特征阈值，在检测到至少一项特征差值大于对应的特征阈值的情况下，确定当前时刻，待搬运库存容器在智能叉车上的位姿发生了偏移。
在本实施例中，容器识别模型是指用于识别货叉上待搬运库存容器的模型，可以通过预先的训练获得。在一实施例中，基于智能叉车上述结构，可以预设多种场景，采集在多种场景下智能叉车搬运库存容器过程中大量的样本RGB图像。基于场景的不同，样本RGB图像中可能包含完整、部分或不包含库存容器及库存容器上物品。从而对样本RGB图像进行分类，将包含库存容器的RGB图像作为正样本，将不包含库存容器的RGB图像作为负样本。并可以采用支持向量机(Support Vector Machine，SVM)进行容器识别模型的训练，得到容器识别模型。
本实施例中，采用训练好的SVM模型对当前RGBD图像帧的RGB图像进行识别，对识别出的待搬运库存容器添加最小外接矩形，并记录在RGB图像中该最小外接矩形的至少两个对角顶点的坐标，作为待搬运存储容器在RGB图像中的位置，例如左上角顶点坐标和右下角顶点坐标，通过该位置可以表示该最小外接矩形的位置和大小。
本实施例中，容器图像采集模块420的摄像头在智能叉车移动的过程中进行图像采集，在此期间，摄像头与智能叉车相对位置固定，在正常工况下，待搬运库存容器和物品的位置不变。在进行特征确定前，可以采用滤波器对深度图像进行去噪处理。例如可以采用高斯低通滤波器对深度图像进行平滑处理，将深度图像中因相机震动或激光接收干扰等引起的噪点进行去除，从而获取每个像素点准确的距离信息。在一实施例中，采用的高斯滤波器模板大小为3×3或5×5，标准差为1。本实施例中深度图像的去噪算法不局限于高斯低通滤波器，任何可以实现深度图像去噪的算法都可以应用于本实施例中。
本实施例中，在去除噪点的深度图像的基础上，由于当前RGBD图像帧的RGB图像与深度图像中的每个像素点具有关联对应的关系，表示了同一场景下的不同图像信息。因此可以依据待搬运存储容器在RGB图像中的位置，从当前RGBD图像帧的深度图像中，截取同样的矩形区域作为目标感兴趣区域。相应的，该目标感兴趣区域包含待搬运存储容器，且包含待搬运存储容器的深度信息。
本实施例中，将深度特征和面积特征作为目标感兴趣区域的关键信息，依据关键信息进行库存容器位姿偏移程度的检测。在一实施例中，深度特征是指深度图像中所有像素点的深度值之和，面积特征是指深度图像的总面积。由于原始图像的像素分辨率较高，为了提高检测效率，本实施例在获得目标感兴趣区域之后，可以先对目标感兴趣区域构建金字塔图像，以对目标感兴趣区域进行降采样处理。在一实施例中，金字塔图像的层数可以根据RGBD传感器的相机焦距来确定。从而对降采样处理后的感兴趣区域图像，求取感兴趣区域图像中所有像素点的深度值之和作为深度特征，计算感兴趣区域图像的总面积作为面积特征。
本实施例中，预设待比对模板可以是指具有安全位姿状态的库存容器的深度图像，也可以是指当前RGBD图像帧之前的图像帧的深度图像。在一实施例中，由于待搬运库存容器在初始举起状态时，待搬运库存容器的位姿较为安全和稳定，因此可以将初始采集的预设数量的RGBD图像帧的深度图像均值，作为预设待比对模板；此外，由于在进行当前RGBD图像帧检测时，当前RGBD图像帧之前的图像帧都是经过检测无位姿偏移或在安全偏移范围之内的，因此 还可以将当前RGBD图像帧的上一帧RGBD图像的深度图像作为预设待比对模板。进而依据上述深度特征和面积特征的确定方法，可以确定预设待比对模板的深度特征和面积特征。从而将目标感兴趣区域的深度特征和面积特征，分别与预设待比对模板的深度特征和面积特征进行比较，得到深度特征差值和面积特征差值。
本实施例依据不同场景下的危险偏移程度，预先确定深度特征阈值和面积特征阈值。即依据深度特征差值和面积特征差值确定位姿偏移程度，超过深度特征阈值和/或面积特征阈值限定的位姿偏移限度，则确定待搬运库存容器当前发生了较大偏移，存在安全隐患，易导致无法安全或正常搬运，或影响仓储环境中其他叉车或机器人的正常搬运。因此在检测到存在较大偏移时，处理模块430进行控制智能叉车制动，并进行报警提示，以使工作人员或其他机器人对该智能叉车货叉上的待搬运库存容器和/或物品进行位姿的调整。在位姿调整后继续进行搬运，直至智能叉车将待搬运库存容器搬运至目的地，并放下待搬运库存容器为止，此时停止图像采集和检测。
示例性的，通过对当前RGBD图像帧中的RGB图像进行容器识别，确定图像中待搬运库存容器的最小外接矩形，及该最小外接矩形坐标信息(X LeftTop,Y LeftTop)和(X RightBottem,Y RightBottem)，并将该坐标信息作为待搬运存储容器在RGB图像中的位置。通过对当前RGBD图像帧中的深度图像进行高斯低通滤波去噪，得到去噪后的深度图像D filter。根据该位置(X LeftTop,Y LeftTop)和(X RightBottem,Y RightBottem)，在D filter上截取对应的矩形区域作为目标感兴趣区域R D1。
假设金字塔图像层数为3，对R D1进行降采样处理。即第一次降采样后得到图像R D11，R D11为R D1尺寸的1/2；第二次降采样后得到图像R D12，R D12为R D1尺寸的1/4；第三次降采样后得到图像R D13，R D13为R D1尺寸的1/8。将图像R D13内所有像素点的深度值进行求和，得到深度特征d，并将图像R D13的面积作为面积特征a。即当前RGBD图像帧的关键信息可以为[R D13,d,a]。在一实施例中，将容器图像采集模块420触发采集的前三帧的均值作为预设待比对模板。依据上述深度特征和图像特征的计算流程，将前三帧的关键信息均值作为预设待比对模板的深度特征和图像特征，即Src[R D13,d,a]。同时将当前RGBD图像帧的上一帧RGBD图像帧作为预设待比对模板，依据上述深度特征和图像特征的计算流程，确定该上一帧RGBD图像帧的深度特征和图像特征构成的关键信息为Current[R D13,d,a]。
从而得到当前RGBD图像帧与初始三帧RGBD图像之间的深度特征差值|d-Src(d)|和面积特征差值|a-Src(a)|，以及当前RGBD图像帧与当前RGBD图像帧的上一帧RGBD图像帧之间的深度特征差值|d-Current(d)|和面积特征差值 |a-Current(a)|。假设依据不同场景下的危险偏移程度，预先确定当前RGBD图像帧与初始三帧RGBD图像帧之间的深度特征阈值为D CurrSrc，面积特征阈值S CurrSrc，预先确定当前RGBD图像帧与当前RGBD图像帧的上一帧RGBD图像帧之间的深度特征阈值为D CurrLast，面积特征阈值S CurrLast。本实施例中，相邻帧为连续变化的，所以正常工况下两相邻图像帧的面积与像素点个数相似。当|d-Src(d)|、|a-Src(a)|、|d-Current(d)|以及|a-Current(a)|这四项特征差值中，存在至少一项超过对应的特征阈值，则确定待搬运库存容器当前发生了较大偏移。智能叉车根据上传的检测数据进行制动和报警，待人工介入后，报警取消，完成调整后，智能叉车继续工作。
本实施例的技术方案，在监测到智能叉车举起库存容器时，触发容器图像采集模块对货叉上的待搬运存储容器进行RGBD图像采集。通过对当前RGBD图像帧的RGB图像进行库存容器识别，确定待搬运库存容器在RGB图像中的位置，依据该位置，截取当前RGBD图像帧的深度图像的目标感兴趣区域。从而对目标感兴趣区域进行深度特征和面积特征的确定，依据预设待比对模板与特征阈值进行特征的比较，并依据特征比较结果，对货叉上的待搬运存储容器进行位姿偏移程度的检测，并依据位姿偏移程度检测结果进行报警提示，以调整待搬运库存容器在智能叉车上的位姿。本申请实施例通过对包含货叉上待搬运存储容器的RGBD图像进行采集和容器识别，实现了对货叉上的待搬运库存容器的位姿偏移进行有效检测，避免了搬运过程中由于待搬运库存容器受到外力或叉车紧急刹车等原因，导致待搬运库存容器在货叉上的位置发生变化而无法安全搬运的问题，从而提高了叉车搬运库存容器的效率和安全性。
实施例三
图5为本申请实施例提供的一种容器位姿偏移检测方法的流程图，本实施例可适用于智能叉车搬运库存容器的情况，该方法可由一种容器位姿偏移检测装置来执行，该装置可以采用软件和/或硬件的方式实现。在一实施例中，该装置配置于智能叉车中。该方法包括如下步骤：
步骤510、在监测到智能叉车完成待搬运库存容器的举起动作的情况下，采集包括待搬运库存容器的RGBD图像帧。
在本实施例中，待搬运库存容器是指仓储环境中叉车所能搬运的库存容器，例如托盘等。相应的，该类待搬运库存容器具有与叉车的货叉相匹配的叉孔结构，用于货叉的插入，以举起待搬运库存容器，对待搬运库存容器进行搬运。该类待搬运库存容器可以位于仓储环境中的任意位置，例如位于存储区域。在一实施例中，待搬运库存容器可以位于存储区域中货架的隔层上。进而在智能 叉车的货叉升降至待搬运库存容器所在高度的情况下，当货叉与待搬运库存容器的叉孔结构对齐时，将货叉插入至待搬运库存容器的叉孔结构中，从而智能叉车承载起待搬运库存容器，实现对待搬运库存容器的拾取和搬运等操作。
在一实施例中，可以采用Kinect等RGBD图像传感器进行图像采集，所采集的图像为RGBD图像(RGB+Depth Map)，RGBD图像中每个像素点包括R(Red，红色)、G(Green，绿色)、B(Blue，蓝色)彩色像素信息以及对应的深度信息。RGBD图像中每个像素点的RGB彩色信息构成RGB图像，RGBD图像中每个像素点的深度信息构成场景的二维像素矩阵，即深度图像。深度图像中每个像素值代表的是该像素点对应的物体与RGBD图像传感器所在平面的距离，每个像素点的位置与该像素点对应的物体在场景中的位置对应，与关联的RGB图像中的位置对应。
在一实施例中，在对待搬运库存容器进行搬运时，智能叉车按照既定路线行驶至待搬运库存容器所在存储位置，例如存储区域或存储具有叉孔结构的托盘的货架区等。智能叉车在控制下，将货叉升降至待搬运库存容器的存储位置所在高度。在一实施例中，可以对货叉所在高度、承载重量或货叉作业动作等进行监测，若监测到货叉升降至与待搬运库存容器所在高度，且完成对待搬运库存容器的举起动作，则触发进行RGBD图像帧连续采集。
步骤520、依据预先确定的容器识别模型，从当前RGBD图像帧的RGB图像中，确定待搬运存储容器在RGB图像中的位置。
在本实施例中，可以基于预先确定的容器识别模型，对RGBD图像帧中待搬运库存容器进行识别，从当前RGBD图像帧的RGB图像中，确定待搬运存储容器在RGB图像中的位置。容器识别模型是指用于识别货叉上待搬运库存容器的模型，可以通过预先的训练获得。在一实施例中，基于智能叉车上述结构，可以预设多种场景，采集在多种场景下智能叉车搬运库存容器过程中大量的样本RGB图像。基于场景的不同，样本RGB图像中可能包含完整、部分或不包含库存容器及库存容器上物品。从而对样本RGB图像进行分类，将包含库存容器的RGB图像作为正样本，将不包含库存容器的RGB图像作为负样本。并可以采用SVM进行容器识别模型的训练，得到容器识别模型。
本实施例中，采用训练好的SVM模型对当前RGBD图像帧的RGB图像进行识别，对识别出的待搬运库存容器添加最小外接矩形，并记录在RGB图像中该最小外接矩形的至少两个对角顶点的坐标，作为待搬运存储容器在RGB图像中的位置，例如左上角顶点坐标和右下角顶点坐标，通过该位置可以表示该最小外接矩形的位置和大小。
步骤530、依据待搬运存储容器在RGB图像中的位置，从当前RGBD图像 帧的深度图像中，截取包含待搬运存储容器的目标感兴趣区域。
在本实施例中，由于当前RGBD图像帧的RGB图像与深度图像中的每个像素点具有关联对应的关系，表示了同一场景下的不同图像信息。因此可以依据待搬运存储容器在RGB图像中的位置，从当前RGBD图像帧的深度图像中，截取同样的矩形区域作为目标感兴趣区域。相应的，该目标感兴趣区域包含待搬运存储容器，且包含待搬运存储容器的深度信息。
步骤540、依据目标感兴趣区域的深度特征和面积特征、预设待比对模板的深度特征和面积特征，以及预设深度特征阈值和预设面积特征阈值，检测待搬运库存容器当前的位姿偏移程度。
在本实施例中，将深度特征和面积特征作为目标感兴趣区域的关键信息，依据关键信息进行库存容器位姿偏移程度的检测。在一实施例中，深度特征是指深度图像中所有像素点的深度值之和，面积特征是指深度图像的总面积。
本实施例中，预设待比对模板可以是指具有安全位姿状态的库存容器的深度图像，也可以是指当前RGBD图像帧之前的图像帧的深度图像。在一实施例中，由于待搬运库存容器在初始举起状态时，待搬运库存容器位姿较为安全和稳定，因此可以将初始采集的预设数量的RGBD图像帧的深度图像均值，作为预设待比对模板；此外，由于在进行当前RGBD图像帧检测时，当前RGBD图像帧之前的图像帧都是经过检测无位姿偏移或在安全偏移范围之内的，因此还可以将当前RGBD图像帧的上一帧RGBD图像帧的深度图像作为预设待比对模板。进而依据上述深度特征和面积特征的确定方法，可以确定预设待比对模板的深度特征和面积特征。从而将目标感兴趣区域的深度特征和面积特征，分别与预设待比对模板的深度特征和面积特征进行比较，得到深度特征差值和面积特征差值。
本实施例依据不同场景下的危险偏移程度，预先确定深度特征阈值和面积特征阈值。即依据深度特征差值和面积特征差值确定位姿偏移程度，超过深度特征阈值和/或面积特征阈值限定的位姿偏移限度，则确定待搬运库存容器当前发生了较大偏移，存在安全隐患，易导致无法安全或正常搬运，或影响仓储环境中其他叉车或机器人的正常搬运。
本实施例的技术方案，在监测到智能叉车完成待搬运库存容器的举起动作时，触发图像采集功能，以使在搬运库存容器的过程中，对货叉上的待搬运存储容器进行连续的RGBD图像采集，从而依据时刻采集的RGBD图像帧，对货叉上的待搬运存储容器进行位姿偏移程度的检测，并依据位姿偏移程度检测结果进行报警提示，以调整待搬运库存容器在智能叉车上的位姿。本申请实施例通过对包含货叉上待搬运存储容器的RGBD图像进行采集和容器识别，实现了 对货叉上的待搬运库存容器的位姿偏移进行有效检测，避免了搬运过程中由于待搬运库存容器受到外力或叉车紧急刹车等原因，导致待搬运库存容器在货叉上的位置发生变化而无法安全搬运的问题，从而提高了叉车搬运库存容器的效率和安全性。
实施例四
本实施例在上述实施例三的基础上，提供了容器位姿偏移检测方法的一个实施方式，能够通过对待搬运库存容器进行识别以及特征的确定，来检测待搬运库存容器的位姿偏移程度。图6为本申请实施例提供的另一种容器位姿偏移检测方法的流程图，如图6所示，该方法包括以下步骤：
步骤610、在监测到智能叉车完成待搬运库存容器的举起动作的情况下，采集包括待搬运库存容器的RGBD图像帧。
步骤620、依据预先确定的容器识别模型，从当前RGBD图像帧的RGB图像中，确定待搬运存储容器在RGB图像中的位置。
在本实施例中，采用训练好的SVM模型对当前RGBD图像帧的RGB图像进行识别，对识别出的待搬运库存容器添加最小外接矩形，并记录在RGB图像中该最小外接矩形的至少两个对角顶点的坐标，作为待搬运存储容器在RGB图像中的位置，通过该位置可以表示该最小外接矩形的位置和大小。
示例性的，通过对当前RGBD图像帧中的RGB图像进行容器识别，确定图像中待搬运库存容器的最小外接矩形，及该最小外接矩形坐标信息(X LeftTop,Y LeftTop)和(X RightBottem,Y RightBottem)，并将该坐标信息作为待搬运存储容器在RGB图像中的位置。
步骤630、依据待搬运存储容器在RGB图像中的位置，从当前RGBD图像帧的深度图像中，截取包含待搬运存储容器的目标感兴趣区域。
在本实施例中，深度相机等图像传感器在智能叉车移动的过程中进行图像采集，在此期间，图像传感器与智能叉车位置相对位置固定，在正常工况下，待搬运库存容器和物品的位置不变。在进行特征确定前，可以采用滤波器对深度图像进行去噪处理。例如可以采用高斯低通滤波器对深度图像进行平滑处理，将深度图像中因图像传感器震动或激光接收干扰等引起的噪点进行去除，从而获取每个像素点准确的距离信息。在一实施例中，采用的高斯滤波器模板大小为3×3或5×5，标准差为1。本实施例中深度图像的去噪算法不局限于高斯低通滤波器，任何可以实现深度图像去噪的算法都可以应用于本实施例中。
本实施例中，在去除噪点的深度图像的基础上，由于当前RGBD图像帧的 RGB图像与深度图像中的每个像素点具有关联对应的关系，表示了同一场景下的不同图像信息。因此可以依据待搬运存储容器在RGB图像中的位置，从当前RGBD图像帧的深度图像中，截取同样的矩形区域作为目标感兴趣区域。相应的，该目标感兴趣区域包含待搬运存储容器，且包含待搬运存储容器的深度信息。
示例性的，在上述示例中，通过对当前RGBD图像帧中的深度图像进行高斯低通滤波去噪，得到去噪后的深度图像D filter。根据该位置(X LeftTop,Y LeftTop)和(X RightBottem,Y RightBottem)，在D filter上截取对应的矩形区域作为目标感兴趣区域。
步骤640、确定目标感兴趣区域的深度特征和面积特征。
在本实施例中，由于原始图像的像素分辨率较高，为了提高检测效率，本实施例在获得目标感兴趣区域之后，可以先对目标感兴趣区域构建金字塔图像，以对目标感兴趣区域进行降采样处理。在一实施例中，金字塔图像的层数可以根据RGBD传感器的相机焦距来确定。从而对降采样处理后的感兴趣区域图像，求取感兴趣区域图像中所有像素点的深度值之和作为深度特征，计算感兴趣区域图像中总面积作为面积特征。
在一实施例中，依据预设层数对所述目标感兴趣区域进行降采样处理，得到降采样后的感兴趣区域图像；将降采样后的感兴趣区域图像各像素点的深度值求和，得到所述目标感兴趣区域的深度特征；确定降采样后的感兴趣区域图像的面积，得到所述目标感兴趣区域的面积特征。
示例性的，在上述示例中，假设金字塔图像层数为3，对目标感兴趣区域R D1进行降采样处理。即第一次降采样后得到图像R D11，R D11为R D1尺寸的1/2；第二次降采样后得到图像R D12，R D12为R D1尺寸的1/4；第三次降采样后得到图像R D13，R D13为R D1尺寸的1/8。将图像R D13内所有像素点的深度值进行求和，得到深度特征d，并将图像R D13的面积作为面积特征a。即当前RGBD图像帧的关键信息可以表示为[R D13,d,a]。
步骤650、依据目标感兴趣区域的深度特征和面积特征，以及预设待比对模板的深度特征和面积特征，确定深度特征差值和面积特征差值。
在本实施例中，预设待比对模板可以是指具有安全位姿状态的库存容器的深度图像，也可以是指当前帧之前的图像帧的深度图像。在一实施例中，由于 待搬运库存容器在初始举起状态时，待搬运库存容器的位姿较为安全和稳定；或者由于在进行当前帧检测时，当前帧之前的图像帧都是经过检测无位姿偏移或在安全偏移范围之内的。因此，预设待比对模板至少可以包括标准容器模板和上一帧RGBD图像帧。在一实施例中，标准容器模板依据初始采集的预设数量的RGBD图像帧中，深度特征和面积特征的均值计算获得。进而依据上述深度特征和面积特征的确定方法，可以确定预设待比对模板的深度特征和面积特征。从而将目标感兴趣区域的深度特征和面积特征，分别与预设待比对模板的深度特征和面积特征进行比较，得到深度特征差值和面积特征差值。
示例性的，在一实施例中，将触发采集的初始前三帧的均值作为预设待比对模板。依据上述深度特征和图像特征的计算流程，将前三帧RGBD图像帧的关键信息均值作为预设待比对模板的深度特征和图像特征，即Src[R D13,d,a]。同时将当前RGBD图像帧的上一帧RGBD图像帧作为预设待比对模板，依据上述深度特征和图像特征的计算流程，确定该上一帧RGBD图像帧的深度特征和图像特征构成的关键信息表示为Current[R D13,d,a]。从而得到当前RGBD图像帧与初始三帧RGBD图像帧之间的深度特征差值|d-Src(d)|和面积特征差值|a-Src(a)|，以及当前RGBD图像帧与上一帧RGBD图像帧之间的深度特征差值|d-Current(d)|和面积特征差值|a-Current(a)|。
步骤660、依据深度特征差值和面积特征差值，以及预设深度特征阈值和预设面积特征阈值，检测待搬运库存容器当前的位姿偏移程度。
在本实施例中，依据不同场景下的危险偏移程度，预先确定深度特征阈值和面积特征阈值。即依据深度特征差值和面积特征差值确定位姿偏移程度，超过深度特征阈值和/或面积特征阈值限定的位姿偏移限度，则确定待搬运库存容器当前发生了较大偏移，存在安全隐患，易导致无法安全或正常搬运，或影响仓储环境中其他叉车或机器人的正常搬运。
在一实施例中，依据当前RGBD图像帧与标准容器模板之间的深度特征差值和面积特征差值，当前RGBD图像帧与当前RGBD图像帧的上一帧RGBD图像帧之间的深度特征差值和面积特征差值，以及预设的标准深度特征阈值、标准面积特征阈值、相邻特征阈值和相邻面积特征阈值，在检测到至少一项特征 差值大于对应的特征阈值的情况下，确定当前时刻，待搬运库存容器在所述智能叉车上的位姿发生了偏移。
本实施例中，将当前RGBD图像帧与标准容器模板之间的深度特征差值，与标准深度特征阈值进行比较；将当前RGBD图像帧与标准容器模板之间的面积特征差值，与标准面积特征阈值进行比较；将当前RGBD图像帧与上一帧RGBD图像帧之间的深度特征差值，与预设的相邻特征阈值进行比较；将当前RGBD图像帧与上一帧RGBD图像帧之间的面积特征差值，与相邻面积特征阈值进行比较。从而在上述四项比较中，若存在至少一项特征差值大于对应的特征阈值，则确定当前时刻，待搬运库存容器在所述智能叉车上的位姿发生了偏移。
示例性的，在上述示例中，假设依据不同场景下的危险偏移程度，预先确定当前RGBD图像帧与初始三帧RGBD图像帧之间的深度特征阈值为D CurrSrc，面积特征阈值S CurrSrc，预先确定当前RGBD图像帧与当前RGBD图像帧的上一帧RGBD图像帧之间的深度特征阈值为D CurrLast，面积特征阈值S CurrLast。因此，当|d-Src(d)|＞D CurrSrc、|a-Src(a)|＞S CurrSrc、|d-Current(d)|＞D CurrLast以及
Figure PCTCN2019128048-appb-000001
中存在至少一项满足时，则确定待搬运库存容器当前发生了较大偏移。
步骤670、在确定待搬运库存容器在智能叉车上的位姿发生了偏移的情况下，进行报警提示，以调整待搬运库存容器在智能叉车上的位姿。
在本实施例中，智能叉车根据上传的检测数据，在检测存在较大偏移时进行制动和报警，以使工作人员或其他机器人对该智能叉车货叉上的待搬运库存容器和/或物品进行位姿的调整，并取消报警。完成调整后，智能叉车继续执行搬运任务，直至将待搬运库存容器搬运至目的地，并放下待搬运库存容器为止，进而停止图像采集和检测。
本实施例的技术方案，在监测智能叉车举起库存容器时，触发对货叉上的待搬运存储容器进行RGBD图像采集。通过对当前RGBD图像帧的RGB图像进行库存容器识别，确定待搬运库存容器在RGB图像中的位置，依据该位置，截取当前RGBD图像帧的深度图像的目标感兴趣区域。从而对目标感兴趣区域进行深度特征和面积特征的确定，依据预设待比对模板进行特征的比较，并依据特征比较结果与特征阈值，对货叉上的待搬运存储容器进行位姿偏移程度的检测，并依据位姿偏移程度检测结果进行报警提示，以调整待搬运库存容器在智能叉车上的位姿。本申请实施例通过对包含货叉上待搬运存储容器的RGBD 图像进行采集和容器识别，实现了对货叉上的待搬运库存容器的位姿偏移进行有效检测，避免了搬运过程中由于待搬运库存容器受到外力或叉车紧急刹车等原因，导致待搬运库存容器在货叉上的位置发生变化而无法安全搬运的问题，从而提高了叉车搬运库存容器的效率和安全性。
实施例五
本申请实施例还提供一种计算机可读存储介质，计算机可读存储介质上存储有计算机程序(或称为计算机可执行指令)，该程序被处理器执行时用于执行一种容器位姿偏移检测方法，该方法包括：在监测到智能叉车完成待搬运库存容器的举起动作的情况下，采集包括所述待搬运库存容器的RGBD图像帧；其中，所述RGBD图像帧包括RGB图像和深度图像；依据预先确定的容器识别模型，从当前RGBD图像帧的RGB图像中，确定所述待搬运存储容器在所述RGB图像中的位置；依据所述待搬运存储容器在所述RGB图像中的位置，从当前RGBD图像帧的深度图像中，截取包含所述待搬运存储容器的目标感兴趣区域；依据所述目标感兴趣区域的深度特征和面积特征、预设待比对模板的深度特征和面积特征，以及预设深度特征阈值和预设面积特征阈值，检测所述待搬运库存容器当前的位姿偏移程度。
本申请实施例所提供的一种计算机可读存储介质，计算机可读存储介质上存储的计算机可执行指令不限于如上所述的方法操作，还可以执行本申请任意实施例所提供的容器位姿偏移检测方法中的相关操作。
通过以上关于实施方式的描述，所属领域的技术人员可以了解到，本申请实施例可借助软件及通用硬件来实现，当然也可以通过硬件实现。基于这样的理解，本申请实施例的技术方案可以以软件产品的形式体现出来，该计算机软件产品可以存储在计算机可读存储介质中，如计算机的软盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、闪存(FLASH)、硬盘或光盘等，包括多个指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请任意实施例所述的方法。
上述装置的实施例中，所包括的多个单元和模块只是按照功能逻辑进行划 分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，多个功能单元的名称也只是为了便于相互区分，并不用于限制本申请的保护范围。

Claims (17)

  1. 一种智能叉车，包括：工作状态监测模块、容器图像采集模块和处理模块；
    所述容器图像采集模块与所述工作状态监测模块以及所述处理模块电连接；
    所述工作状态监测模块，设置为监测所述智能叉车搬运待搬运库存容器的工作状态，并依据所述工作状态，控制所述容器图像采集模块进行图像采集；
    所述容器图像采集模块，设置为响应所述工作状态监测模块的触发，采集包括所述待搬运库存容器的RGBD图像帧，并将所述RGBD图像帧传输至所述处理模块；
    所述处理模块，设置为接收所述RGBD图像帧，以及依据所述RGBD图像帧检测所述待搬运库存容器的位姿偏移程度；并依据位姿偏移程度检测结果进行报警提示，以调整所述待搬运库存容器在所述智能叉车上的位姿。
  2. 根据权利要求1所述的智能叉车，其中，所述容器图像采集模块包括至少一个摄像头，所述摄像头安装于所述智能叉车的货叉上方。
  3. 根据权利要求1所述的智能叉车，其中，所述RGBD图像帧包括红绿蓝RGB图像和深度图像，所述RGBD图像帧中用于表征所述待搬运库存容器的图像位于所述RGBD图像帧中的指定区域内。
  4. 根据权利要求1所述的智能叉车，其中，所述工作状态监测模块是设置为通过如下方式依据所述工作状态，控制所述容器图像采集模块进行图像采集：
    在监测到所述智能叉车完成所述待搬运库存容器的举起动作的情况下，触发所述容器图像采集模块进行图像采集；
    在监测到所述智能叉车完成所述待搬运库存容器的放置动作的情况下，终止所述容器图像采集模块的图像采集动作。
  5. 根据权利要求3所述的智能叉车，其中，所述处理模块是设置为通过如下方式依据所述RGBD图像帧检测所述待搬运库存容器的位姿偏移程度：
    依据预先确定的容器识别模型，从当前RGBD图像帧的RGB图像中，确定所述待搬运存储容器在所述RGB图像中的位置；
    依据所述待搬运存储容器在所述RGB图像中的位置，从所述当前RGBD图像帧的深度图像中，截取包含所述待搬运存储容器的目标感兴趣区域；
    Figure PCTCN2019128048-appb-100001
    确定所述目标感兴趣区域的深度特征和面积特征，并依据所述目标感兴趣区域的深度特征和面积特征，以及预设待比对模板的深度特征和面积特征，确定深度特征差值和面积特征差值；
    依据所述深度特征差值和所述面积特征差值，以及预设深度特征阈值和预设面积特征阈值，检测所述待搬运库存容器当前的位姿偏移程度。
  6. 根据权利要求5所述的智能叉车，其中，所述处理模块是设置为通过如下方式确定所述目标感兴趣区域的深度特征和面积特征：
    依据预设层数对所述目标感兴趣区域进行降采样处理，得到降采样后的感兴趣区域图像；
    将所述降采样后的感兴趣区域图像中所有像素点的深度值求和，得到所述目标感兴趣区域的深度特征；
    确定所述降采样后的感兴趣区域图像的面积，得到所述目标感兴趣区域的面积特征。
  7. 根据权利要求5所述的智能叉车，其中，所述预设待比对模板至少包括标准容器模板和所述当前RGBD图像帧的上一帧RGBD图像帧。
  8. 根据权利要求7所述的智能叉车，其中，所述标准容器模板依据初始采集的预设数量的RGBD图像帧的深度特征均值和面积特征均值计算获得。
  9. 根据权利要求7所述的智能叉车，其中，所述处理模块是设置为通过如下方式依据所述深度特征差值和所述面积特征差值，以及预设深度特征阈值和预设面积特征阈值，检测所述待搬运库存容器当前的位姿偏移程度：
    在检测到所述当前RGBD图像帧与所述标准容器模板之间的深度特征差值大于预设的标准深度特征阈值、所述当前RGBD图像帧与所述标准容器模板之间的面积特征差值大于预设的标准面积特征阈值、所述当前RGBD图像帧与所述当前RGBD图像帧的上一帧RGBD图像帧之间的深度特征差值大于预设的相邻特征阈值以及所述当前RGBD图像帧与所述当前RGBD图像帧的上一帧RGBD图像帧之间的面积特征差值大于预设的相邻面积特征阈值中的至少一项的情况下，确定当前时刻，所述待搬运库存容器在所述智能叉车上的位姿发生了偏移。
  10. 一种容器位姿偏移检测方法，包括：
    在监测到智能叉车完成待搬运库存容器的举起动作的情况下，采集包括所述待搬运库存容器的RGBD图像帧；其中，所述RGBD图像帧包括RGB图像和深度图像；
    依据预先确定的容器识别模型，从当前RGBD图像帧的RGB图像中，确定所述待搬运存储容器在所述RGB图像中的位置；
    依据所述待搬运存储容器在所述RGB图像中的位置，从所述当前RGBD图 像帧的深度图像中，截取包含所述待搬运存储容器的目标感兴趣区域；
    依据所述目标感兴趣区域的深度特征和面积特征、预设待比对模板的深度特征和面积特征，以及预设深度特征阈值和面积特征阈值，检测所述待搬运库存容器当前的位姿偏移程度。
  11. 根据权利要求10所述的方法，其中，所述依据所述目标感兴趣区域的深度特征和面积特征、预设待比对模板的深度特征和面积特征，以及预设深度特征阈值和面积特征阈值，检测所述待搬运库存容器当前的位姿偏移程度，包括：
    确定所述目标感兴趣区域的深度特征和面积特征；
    依据所述目标感兴趣区域的深度特征和面积特征，以及预设待比对模板的深度特征和面积特征，确定深度特征差值和面积特征差值；
    依据所述深度特征差值和面积特征差值，以及预设深度特征阈值和预设面积特征阈值，检测所述待搬运库存容器当前的位姿偏移程度。
  12. 根据权利要求11所述的方法，其中，所述确定所述目标感兴趣区域的深度特征和面积特征，包括：
    依据预设层数对所述目标感兴趣区域进行降采样处理，得到降采样后的感兴趣区域图像；
    将所述降采样后的感兴趣区域图像中所有像素点的深度值求和，得到所述目标感兴趣区域的深度特征；
    确定所述降采样后的感兴趣区域图像的面积，得到所述目标感兴趣区域的面积特征。
  13. 根据权利要求11所述的方法，其中，所述预设待比对模板至少包括标准容器模板和所述当前RGBD图像帧的上一帧RGBD图像帧。
  14. 根据权利要求13所述的方法，其中，所述标准容器模板依据初始采集的预设数量的RGBD图像帧的深度特征均值和面积特征均值计算获得。
  15. 根据权利要求13所述的方法，其中，所述依据所述深度特征差值和面积特征差值，以及预设深度特征阈值和预设面积特征阈值，检测所述待搬运库存容器当前的位姿偏移程度，包括：
    在检测到所述当前RGBD图像帧与所述标准容器模板之间的深度特征差值大于预设的标准深度特征阈值、所述当前RGBD图像帧与所述标准容器模板之间的面积特征差值大于预设的标准面积特征阈值、所述当前RGBD图像帧与所述当前RGBD图像帧的上一帧RGBD图像帧之间的深度特征差值大于预设的相 邻特征阈值以及所述当前RGBD图像帧与所述当前RGBD图像帧的上一帧RGBD图像帧之间的面积特征差值大于预设的相邻面积特征阈值中的至少一项的情况下，确定当前时刻，所述待搬运库存容器在所述智能叉车上的位姿发生了偏移。
  16. 根据权利要求10所述的方法，在所述检测所述待搬运库存容器当前的位姿偏移程度之后，还包括：
    在确定所述待搬运库存容器在所述智能叉车上的位姿发生了偏移的情况下，进行报警提示，以调整所述待搬运库存容器在所述智能叉车上的位姿。
  17. 一种计算机可读存储介质，存储有计算机程序，所述计算机程序被处理器执行时实现如权利要求10-16中任一项所述的容器位姿偏移检测方法。
PCT/CN2019/128048 2019-04-25 2019-12-24 一种智能叉车以及容器位姿偏移检测方法 WO2020215772A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
KR1020217035759A KR102461759B1 (ko) 2019-04-25 2019-12-24 지능형 지게차 및 컨테이너 위치 및 자세 편이 검출 방법
JP2021563272A JP7206421B2 (ja) 2019-04-25 2019-12-24 スマートフォークリフト及び容器位置姿勢ずれの検出方法
EP19926162.9A EP3960692A4 (en) 2019-04-25 2019-12-24 INTELLIGENT FORKLIFT TRUCK AND CONTAINER POSITION AND POSTURE DEVIATION DETECTION METHOD
US17/508,784 US11625854B2 (en) 2019-04-25 2021-10-22 Intelligent forklift and method for detecting pose deviation of container

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910337791.XA CN110054121B (zh) 2019-04-25 2019-04-25 一种智能叉车以及容器位姿偏移检测方法
CN201910337791.X 2019-04-25

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/508,784 Continuation US11625854B2 (en) 2019-04-25 2021-10-22 Intelligent forklift and method for detecting pose deviation of container

Publications (1)

Publication Number Publication Date
WO2020215772A1 true WO2020215772A1 (zh) 2020-10-29

Family

ID=67320596

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/128048 WO2020215772A1 (zh) 2019-04-25 2019-12-24 一种智能叉车以及容器位姿偏移检测方法

Country Status (6)

Country Link
US (1) US11625854B2 (zh)
EP (1) EP3960692A4 (zh)
JP (1) JP7206421B2 (zh)
KR (1) KR102461759B1 (zh)
CN (1) CN110054121B (zh)
WO (1) WO2020215772A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022207215A1 (de) * 2021-03-31 2022-10-06 Still Gmbh Verfahren und system zur lasterkennung bei einem flurförderzeug
US11524846B2 (en) * 2020-10-19 2022-12-13 Gideon Brothers d.o.o. Pose determination by autonomous robots in a facility context

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110054121B (zh) 2019-04-25 2021-04-20 北京极智嘉科技有限公司 一种智能叉车以及容器位姿偏移检测方法
CN110781794A (zh) * 2019-10-21 2020-02-11 兰剑智能科技股份有限公司 一种智能识别的方法及系统
CN110950277A (zh) * 2019-12-16 2020-04-03 浙江迈睿机器人有限公司 用于agv叉车的托盘姿态识别系统及其方法
CN111618848B (zh) * 2020-04-28 2022-08-12 平安科技（深圳）有限公司 多机器人的协作控制方法、装置及计算机设备
CN111874512B (zh) * 2020-06-10 2022-02-22 北京旷视机器人技术有限公司 位置调整方法、装置、托举式机器人及计算机存储介质
US20220016779A1 (en) * 2020-07-15 2022-01-20 The Board Of Trustees Of The University Of Illinois Autonomous Robot Packaging of Arbitrary Objects
CN112614183A (zh) * 2020-12-25 2021-04-06 深圳市镭神智能系统有限公司 托盘位姿检测方法、装置、设备和存储介质
CN112507984B (zh) * 2021-02-03 2021-05-11 苏州澳昆智能机器人技术有限公司 一种基于图像识别的输送物料异常识别方法、装置及系统
KR20240084401A (ko) * 2022-12-06 2024-06-13 숭실대학교산학협력단 연속적인 rgb-d 프레임 세트를 활용한 고품질 깊이 영상 생성 방법 및 이의 장치
EP4389681A1 (en) * 2022-12-22 2024-06-26 TOYOTA MATERIAL HANDLING MANUFACTURING ITALY S.p.A Load anomaly recognition
CN117437187B (zh) * 2023-10-13 2024-06-25 深圳市锐巽自动化设备有限公司 一种雷管检测方法、系统及存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106672859A (zh) * 2017-01-05 2017-05-17 深圳市有光图像科技有限公司 一种基于叉车视觉识别托盘的方法及叉车
CN107031629A (zh) * 2017-03-10 2017-08-11 上海大学 一种3d机器视觉叉车行驶辅助安全系统及方法
CN107123142A (zh) * 2017-05-09 2017-09-01 北京京东尚科信息技术有限公司 位姿估计方法和装置
KR101834336B1 (ko) * 2016-09-27 2018-03-06 삼성중공업 주식회사 자재적치장 영상모니터링을 통한 자재위치 인식시스템 및 지게차
CN109573897A (zh) * 2018-12-21 2019-04-05 宁波如意股份有限公司 基于视觉的叉车agv货叉起身高度定位的系统及方法
CN109650292A (zh) * 2019-02-02 2019-04-19 北京极智嘉科技有限公司 一种智能叉车以及智能叉车的位置调整方法和介质
CN110054121A (zh) * 2019-04-25 2019-07-26 北京极智嘉科技有限公司 一种智能叉车以及容器位姿偏移检测方法

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CH536790A (de) * 1971-07-20 1973-05-15 Oehler Wyhlen Lagertechnik Ag Verfahren und Einrichtung zur Feinpositionierung einer vertikal beweglichen Plattform vor einem Lagerfach
DE2932899C2 (de) * 1979-08-14 1981-09-24 Jungheinrich Unternehmensverwaltung Kg, 2000 Hamburg Vorrichtung zum berührunglosen Messen der Höhe eines Lastträgerfahrzeug über einem Bezugspunkt
JP2006017634A (ja) * 2004-07-02 2006-01-19 Kubota Corp 移動作業機の位置検出装置
JP4731934B2 (ja) * 2005-02-07 2011-07-27 株式会社アイチコーポレーション 伸縮ブームのワイヤ切断検出装置
JP2006240873A (ja) * 2005-03-07 2006-09-14 Apurikotto Instruments Kk フォークモニタリング装置
JP4666154B2 (ja) * 2005-09-20 2011-04-06 株式会社豊田自動織機 フォークリフトにおける荷役作業支援装置
JP5069936B2 (ja) * 2007-04-18 2012-11-07 日立Ｇｅニュークリア・エナジー株式会社 資材保管位置管理システムおよびその方法
JP2010100408A (ja) * 2008-10-24 2010-05-06 Toyota Industries Corp スタッカクレーン
JP5900987B2 (ja) * 2014-04-16 2016-04-06 ニチユ三菱フォークリフト株式会社 荷役車両
JP2015228184A (ja) * 2014-06-02 2015-12-17 富士通株式会社 監視プログラム、監視システムおよび監視方法
US10328578B2 (en) * 2017-04-21 2019-06-25 X Development Llc Methods and systems for detecting, recognizing, and localizing pallets
CN107218927B (zh) * 2017-05-16 2019-08-09 上海交通大学 一种基于tof相机的货物托盘检测系统和方法
CN107562049A (zh) * 2017-08-09 2018-01-09 深圳市有光图像科技有限公司 一种通过对比图像识别智能叉车的位置的方法及智能叉车
CN107610176B (zh) * 2017-09-15 2020-06-26 斯坦德机器人（深圳）有限公司 一种基于Kinect的栈板动态识别与定位方法、系统及介质
CN109368552A (zh) * 2018-10-22 2019-02-22 广州微牌智能科技有限公司 托盘入叉方法、装置、系统、设备和存储介质

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101834336B1 (ko) * 2016-09-27 2018-03-06 삼성중공업 주식회사 자재적치장 영상모니터링을 통한 자재위치 인식시스템 및 지게차
CN106672859A (zh) * 2017-01-05 2017-05-17 深圳市有光图像科技有限公司 一种基于叉车视觉识别托盘的方法及叉车
CN107031629A (zh) * 2017-03-10 2017-08-11 上海大学 一种3d机器视觉叉车行驶辅助安全系统及方法
CN107123142A (zh) * 2017-05-09 2017-09-01 北京京东尚科信息技术有限公司 位姿估计方法和装置
CN109573897A (zh) * 2018-12-21 2019-04-05 宁波如意股份有限公司 基于视觉的叉车agv货叉起身高度定位的系统及方法
CN109650292A (zh) * 2019-02-02 2019-04-19 北京极智嘉科技有限公司 一种智能叉车以及智能叉车的位置调整方法和介质
CN110054121A (zh) * 2019-04-25 2019-07-26 北京极智嘉科技有限公司 一种智能叉车以及容器位姿偏移检测方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3960692A4

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11524846B2 (en) * 2020-10-19 2022-12-13 Gideon Brothers d.o.o. Pose determination by autonomous robots in a facility context
US11858741B2 (en) 2020-10-19 2024-01-02 Gideon Brothers d.o.o. Safety mode toggling by autonomous robots in a facility context
US11866258B2 (en) 2020-10-19 2024-01-09 Gideon Brothers d.o.o. User interface for mission generation of area-based operation by autonomous robots in a facility context
US11958688B2 (en) 2020-10-19 2024-04-16 Gideon Brothers d.o.o. Area-based operation by autonomous robots in a facility context
WO2022207215A1 (de) * 2021-03-31 2022-10-06 Still Gmbh Verfahren und system zur lasterkennung bei einem flurförderzeug

Also Published As

Publication number Publication date
CN110054121A (zh) 2019-07-26
US20220067960A1 (en) 2022-03-03
KR20210145802A (ko) 2021-12-02
EP3960692A4 (en) 2022-10-05
EP3960692A1 (en) 2022-03-02
US11625854B2 (en) 2023-04-11
JP7206421B2 (ja) 2023-01-17
JP2022525441A (ja) 2022-05-13
KR102461759B1 (ko) 2022-11-01
CN110054121B (zh) 2021-04-20

Similar Documents

Publication Publication Date Title
WO2020215772A1 (zh) 一种智能叉车以及容器位姿偏移检测方法
WO2021249568A1 (zh) 仓储机器人的控制方法、装置、设备及可读存储介质
US20200294244A1 (en) Robotic system with automated package registration mechanism and methods of operating the same
JP6805465B2 (ja) センサ誘導式ロボットを用いたボックスの位置特定、分離、およびピッキング
US9630316B2 (en) Real-time determination of object metrics for trajectory planning
EP3033293B1 (en) Method and system for automatically landing containers on a landing target using a container crane
US9205562B1 (en) Integration of depth points into a height map
US9393686B1 (en) Moveable apparatuses having robotic manipulators and conveyors to facilitate object movement
CN112070838B (zh) 基于二维-三维融合特征的对象识别与定位方法及装置
CN113264312A (zh) 货箱的提取方法、装置、系统、机器人和存储介质
US20220371200A1 (en) Robotic system for object size measurement
EP3418244B1 (en) Loading a container on a landing target
JP2022135879A (ja) 物体検出のために安全ボリュームリストを生成するための方法および計算システム
CN115159402B (zh) 货物放取方法、装置、电子设备及机器可读存储介质
CN117555308B (zh) 一种基于无人叉车的托盘回收方法、系统及存储介质
JP7385424B2 (ja) 移載対象ワーク向き認識システム
TWI799051B (zh) 叉取棧板的自動搬運車和叉取棧板方法
JP7408107B2 (ja) 物体取り扱いを伴うロボットシステムのためのシステム及び方法
US20240208736A1 (en) Ai-powered load stability estimation for pallet handling
JP2024078012A (ja) 荷姿状態検知装置
JP2024038642A (ja) ピッキングシステム
KR20230174126A (ko) 디팔레타이징 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19926162

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021563272

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 20217035759

Country of ref document: KR

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2019926162

Country of ref document: EP

Effective date: 20211125