Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/50—Depth or shape recovery
  • G06T7/55—Depth or shape recovery from multiple images

Definitions

• This disclosure relates to an image processing device, an image processing method, and a program.
• robots have been introduced into various situations, automating tasks. For example, when sorting objects that are merchandise in a warehouse, images of the objects are taken and the type of object is identified from the images. For this reason, it is necessary to obtain and register images of the objects that are merchandise in advance.
• Patent Document 1 describes a robot that autonomously extracts and learns images of object parts. Specifically, Patent Document 1 describes how a motion area image is extracted from a difference image of a moving image when a robot is brought into contact with an object and moved, and the robot is masked from its position to extract images of object parts.
• the objective of this disclosure is therefore to provide an image processing device that can solve the above-mentioned problem of the difficulty in acquiring an image of an object part.
• An image processing device includes: an acquisition unit for acquiring an image including an object being manipulated by a movable robot; A detection unit that detects a posture of the robot when it is moving; an extraction unit that extracts an object image of the object portion from the image based on a posture of the robot with respect to the image; Equipped with The structure is as follows.
• an image processing method includes: acquiring an image including an object being manipulated by a movable robot; Detecting the posture of the robot when it is moving; extracting an object image of the object portion from the image based on a pose of the robot with respect to the image;
• the structure is as follows.
• a program includes: acquiring an image including an object being manipulated by a movable robot; Detecting the posture of the robot when it is moving; extracting an object image of the object portion from the image based on a pose of the robot with respect to the image; The processing is carried out by a computer.
• the structure is as follows.
• this disclosure makes it easy to obtain images of object parts.
• FIG. 1 is a block diagram showing an overall configuration of an image processing system according to a first embodiment of the present disclosure.
• FIG. 2 is a diagram showing a state of processing by the image processing device disclosed in FIG. 1;
• FIG. 2 is a diagram showing a state of processing by the image processing device disclosed in FIG. 1;
• FIG. 2 is a diagram showing a state of processing by the image processing device disclosed in FIG. 1;
• FIG. 2 is a diagram showing a state of processing by the image processing device disclosed in FIG. 1;
• 2 is a flowchart showing the operation of the image processing device disclosed in FIG. 1 .
• FIG. 11 is a block diagram showing a hardware configuration of an image processing device according to a second embodiment of the present disclosure.
• FIG. 11 is a block diagram showing a configuration of an image processing device according to a second embodiment of the present disclosure.
• Fig. 1 is a diagram for explaining the configuration of an image processing system
• Fig. 2 to Fig. 6 are diagrams for explaining the processing operation of the image processing system.
• the image processing system in this embodiment generates an object image, which is an image of an object part, from a captured image.
• the image processing system has a function of generating an object image for pre-registration or learning in order to distinguish the type of object as described later.
• the image processing system is configured to include a robot arm 30 that manipulates an object 40, a camera 20 that captures an image showing the object 40, and an image processing device 10 that generates an object image from the captured image.
• the image processing system is used in a warehouse where objects that are products are sorted. For example, the image processing system first captures an image of the object 40 with the camera 20, and determines the type of object 40 shown in the captured image. At this time, if the image processing system cannot determine the type of object 40, it performs an object image generation process to newly register or learn the object image and type so that the type of object 40 can be determined in the future.
• the image processing system may have a function of sorting objects when used in the above-mentioned scenes.
• the image processing system may have a function of determining the type of object 40 shown in the captured image as described above, and if the type of object 40 can be determined, holding the object 40 with the robot arm 30, moving the object 40 to a target location according to the sorting rules, and sorting the object 40.
• the image processing system may take an image of the object 40 held by the robot arm 30, determine the type of the object 40, and if the type of the object 40 can be determined, move the object 40 to a target location according to the sorting rules, and sort the object 40.
• the image processing system may perform the process of generating an object image of the object part as it is.
• the process of determining the type of object 40 by the image processing system may be performed by registering an object image whose type has been determined in advance and matching the registered object image with the object 40 shown in the captured image, or may be performed by learning and generating a model for determining the type of object from a captured image in advance, and using the model.
• any method may be used to determine the type of object 40 from the image captured by the image processing system.
• the following describes the configuration for realizing the function of generating an object image of the object 40 using the image processing device 10 provided in the image processing system.
• the image processing device 10 is composed of one or more information processing devices each having a calculation device and a storage device. As described above, the camera 20 and the robot arm 30 are connected to the image processing device 10, and the image processing device 10 has the function of controlling image capture by the camera 20 and controlling the movement of the robot arm 30. As shown in FIG. 1, the image processing device 10 is composed of a photographing unit 11, an extraction unit 12, and a generation unit 13. The functions of the photographing unit 11, the extraction unit 12, and the generation unit 13 can be realized by the calculation device executing a program for realizing each function stored in the storage device. The image processing device 10 is also composed of a camera image storage unit 16 and an object image storage unit 17. The camera image storage unit 16 and the object image storage unit 17 are composed of a storage device. Each component will be described in detail below.
• the photographing unit 11 controls the operation of the robot arm 30 to operate the object 40, and also photographs an image including the object 40 being operated by the camera 20, and acquires it as a photographed image.
• the robot arm 30 includes a movable arm unit 31 and an operation unit 32 that adsorbs and holds the object 40 provided at the tip of the arm unit 31.
• the operation unit 32 of the robot arm 30 is not limited to adsorbing and holding the object 40, and may hold the object 40 in any structure, such as by grasping and holding it.
• the operation unit 32 does not necessarily have to hold the object 40, and may simply have a structure that performs some operation on the object 40, such as pushing the object 40 to move it or rolling it.
• the robot arm 30 is not necessarily limited to being a robot having an arm unit 31, and may be a robot of any structure.
• the object 40 is a rectangular parallelepiped is described, but the object 40 may have any shape.
• the photographing unit 11 first drives the arm unit 31 to control its position, and controls the holding operation by the operation unit 32 to hold the object 40 with the operation unit 32.
• the photographing unit 11 controls the position of the arm unit 31 so that the positions of the operation unit 32 and the held object 40 are within the shooting range of the image captured by the camera 20, and captures the image with the camera 20.
• the photographing unit 11 stores position information representing the shooting range set in advance according to the installation position and shooting angle of the camera 20, and by controlling the position of the arm unit 31 according to the position information of the shooting range, it is possible to set the operation unit 32 and the held object 40 to be located within the shooting range.
• the photographing unit 11 detects the position of the robot arm 30 it controls, particularly the position of the operation unit 32, and performs calibration with the shooting range of the camera 20, and can set the robot arm 30 to be located within the image captured by the camera 20, i.e., within the captured image.
• the photographing unit 11 photographs an image with the camera 20 as described above, and as shown in FIG. 4 (4-1), photographs an image showing a part of the arm unit 31, the operation unit 32, and the object 40 being held. At this time, the photographed image shows a part of the object 40 seen from one direction. For example, if the object 40 is a rectangular parallelepiped, the photographed image shows a first of the six faces positioned at the front.
• the photographing unit 11 then stores the photographed image in the camera image storage unit 16. At this time, the photographing unit 11 associates position information representing the posture of the robot arm 30 detected at the time of photographing with the photographed image and stores it.
• the photographing unit 11 stores position information of the robot arm 30 in the photographed image using position information of the photographing range of the image photographed by the camera 20 described above, and the position (angle) of the arm unit 31 and the position of the operation unit 32 of the robot arm 30 at the time of photographing.
• the photographing unit 11 also controls the robot arm 30 to move while holding the object 40.
• the photographing unit 11 moves the robot arm 30 by rotating the joint angle of the arm 31 to move the position of the operation unit 32 as shown by the arrow in FIG. 2 (2-1), or by rotating the operation unit 32 around the longitudinal axis of the arm 31 as shown by the arrow in FIG. 2 (2-2).
• the photographing unit 11 then obtains an image taken after the robot arm 30 has been moved, and stores the images taken before and after the movement in the camera image storage unit 16 in association with each other.
• the photographing unit 11 stores the image taken before the movement, indicated by the dotted line, and the image taken after the movement, indicated by the solid line, in association with each other, as shown in FIG. 3 (3-2).
• the photographing unit 11 moves the position of the arm unit 31 so that the posture of the robot arm 30 is different from when the photographed image was already taken, as described above.
• the photographing unit 11 moves the robot arm 30 so that another part of the object 40 seen from another direction different from the part seen from one direction in the photographed image already taken is shown in the new photographed image.
• the photographing unit 11 photographs an image with the camera 20 after moving the robot arm 30, so that it is possible to photograph an image in which another part of the object 40 seen from another direction is located in front, as shown in FIG. 4 (4-2), unlike the photographed image in FIG. 4 (4-1).
• the photographing unit 11 associates the position information representing the posture of the robot arm 30 detected during photographing with the photographed image and stores it in the camera image storage unit 16.
• the photographing unit 11 further moves the position of the arm unit 31 so that the posture of the robot arm 30 is different from when the photographed image was taken in the past, and repeats photographing of the photographed image by the camera 20.
• the photographing unit 11 associates position information indicating the posture of the robot arm 30 detected at the time of photographing with each photographed image and stores it in the camera image storage unit 16. Note that by associating the position information of the robot arm 30 with the photographed image, it is possible to specify from which direction each photographed image is an image of the object 40.
• the photographing unit 11 also controls the robot arm 30 to hold different holding points of the object 40.
• the photographing unit 11 controls the operation of the robot arm 30 so as to change the holding point of the object 40 by the operation unit 32 of the robot arm 30.
• the photographing unit 11 takes a photographed image each time the holding point of the object 40 by the robot arm 30 is changed.
• the photographing unit 11 releases the holding of the face already held on the object 40, adsorbs and holds another face different from the face, and obtains a photographed image in which the released face is positioned in front.
• the photographing unit 11 associates position information representing the posture of the robot arm 30 detected at the time of photographing with each photographed image and stores it in the camera image storage unit 16.
• the image capturing unit 11 may store position information associated with the operation of changing the holding position of the robot arm 30 on the object 40 in association with the captured image. This makes it possible to identify the direction from which the captured image of the object 40 was viewed.
• the extraction unit 12 extracts an object image, which is a partial image including the object 40, based on the captured image and the position information representing the posture of the robot arm 30 as described above. At this time, the extraction unit 12 extracts an object image candidate g, which is a candidate for the object image, from the captured image based on the position and movement direction of the operation unit 32 in the captured image. For example, the extraction unit 12 acquires position information of the operation unit 32 located at the tip of the arm unit 31 of the robot arm 30 in a captured image such as that shown in FIG.
• the extraction unit 12 sets an extension of the position of the operation unit 32, that is, a region of a predetermined range located further from the position of the operation unit 32 toward the tip direction along the longitudinal direction of the arm unit 31, as the object extraction region R. Then, the extraction unit 12 extracts a partial image that can be recognized as an object existing in the set object extraction region R as the object image candidate g. For example, the extraction unit 12 performs edge processing in the object extraction region R, and extracts a partial image that can be recognized as an object surrounded by a contour line as the object image candidate g. This makes it possible to extract, for example, an object image candidate g as shown in FIG. 4 (1-11) from the captured image shown in FIG. 4 (4-1).
• the extraction unit 12 acquires position information and a moving direction of the operation unit 32 located at the tip of the arm unit 31 of the robot arm 30 in successive captured images in which the arm unit 31 of the robot arm 30 is moving as indicated by the arrow in FIG. 3 (3-2).
• the extraction unit 12 then extracts partial images that can be recognized as objects moving in the same direction near the position of the operation unit 32 as object image candidates g.
• the extraction unit 12 extracts the object image candidates g by, for example, performing an optical flow estimation process on the successive captured images. This makes it possible to extract object image candidates g as shown in FIG. 4 (1-11) from the captured images shown in FIG. 4 (4-1).
• the extraction unit 12 extracts object image candidates g for each of the captured images captured from all directions. That is, for example, object image candidates g as shown in FIG. 4 (1-11), (4-12), ... can be extracted from each of the captured images shown in FIG. 4 (4-1), (4-2), ....
• the method of extracting object image candidates g by the extraction unit 12 described above is not limited to the above-mentioned method.
• the extraction unit 12 may extract object image candidates g by identifying partial images that can be recognized as some kind of object located around the position information of the operation unit 32 based on the position information.
• the extraction unit 12 generates an object image G by further deleting image portions corresponding to the arm unit 31 and the operation unit 32 from the object image candidate g extracted as described above.
• the extraction unit 12 acquires shape information of the arm unit 31 and the operation unit 32 in advance, and deletes image portions of the arm unit 31 and the operation unit 32 present in the object image candidate g using the shape information and position information of the arm unit 31 and the operation unit 32.
• the extraction unit 12 can delete image portions of the robot arm 30 as shown by gray areas in Figures 5 (4-1), (4-2), ... from each object image candidate g, and generate object images G of only the object portion.
• the extraction unit 12 can extract object images G of the object 40 viewed from all directions.
• an object image G in which each of the six faces is positioned in the front direction can be extracted.
• the extraction unit 12 may delete the image portion of the robot arm 30 from the object image candidate g by any method, or may create the object image G without deleting the image portion of the robot arm 30 from the object image candidate g.
• the generation unit 13 (extraction unit) generates a three-dimensional object image G' of an object 40 from multiple object images G extracted for one object 40. For example, the generation unit 13 recognizes the shooting direction of each object image G with respect to the object 40 based on the position information of the robot arm 30 associated with the captured images that are the source of the multiple object images G, and generates a three-dimensional object image G' from the multiple object images G based on the shooting direction. The generation unit 13 then stores the generated three-dimensional object image G' in the object image storage unit 17. Note that the three-dimensional object image G' stored in the object image storage unit 17 can be registered as an image of the object 40, which is a new type of product, or used to generate a model that distinguishes the type of object, as described above.
• the image processing device 10 controls the operation of the robot arm 30, and causes the robot arm 30 to hold and manipulate the object 40 (step S1). In addition, the image processing device 10 captures an image including the object 40 being manipulated by the robot arm 30 with the camera 20, and acquires it as a captured image (step S2). At this time, the image processing device 10 detects the position of the robot arm 30 in the captured image, and stores this position information in association with the captured image.
• the image processing device 10 acquires a captured image of the robot arm 30 after it has been moved (step S4). At this time, the image processing device 10 detects the position of the robot arm 30 in the captured image, as described above, and stores the position information in association with the captured image. This allows the image processing device 10 to acquire captured images of the object 40 viewed from various directions. Note that the image processing device 10 may store captured images before and after movement in association with each other.
• the image processing device 10 releases the robot arm 30 from holding the object 40 and changes the operation location to hold another part of the object 40 (Yes in step S5), it acquires a captured image after the operation location has been changed by the robot arm 30 (step S2). At this time, the image processing device 10 detects the position of the robot arm 30 in the captured image, as described above, and stores this position information in association with the captured image. This allows the image processing device 10 to also acquire an image of the part where the object 40 was held by the robot arm 30.
• the image processing device 10 extracts an object image, which is an image of the object 40 portion, based on the captured image as described above and the position information representing the posture of the robot arm 30 (step S6).
• the image processing device 10 extracts object image candidates g, which are candidates for the object image, from the captured image based on the position and movement direction of the operation unit 32 in the captured image.
• the image processing device 10 further deletes the image portions corresponding to the arm unit 31 and operation unit 32 from the extracted object image candidates g to generate an object image G. This allows the image processing device 10 to generate object images G in which the object 40 is viewed from all directions.
• the image processing device 10 generates a three-dimensional object image G' of the object 40 from the multiple object images G generated for one object 40 (step S7). Then, the image processing device stores the generated three-dimensional object image G' in the object image storage unit 17.
• the image processing device 10 can extract an object image by using a captured image of the object 40 held by the robot arm 30 and position information of the robot arm 30. This allows an object image to be acquired through simple image processing. In particular, by capturing images while moving the object held by the robot arm 30 or capturing images while changing the position at which the object 40 is held, object images viewed from various directions can be acquired, and high-quality three-dimensional object images of the object 40 can be generated. As a result, in situations where the objects 40 are automatically sorted, high-quality object images can be registered and used for learning, which can improve the accuracy and speed of the sorting work.
• Fig. 7 to Fig. 8 are block diagrams showing the configuration of an image processing device in embodiment 2. Note that this embodiment shows an outline of the configuration of the image processing device described in the above embodiment.
• the image processing device 100 is configured as a general information processing device, and is equipped with the following hardware configuration, as an example.
• ⁇ CPU Central Processing Unit
• ROM Read Only Memory
• RAM Random Access Memory
• Program group 104 loaded into RAM 103
• a storage device 105 for storing the program group 104
• a drive device 106 that reads and writes data from and to a storage medium 110 outside the information processing device.
• a communication interface 107 that connects to a communication network 111 outside the information processing device
• Input/output interface 108 for inputting and outputting data
• a bus 109 that connects each component
• FIG. 7 shows an example of the hardware configuration of an information processing device that is the image processing device 100, and the hardware configuration of the information processing device is not limited to the above-mentioned case.
• the information processing device may be configured with a part of the above-mentioned configuration, such as not having the drive device 106.
• the information processing device may use a GPU (Graphic Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination of these.
• the image processing device 100 can be equipped with the acquisition unit 121, detection unit 122, and extraction unit 123 shown in FIG. 8 by having the CPU 101 acquire the program group 104 and execute it.
• the program group 104 is stored in advance in the storage device 105 or ROM 102, for example, and is loaded into the RAM 103 and executed by the CPU 101 as necessary.
• the program group 104 may be supplied to the CPU 101 via the communication network 111, or may be stored in advance in the storage medium 110, and the drive device 106 may read out the programs and supply them to the CPU 101.
• the acquisition unit 121, detection unit 122, and extraction unit 123 described above may be constructed with dedicated electronic circuits for realizing such means.
• the acquisition unit 121 acquires images including an object being operated by a movable robot. At this time, it is preferable that the acquisition unit 121 acquires images of the object viewed from multiple directions.
• the detection unit 122 detects the posture of the robot when it is moving. At this time, it is preferable that the detection unit 122 detects the posture including the position of the operation unit that operates the object of the robot.
• the extraction unit 123 extracts an object image of the object part from the image based on the posture of the robot relative to the image. At this time, it is preferable that the extraction unit 123 extracts the object image based on the position of the robot's operation part, and it is also preferable that the extraction unit 123 extracts each object image from a plurality of images.
• Non-transitory computer readable medium includes various types of tangible storage medium.
• Examples of non-transitory computer readable medium include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R/W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)).
• the program may also be supplied to a computer by various types of transitory computer readable medium. Examples of transitory computer readable medium include electrical signals, optical signals, and electromagnetic waves.
• the temporary computer-readable medium can provide the program to the computer via a wired communication path, such as an electric wire or optical fiber, or via a wireless communication path.
• the present disclosure has been described above with reference to the above-mentioned embodiments, but the present disclosure is not limited to the above-mentioned embodiments.
• Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.
• at least one or more of the functions of the above-mentioned acquisition unit 121, detection unit 122, and extraction unit 123 may be executed by an information processing device installed and connected anywhere on the network, that is, they may be executed by so-called cloud computing.
• the detection unit detects the posture including a position of an operation unit of the robot that operates the object, The extraction unit extracts the object image based on a position of the operation unit in the image.
• Image processing device. (Appendix 3) 3.
• the image processing device according to claim 2 The extraction unit extracts the object image based on an extension point of the position of the operation unit in the image.
• Image processing device. (Appendix 4) 4.
• the image processing device according to claim 2 The detection unit acquires the attitude including a moving direction of the operation unit; The extraction unit extracts the object image based on a moving direction of the operation unit in the image.
• Image processing device. (Appendix 5) 5.
• the image processing device extracts the object image based on a portion in the image that moves in the same direction as the movement direction of the operation unit.
• Image processing device. Appendix 6) 6.
• An image processing device A control unit for controlling an operation of the robot is provided. The control unit controls the robot to manipulate the object in different postures, the acquisition unit acquires the images when the robot manipulates the object in different postures, the extraction unit extracts the object image from each of the images based on the posture of the robot with respect to each of the images, and further generates a three-dimensional object image from the extracted object image.
• Image processing device. Appendix 7) 7. An image processing device according to claim 1, A control unit for controlling an operation of the robot is provided.
• the control unit controls the robot to operate different operation points of the object, the acquisition unit acquires the images when the robot operates different operation locations of the object, the extraction unit extracts the object image from each of the images based on the posture of the robot with respect to each of the images, and further generates a three-dimensional object image from the extracted object image.
• Image processing device (Appendix 8) acquiring an image including an object being manipulated by a movable robot; Detecting the posture of the robot when it is moving; extracting an object image of the object portion from the image based on a pose of the robot with respect to the image; Image processing methods. (Appendix 9) 9.
• An image processing method comprising: Detecting the posture of the robot including a position of an operation unit that operates the object; extracting the object image based on a position of the operation unit in the image; Image processing methods. (Appendix 10) 10. The image processing method according to claim 8, further comprising: Controlling the robot to manipulate the object in different postures; acquiring the images when the robot is manipulating the object in different postures; extracting the object image from each of the images based on the posture of the robot with respect to each of the images, and generating a three-dimensional object image from the extracted object image. Image processing methods. (Appendix 11) 11.
• An image processing method comprising: Controlling the robot to operate different operation points of the object; acquiring the images when the robot operates different operation points of the object, extracting the object image from each of the images based on the posture of the robot with respect to each of the images, and generating a three-dimensional object image from the extracted object image.
• Image processing methods (Appendix 12) acquiring an image including an object being manipulated by a movable robot; Detecting the posture of the robot when it is moving; extracting an object image of the object portion from the image based on a pose of the robot with respect to the image; A computer-readable storage medium that stores a program for causing a computer to execute processing.

Landscapes

• Engineering & Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Manipulator (AREA)
• Image Analysis (AREA)

Priority Applications (2)

Applications Claiming Priority (1)

Publications (1)

Family

ID=92420984

Family Applications (1)

Country Status (2)

Citations (3)

• 2023
  • 2023-02-17 JP JP2025500601A patent/JPWO2024171452A1/ja active Pending
  • 2023-02-17 WO PCT/JP2023/005801 patent/WO2024171452A1/ja not_active Ceased

Patent Citations (3)

Also Published As

Similar Documents

Legal Events