JP2020149162A - Information processing apparatus, image processing program and image processing method - Google Patents

Abstract

To provide an information processing apparatus, an image processing program and an image processing method that precisely determine a pixel value of a pixel whose depth cannot be measured in a depth image.SOLUTION: An information processing apparatus comprises: a depth image acquisition part 22 which acquires a depth image; an RGB image acquisition part 20 which acquires a two-dimensional image; a defective region extraction part 24 which extracts a defective region including a pixel whose depth cannot be measured in a part of the depth image; a corresponding region extraction part 26 which extracts a first region corresponding to the defective region from the two-dimensional image, a second region having a higher similarity with the first region than specified, from the two-dimensional image, and a third region of a depth image corresponding to the second region; and a complementation part 28 which determines, based upon a pixel value of a pixel included in the third region of the depth image corresponding to the second region, a pixel value of the unmeasurable pixel included in the defective region to complement the defective region.SELECTED DRAWING: Figure 3

Description

The present invention relates to an information processing device, an image processing program, and an image processing method.

In a distribution warehouse, a work (picking work) is performed in which necessary products are collected from a cardboard case packed in product units by a manufacturer for each retail store and packed in a container. At present, picking work is mainly manual work, but in order to reduce labor costs and solve labor shortages, automation using robots (robot picking) is being studied.

In robot picking, product information (shape information, location information, etc.) in the distribution warehouse is required, but when handling a wide variety of products in the distribution warehouse, information on individual products is obtained in advance. It's difficult to keep. Therefore, it is necessary to acquire the shape information and the position information of the product at the site. On the other hand, conventionally, a technique of using a color image or a depth image to acquire the shape of a reflecting object, and a technique of using an imaging means or a distance measuring means to stably identify a target object are known. (See, for example, Patent Documents 1 and 2).

Special Table 2015-526692 Japanese Unexamined Patent Publication No. 7-88791

In order to obtain three-dimensional position information of the product to be picked, for example, an RGBD camera may be used. An RGB image and a depth image (Dept image) can be obtained from the RGBD camera, but due to the reflection of the illumination light on the product surface, a part of the depth image includes a pixel whose depth cannot be measured (hereinafter, , Called a defect area) may occur. When such a defect area occurs, it becomes impossible to obtain accurate position information of the product.

In one aspect, it is an object of the present invention to provide an information processing apparatus, an image processing program, and an image processing method capable of accurately determining the pixel value of a pixel whose depth cannot be measured in a depth image.

In one aspect, the information processing apparatus includes an acquisition unit that acquires a depth image and a two-dimensional image of the placement location where the product is placed, and a pixel whose depth cannot be measured in a part of the depth image. When a defect region exists, the first extraction unit that extracts the first region corresponding to the defect region from the two-dimensional image and the second extraction unit that has a degree of similarity between the two-dimensional image and the first region is a predetermined value or more. A pixel whose depth included in the defect region of the depth image cannot be measured based on the pixel values of the second extraction unit for extracting the region and the pixels included in the third region of the depth image corresponding to the second region. It is provided with a determination unit for determining the pixel value of.

It is possible to accurately determine the pixel value of a pixel whose depth cannot be measured in a depth image.

It is a figure which shows roughly the structure of the robot system which concerns on one Embodiment. It is a figure which shows the hardware configuration of the information processing apparatus of FIG. It is a functional block diagram of the information processing apparatus of FIG. It is a flowchart which shows the process of the information processing apparatus of FIG. It is a figure which shows an example of the RGB image. It is a figure which shows an example of a depth image. It is a figure which shows one of the defect regions in a depth image. FIG. 8A is a diagram showing a first region and a second region in an RGB image, and FIG. 8B is a diagram showing a third region and a defect region in a depth image. It is a figure for demonstrating the example which extracts the 2nd region while rotating the 1st region. It is a figure which shows the correction pixel (x, y) and the corresponding pixel (x, y) in a depth image. It is a figure which shows the depth image after complementing one defect area. It is a figure which shows the depth image after complementing all the defect regions.

Hereinafter, one embodiment of the robot system will be described in detail with reference to FIGS. 1 to 12. FIG. 1 schematically shows the configuration of the robot system 100 according to the embodiment. The robot system 100 is a system that executes a work (picking work) of collecting necessary products for each retail store from a cardboard case packed in product units by a manufacturer and packing them in a container in a distribution warehouse. In addition, the product in this embodiment shall be a canned product, a boxed product, or the like. That is, if they are the same product, there is no individual difference in shape and the like, and it is assumed that they have the same appearance.

As shown in FIG. 1, the robot system 100 includes an information processing device 10, a camera 12, and a robot 14. The information processing device 10 and the camera 12 and the robot 14 are connected so as to be capable of wireless communication or wired communication. A plurality of cameras 12 and robots 14 may be connected to the information processing device 10, but for convenience of explanation, one camera 12 and one robot 14 are connected to the information processing device 10. It is assumed that it has been done.

The information processing device 10 is a terminal device such as a PC (Personal Computer), and has a hardware configuration as shown in FIG. Specifically, as shown in FIG. 2, the information processing apparatus 10 includes a CPU (Central Processing Unit) 90, a ROM (Read Only Memory) 92, a RAM (Random Access Memory) 94, and a storage unit (here, an HDD (Hard)). Disk Drive)) 96, communication interface 97, drive 99 for portable storage medium, and the like are provided. Each component of the information processing device 10 is connected to the bus 98. In the information processing apparatus 10, the CPU 90 reads a program (including an image processing program) stored in the ROM 92 or HDD 96, or a program (including an image processing program) read from the portable storage medium 91 by the portable storage medium drive 99. By executing, the functions of each part shown in FIG. 3 are realized. The functions of each part in FIG. 3 may be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The details of each part of FIG. 3 will be described later.

The camera 12 is an RGBD (Red Green Blue Depth) camera, which is capable of capturing an RGB image (color image) as a two-dimensional image and a depth image (Dept image). The RGB image and the depth image captured by the camera 12 are transmitted to the information processing device 10. The camera 12 is installed directly above the cardboard case in which the product to be picked is packed and the upper part is open, and the camera 12 executes a picture inside the cardboard case at a timing immediately before the robot 14 picks the product. The pixel value of the depth image represents the depth in the image, that is, the height of the photographed product. Further, the camera 12 does not necessarily have to be installed directly above the cardboard case, and may be installed at a position deviated from directly above the cardboard case.

The robot 14 executes the picking operation according to the instruction from the information processing device 10. In the picking work, the robot 14 picks the product to be picked in the cardboard case and executes the work of loading it in the container. It is assumed that at least two identical products are contained in the cardboard case.

Next, the function of the information processing device 10 will be described with reference to FIG.

As shown in FIG. 3, in the information processing apparatus 10, when the CPU 90 executes a program, the RGB image acquisition unit 20, the depth image acquisition unit 22, the defect area extraction unit 24, the corresponding area extraction unit 26, and the complement unit 28 are executed. , And the function as the robot control unit 30 are realized.

The RGB image acquisition unit 20 acquires an RGB image taken by the camera 21. As an example, the RGB image acquired by the RGB image acquisition unit 20 is assumed to be an image as shown in FIG. The RGB image of FIG. 5 is an image in which canned foods are arranged at six places in the cardboard case. In the example of FIG. 5, it is assumed that the heights of the upper surfaces of the canned foods arranged at each of the six places are almost the same.

The depth image acquisition unit 22 acquires a depth image taken by the camera 21. It is assumed that the RGB image and the depth image taken by the same camera 21 at the same timing are images taken in the same range, and they are associated with each other by giving a common identifier or the like. FIG. 6 shows an example of a depth image corresponding to the RGB image of FIG. In the image of FIG. 6, the height (depth) of the edge portion of the corrugated cardboard case and the upper surface of the can can be measured. The image of FIG. 6 is an image that is displayed darker as the height increases. The depth image may be a grayscale image as shown in FIG. 6 (an image that expresses high and low by light and darkness) or a color image (an image that expresses high and low by color).

In the depth image, there are pixels whose depth (height) cannot be measured due to the influence of reflection of illumination light (hereinafter, also referred to as “unmeasurable pixels”), and the pixels are shown in black in FIG. , Defect values (for example, pixel values with 0 brightness) are assigned. As a result, a pixel whose depth can be measured and a pixel whose depth cannot be measured are clearly distinguished. It is assumed that the defect value is predetermined for each camera 12.

The defect region extraction unit 24 extracts the defect region from the depth image when the depth image acquired by the depth image acquisition unit 22 has a region (defect region) in which the depth cannot be measured. Here, the defect region means a rectangular region including non-measurable pixels. When the defect region is extracted, the defect region extraction unit 24 passes the position information of the defect region to the corresponding region extraction unit 26.

The corresponding area extraction unit 26 extracts an RGB image region (first region) corresponding to the defect region extracted by the defect region extraction unit 24, and is similar to the first region in the RGB image (similarity is equal to or higher than a predetermined degree). ) Extract another region (second region). Further, the corresponding region extraction unit 26 extracts the region (third region) of the depth image corresponding to the extracted second region, and passes the position information of the third region to the complementary unit 28. Here, the RGB image region (first region) corresponding to the defect region means an region in the RGB image in which the defect region in the depth image is captured in the same range as the captured region. Further, the region of the depth image (third region) corresponding to the second region means a region in the depth image in which the same range as the second region in the RGB image is captured.

The complementing unit 28 determines the pixel value of the non-measurable pixel included in the defect region based on the pixel value of each pixel in the third region, and complements the defect region.

The robot control unit 30 controls the picking operation of the product by the robot 14 based on the depth image (if the defect region exists, the depth image after complementation) and the RGB image.

(About the processing of the information processing device 10)
Next, the details of the processing executed by the information processing apparatus 10 will be described in detail with reference to other drawings as appropriate according to the flowchart of FIG. The timing at which the processing of FIG. 4 is performed is the timing at which the RGB image and the depth image in the cardboard case are taken by the camera 12.

In the process of FIG. 4, first, in step S10, the RGB image acquisition unit 20 acquires an RGB image, and the depth image acquisition unit 22 acquires a depth image. Here, for example, it is assumed that an image as shown in FIG. 5 is acquired as an RGB image, and an image as shown in FIG. 6 is acquired as a depth image.

Next, in step S12, the defect region extraction unit 24 identifies one defect region of the depth image. Here, the defect region is a rectangular region including non-measurable pixels (pixels having a defect value) as described above. The defect region extraction unit 24 binarizes the depth image with the defect value and the other pixel values, and labels each pixel to specify a rectangular range (defect region) including the non-measurable pixel. FIG. 7 shows a state in which one defect region (rectangular range shown by a thick solid line) is specified from the depth image. The defect region extraction unit 24 may not specify a small defect region that does not affect the picking operation by the robot 14. That is, the defective area smaller than the predetermined area (less than the predetermined number of pixels) may be ignored.

Next, in step S14, the corresponding region extraction unit 26 determines whether or not the defective region has been identified. That is, it is determined whether or not a defective region remains in the depth image. If the determination in step S14 is denied, the process proceeds to step S28, but if it is affirmed, the process proceeds to step S16.

When the process proceeds to step S16, the corresponding region extraction unit 26 extracts the region (first region) of the RGB image corresponding to the specified defect region. Specifically, the corresponding region extraction unit 26 specifies the coordinate value of the defect region in the depth image, and extracts the region of the RGB image that matches the specified coordinate value as the first region. Here, it is assumed that the region shown by the thick solid line frame in FIG. 8A is extracted as the first region.

Next, in step S18, the corresponding region extraction unit 26 extracts a second region similar to the first region from the RGB image. As described above, since a plurality of the same products are arranged in the corrugated cardboard case, there is a high possibility that a portion similar to the first region exists in the RGB image. Therefore, the corresponding region extraction unit 26 performs template matching using the image of the first region as a template, and extracts a region similar to the first region (similarity is equal to or higher than a predetermined value) as the second region. Here, as an example, it is assumed that the region shown by the thick broken line frame in FIG. 8A is extracted as the second region. When the RGB image does not have a region similar to the template (first region), the corresponding region extraction unit 26 rotates the template of the first region by a predetermined angle (for example, 10 °) while rotating the RGB image. The process of searching for the second region may be repeatedly executed within. As a result, even if the orientation of the product photographed in the first region is different from that of the other products as shown in FIG. 9, the second region can be extracted as shown by the thick broken line frame.

Next, in step S20, the corresponding region extraction unit 26 extracts the third region of the depth image corresponding to the second region. In this case, the corresponding region extraction unit 26 specifies the coordinate value of the second region in the RGB image, and extracts the region of the depth image that matches the specified coordinate value as the third region. In FIG. 8B, a third region corresponding to the second region shown in FIG. 8A is shown by a thick broken line frame.

Next, in step S22, the complementary unit 28 determines the pixel value of the non-measurable pixel in the defect region based on the pixel value of each pixel in the third region. Hereinafter, the process of step S22 will be described in detail.

First, among the pixels whose depth can be measured from the defect region, the complementary unit 28 uses pixels other than the pixels indicating the depth (height) of the bottom (the surface on which the product is arranged) of the corrugated cardboard case as correction pixels. Identify. It is assumed that the depth (height) of the bottom of the corrugated cardboard case has been measured in advance and is stored in the complementary unit 28.

Next, the complementary unit 28 acquires the pixel value I _A (x, y) of each of the correction pixels (x, y) specified in the defect region. Further, the complementing unit 28 acquires the pixel value I _B (x, y) of the pixel (corresponding pixel (x, y)) corresponding to the correction pixel (x, y) in the defect region in the third region. .. Note that FIG. 10 shows an example of a correction pixel (x, y) and a corresponding pixel (x, y). In FIG. 10, the pixel located in the gray portion in the defect region becomes the correction pixel (x, y), and exists at the position corresponding to the correction pixel (x, y) in the defect region in the third region. The pixels to be used are the corresponding pixels (x, y). In this embodiment, it is assumed that n correction pixels (x, y) and n corresponding pixels (x, y) can be acquired.

Next, the complementary unit 28 has a pixel value I _A (x, y) of the correction pixel (x, y) and a pixel value I _B (x) of the corresponding pixel (x, y) based on the following equation (1). Find the difference δ (x, y) from, y), respectively.
δ (x, y) = I A (x, y) -I B (x, y) ... (1)

Next, the complementary unit 28 obtains the average value of all the obtained δ (x, y) from the following equation (2).

Next, the complement 28 corresponds to the pixel value I _A '(x, y) of each non-measurable pixel (x, y) in the defect region corresponding to the non-measurable pixel (x, y) in the third region. pixel (x, y) each pixel value I _B '(x, y) is used to determine the following equation (3).

By obtaining the pixel value I _A '(x, y) of each unmeasurable pixel (x, y) in the defective region using the above equation (3), the depth between the third region and the defective region (on the upper surface of the product). Even when the height) is significantly different, the pixel value I _A '(x, y) of the non-measurable pixel (x, y) can be accurately determined.

When it is clear that the heights of the upper surfaces of the products arranged on the corrugated cardboard case are almost the same, the complementary unit 28 uses the above method to measure the pixel values of the unmeasurable pixels (x, y). It is not necessary to determine I _A '(x, y). That is, the complementary unit 28 sets the pixel value I _A '(x, y) of the unmeasurable pixel (x, y) in the defect region as the pixel value I _B of the corresponding pixel (x, y) in the third region. '(x, y) may be adopted as it is. By doing so, the pixel values of the non-measurable pixels (x, y) can be easily determined, so that the processing amount can be reduced.

Returning to FIG. 4, in the next step S24, the complementing unit 28 complements the defective region of the depth image with the determined pixel value. As a result, the depth image (FIG. 6) acquired by the depth image acquisition unit 22 is complemented as shown in FIG.

Next, in step S26, the defect region extraction unit 24 identifies one unspecified defect region in the depth image. After that, the process returns to step S14. After that, the processes of steps S16 to S26 are repeated until the determination in step S14 is denied, and when the determination in step S14 is denied, the process proceeds to step S28. At the stage of shifting to step S28, as shown in FIG. 12, there is no defect region in the depth image.

When the process proceeds to step S28, the robot control unit 30 gives an instruction to the robot based on the depth image and the RGB image. In this case, the robot control unit 30 uses the acquired depth image if there is no defect region in the depth image acquired by the depth image acquisition unit 22, but if there is a defect region, the robot control unit 30 uses the complemented depth image. It shall be used. The robot control unit 30 identifies the three-dimensional position and shape of the product (canned product) based on the depth image and the RGB image, drives the robot 14 based on the specific result, and executes the product pick-up work on the robot 14. Let me.

When the processes up to step S28 are performed as described above, all the processes of FIG. 4 are completed.

As can be seen from the above description, in the present embodiment, the RGB image acquisition unit 20 and the depth image acquisition unit 22 acquire a depth image and an RGB image obtained by photographing the corrugated cardboard case on which the product is placed. The function as an acquisition unit is realized. In addition, the corresponding region extraction unit 26 extracts the first region corresponding to the defect region from the RGB image, and the second region having a similarity with the first region or more from the RGB image. 2 The function as an extraction unit is realized.

As described in detail above, according to the present embodiment, the depth image acquisition unit 22 and the RGB image acquisition unit 20 acquire the depth image and the RGB image obtained by photographing the corrugated cardboard case on which the product is placed. Further, when a defect region including a non-measurable pixel exists in a part of the depth image, the corresponding region extraction unit 26 extracts the first region corresponding to the defect region from the RGB image and the first region from the RGB image. A second region having a degree of similarity with or more than a predetermined value is extracted. Then, the complementing unit 28 determines the pixel value of the non-measurable pixel included in the defect region based on the pixel value of the pixel included in the third region of the depth image corresponding to the second region. As a result, in the present embodiment, even if a non-measurable pixel exists in a part of the depth image, the pixel value of the non-measurable pixel is accurately determined based on the pixel value of another pixel in the depth image. Can be done.

Further, according to the present embodiment, the complementary unit 28 identifies pixels (correction pixels (x, y)) for which the depth (height) in the defect region can be measured. Then, the complementary unit 28 has a pixel value I _A (x, y) of the specified correction pixel and a pixel value of the corresponding pixel (x, y) in the third region corresponding to the correction pixel (x, y). Defects based on the difference from I _B (x, y) and the pixel value I _B '(x, y) of the pixels (x, y) in the third region corresponding to the unmeasurable pixels in the defect region. The pixel value I _A '(x, y) of the non-measurable pixel in the region is determined (see the above equation (3)). As a result, even if the depth of the defective region and the depth of the third region are significantly different (when the height of the upper surface of the can is different), an appropriate value is determined as the pixel value I _A '(x, y). can do.

Further, according to the present embodiment, the complementary unit 28 is a mounting surface (bottom surface of the corrugated cardboard case) on which the product is placed among the pixels whose depth in the defect region can be measured as the correction pixels (x, y). ), Specify the pixel that was shot. As a result, the pixel value I _A '(x, y) can be accurately determined even when the defect region or the third region includes a portion where the bottom surface of the corrugated cardboard case is photographed.

In the above embodiment, the case where template matching is performed using the defective region as a template when searching for the first region has been described, but the present invention is not limited to this. For example, a method of specifying the first region by using the local feature amount of the defect region, a method of specifying the first region by shape matching using the pattern shape of the image in the defect region, or the like may be used. Which method should be adopted may be determined based on the image quality of the depth image, the size of the defect area (the number of pixels), and the like.

In the above embodiment, the case of acquiring the RGB image and the depth image from one camera 12 has been described, but the present invention is not limited to this. For example, the depth image may be acquired from the three-dimensional sensor, and the RGB image may be acquired from an RGB camera capable of separately capturing the region corresponding to the depth image obtained from the three-dimensional sensor.

In the above embodiment, the case where the depth image is a depth image used for picking up a product by the robot 14 has been described, but the present invention is not limited to this, and other depth images may be used.

In the above embodiment, the case where the information processing device 10 controls the operation of the robot 14 has been described, but the present invention is not limited to this. That is, the information processing device 10 may only transmit the RGB image and the complemented depth image to the robot 14. In this case, the control device included in the robot 14 may control the robot 14 by using the RGB image and the depth image after complementation.

In the above embodiment, an example of obtaining the pixel value of the non-measurable pixel by using the above equations (1) to (3) has been described, but the present invention is not limited to this, and is included in the third region by other methods. The pixel value of the non-measurable pixel may be obtained based on the pixel value.

In the above embodiment, the case where the RGB image is used as the two-dimensional image has been described, but the present invention is not limited to this, and a gray image may be used as the two-dimensional image. In this case, if the camera is capable of capturing a gray image, the captured gray image may be used, and if the camera is capable of capturing an RGB image, the gray converted from the RGB image may be used. Images may be used.

In the above embodiment, the case where the information processing device 10 has the function of the robot control unit 30 has been described, but the present invention is not limited to this. That is, a robot control device that controls the robot 14 may be prepared separately from the information processing device 10 so that the robot control device has the function of the robot control unit 30.

The above processing function can be realized by a computer. In that case, a program that describes the processing content of the function that the processing device should have is provided. By executing the program on a computer, the above processing function is realized on the computer. The program describing the processing content can be recorded on a computer-readable storage medium (excluding the carrier wave).

When a program is distributed, it is sold in the form of a portable storage medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disc Read Only Memory) on which the program is recorded. It is also possible to store the program in the storage device of the server computer and transfer the program from the server computer to another computer via the network.

The computer that executes the program stores, for example, the program recorded on the portable storage medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes the processing according to the program. The computer can also read the program directly from the portable storage medium and execute the process according to the program. In addition, the computer can sequentially execute processing according to the received program each time the program is transferred from the server computer.

The above-described embodiment is an example of a preferred embodiment of the present invention. However, the present invention is not limited to this, and various modifications can be made without departing from the gist of the present invention.

Regarding the description of the above embodiments, the following additional notes will be further disclosed.
(Appendix 1) An acquisition unit that acquires a depth image and a two-dimensional image of the place where the product is placed, and
A first extraction unit that extracts a first region corresponding to the defect region from the two-dimensional image when a defect region including a pixel whose depth cannot be measured exists in a part of the depth image.
A second extraction unit that extracts a second region having a degree of similarity to the first region or more from the two-dimensional image,
A determination unit that determines the pixel value of a pixel whose depth is included in the defect region of the depth image and whose depth cannot be measured, based on the pixel value of the pixel included in the third region of the depth image corresponding to the second region.
Information processing device equipped with.
(Appendix 2) The information processing apparatus according to Appendix 1, wherein the determination unit determines the pixel value of each pixel included in the third region as the pixel value of each pixel included in the defect region. ..
(Appendix 3) The determination unit identifies a pixel whose depth in the defect region can be measured, and the pixel value of the specified pixel and the pixel value of the pixel in the third region corresponding to the specified pixel. The pixel value of the pixel whose depth in the defect region cannot be measured is determined based on the difference from the above and the pixel value of the pixel in the third region corresponding to the pixel whose depth in the defect region cannot be measured. The information processing apparatus according to Appendix 1, wherein the information processing device is characterized by the above.
(Appendix 4) The determination unit is characterized in that, among the pixels whose depth in the defect region can be measured, the pixel obtained by photographing a pixel other than the mounting surface on which the product is mounted is specified. The information processing device described.
(Appendix 5) Obtain a depth image and a two-dimensional image of the place where the product is placed.
When a defect region including pixels whose depth cannot be measured exists in a part of the depth image, a first region corresponding to the defect region is extracted from the two-dimensional image.
A second region having a degree of similarity to the first region or more is extracted from the two-dimensional image.
Based on the pixel value of the pixel included in the third region of the depth image corresponding to the second region, the pixel value of the pixel whose depth included in the defect region of the depth image cannot be measured is determined.
An image processing program that allows a computer to perform processing.
(Appendix 6) In the process to be determined,
The image processing program according to Appendix 5, wherein the pixel value of each pixel included in the third region is determined as the pixel value of each pixel included in the defect region.
(Appendix 7) In the process to be determined,
The pixel whose depth in the defect region can be measured is specified, the difference between the pixel value of the specified pixel and the pixel value of the pixel in the third region corresponding to the specified pixel, and the defect region. The pixel value of the pixel in the third region corresponding to the pixel whose depth cannot be measured is determined, and the pixel value of the pixel whose depth in the defect region cannot be measured is determined. Image processing program.
(Appendix 8) In the process to be determined,
Among the pixels whose depth in the defect region can be measured, the pixels obtained by photographing the pixels other than the mounting surface on which the product is mounted are specified.
The image processing program according to Appendix 7, characterized by the above.
(Appendix 9) Obtain a depth image and a two-dimensional image of the place where the product is placed.
When a defect region including pixels whose depth cannot be measured exists in a part of the depth image, a first region corresponding to the defect region is extracted from the two-dimensional image.
A second region having a degree of similarity to the first region or more is extracted from the two-dimensional image.
Based on the pixel value of the pixel included in the third region of the depth image corresponding to the second region, the pixel value of the pixel whose depth included in the defect region of the depth image cannot be measured is determined.
An image processing method characterized in that processing is performed by a computer.

10 Information processing device 20 RGB image acquisition unit (part of acquisition unit)
22 Depth image acquisition section (part of the acquisition section)
26 Corresponding area extraction unit (first extraction unit, second extraction unit)
28 Complementary part (decision part)

Claims (6)

An acquisition unit that acquires a depth image and a two-dimensional image of the place where the product is placed,
A first extraction unit that extracts a first region corresponding to the defect region from the two-dimensional image when a defect region including a pixel whose depth cannot be measured exists in a part of the depth image.
A second extraction unit that extracts a second region having a degree of similarity to the first region or more from the two-dimensional image,
A determination unit that determines the pixel value of a pixel whose depth is included in the defect region of the depth image and whose depth cannot be measured, based on the pixel value of the pixel included in the third region of the depth image corresponding to the second region.
Information processing device equipped with. The information processing apparatus according to claim 1, wherein the determination unit determines a pixel value of each pixel included in the third region as a pixel value of each pixel included in the defect region. The determination unit identifies a pixel whose depth in the defect region can be measured, and the difference between the pixel value of the specified pixel and the pixel value of the pixel in the third region corresponding to the specified pixel. The pixel value of the pixel whose depth in the defect region cannot be measured is determined based on the pixel value of the pixel in the third region corresponding to the pixel whose depth in the defect region cannot be measured. The information processing apparatus according to claim 1. The information according to claim 3, wherein the determination unit identifies a pixel obtained by photographing a pixel other than the mounting surface on which the product is mounted, among the pixels whose depth in the defect region can be measured. Processing equipment. Obtain a depth image and a two-dimensional image of the place where the product is placed,
When a defect region including pixels whose depth cannot be measured exists in a part of the depth image, a first region corresponding to the defect region is extracted from the two-dimensional image.
A second region having a degree of similarity to the first region or more is extracted from the two-dimensional image.
Based on the pixel value of the pixel included in the third region of the depth image corresponding to the second region, the pixel value of the pixel whose depth included in the defect region of the depth image cannot be measured is determined.
An image processing program that allows a computer to perform processing. Obtain a depth image and a two-dimensional image of the place where the product is placed,
When a defect region including pixels whose depth cannot be measured exists in a part of the depth image, a first region corresponding to the defect region is extracted from the two-dimensional image.
A second region having a degree of similarity to the first region or more is extracted from the two-dimensional image.
Based on the pixel value of the pixel included in the third region of the depth image corresponding to the second region, the pixel value of the pixel whose depth included in the defect region of the depth image cannot be measured is determined.
An image processing method characterized in that processing is performed by a computer.

Images