JP2016075658A - Information process system and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information process system and an information processing method that can suppress a decrease in recognition precision of a subject.SOLUTION: A pattern projection part projects pattern light on a subject. Imaging parts 3, 4 image the subject respectively and generate luminance images. A cost calculation part 53 calculates a cost value of a candidate pixel as a candidate for a corresponding pixel. A composite cost calculation part 61 aggregates cost values to calculate a route cost value, and calculates a composite cost value. A subpixel estimation part 62 executes subpixel estimation on the basis of a shift quantity corresponding to a minimum value of the composite cost value and a composite cost value at a shift quantity adjoining thereto. A generation part 63 generates a parallax image on the basis of parallax values in subpixel units. An analysis part analyzes whether a luminance image or parallax image has image abnormality. A dimming control part executes dimming control over the pattern light.SELECTED DRAWING: Figure 11

Description

  The present invention relates to an information processing system and an information processing method.

  In recent years, as a technique for measuring a distance from an object, a stereo camera having two cameras is used, and a stereo matching process using the principle of triangulation is used. Stereo matching processing is to obtain a parallax value by matching a corresponding area between a reference image captured by one camera and a comparison image captured by the other camera, and from the parallax value, a stereo camera, This is a process for calculating the distance to an object included in an image.

  However, if the texture is weak in an image obtained by imaging an object that is a subject, it is difficult to appropriately match the corresponding region when the stereo matching process described above is performed, and it is difficult to derive an accurate parallax value. Here, the texture refers to, for example, a pattern, a pattern, a pattern, a color, or a dot that appears depending on the brightness of each pixel of the image.

  Therefore, a technique has been proposed in which a texture is formed by irradiating light having a pattern (hereinafter referred to as pattern light) from a light projecting unit, and a stereo matching process is performed on an image obtained by imaging a subject on which the texture is formed. (Patent Document 1). Here, the pattern light pattern means a light intensity distribution, a color distribution, or the like for creating the texture described above. As described above, the accuracy of deriving the parallax value is improved by performing the stereo matching process on the texture-formed image.

  However, in the technique described in Patent Document 1, for example, depending on the installation state of the subject, an image abnormality such as a blackened state in the captured image, a saturated state due to whiteout, or a reflection of light from the light projecting unit has occurred. In some cases, the recognition accuracy of the subject may be affected.

  The present invention has been made in view of the above, and an object thereof is to provide an information processing system and an information processing method capable of suppressing a decrease in recognition accuracy of a subject.

  In order to solve the above-described problems and achieve the object, the present invention includes a light projecting unit that irradiates a subject with pattern light having a specific pattern to form a texture on the subject, and the texture is formed. An imaging unit that images the subject, a derivation unit that derives distance information to the subject based on a captured image captured by the imaging unit, and at least one of the captured image or the image based on the distance information An analysis means for analyzing presence / absence of an image abnormality and a dimming means for executing dimming control when the analysis means analyzes that the image abnormality exists are provided.

  According to the present invention, a highly accurate parallax value can be derived.

FIG. 1 is a diagram for explaining the principle of calculating the distance from an imaging device to an object. FIG. 2 is a conceptual diagram illustrating a reference image, a high-density parallax image, and an edge parallax image. FIG. 3 is an explanatory diagram for obtaining a corresponding pixel in a comparison image corresponding to a reference pixel in the reference image. FIG. 4 is a graph showing an example of the relationship between the shift amount and the cost value. FIG. 5 is a conceptual diagram for calculating the synthesis cost. FIG. 6 is a graph showing an example of the relationship between the shift amount and the synthesis cost value. FIG. 7 is a diagram illustrating an example of the overall configuration of the transport system according to the present embodiment. FIG. 8 is a diagram illustrating an example of a hardware configuration of the disparity value deriving system according to the present embodiment. FIG. 9 is a diagram illustrating an example of reflection of pattern light in an image. FIG. 10 is a diagram illustrating an example of discontinuity of parallax values caused by reflection. FIG. 11 is a diagram illustrating an example of a functional block configuration of the stereo camera according to the present embodiment. FIG. 12 is a diagram illustrating the parallax of the subject and the parallax of the background in the transport system according to the present embodiment. FIG. 13 is a diagram illustrating an example of the relationship between the shift amount and the synthesis cost value in the present embodiment. FIG. 14 is a diagram for explaining a graph of the synthesis cost value when the pattern light is irradiated and when the pattern light is not irradiated. FIG. 15 is a diagram for explaining subpixel estimation by parabolic fitting. FIG. 16 is a diagram for explaining subpixel estimation by the method of least squares. FIG. 17 is a flowchart illustrating an example of the operation of the transport system according to the present embodiment. FIG. 18 is a flowchart illustrating an example of the operation of the light control according to the present embodiment. FIG. 19 is an example of an external perspective view of the stereo camera of the present embodiment. FIG. 20 is an example of a plan view and a bottom view of the stereo camera of this embodiment. FIG. 21 is an example of a front view, a rear view, and a side view of the stereo camera of this embodiment. FIG. 22 is a diagram illustrating an example of a hardware configuration of the disparity value deriving system according to the first modification of the present embodiment. FIG. 23 is a diagram illustrating an example of a functional block configuration of a stereo camera according to the second modification of the present embodiment. FIG. 24 is an explanatory diagram for obtaining corresponding pixels in the comparison image corresponding to the reference pixels in the reference image of the disparity value deriving system according to Modification 3 of the present embodiment. FIG. 25 is a diagram illustrating an example of a component information selection screen in the disparity value deriving system according to the fourth modification of the present embodiment. FIG. 26 is an explanatory diagram for obtaining a corresponding pixel in a comparison image corresponding to a reference pixel in a reference image of the disparity value deriving system according to the fourth modification of the present embodiment. FIG. 27 is a diagram illustrating an example of a hardware configuration of the disparity value deriving system according to the fifth modification of the present embodiment. FIG. 28 is a flowchart illustrating an example of the operation of the transport system according to the fifth modification of the present embodiment.

[Outline of Ranging Method Using SGM Method]
First, an outline of a distance measuring method using an SGM (Semi-Global Matching) method will be described with reference to FIGS. 1 to 6 before a specific description of the present embodiment.

(Principles of ranging)
FIG. 1 is a diagram for explaining the principle of calculating the distance from an imaging device to an object. With reference to FIG. 1, the principle of deriving a parallax value for an object in a stereo camera by stereo matching processing and measuring the distance from the stereo camera to the object using this parallax value will be described. In the following, in order to simplify the description, an example of matching in units of pixels will be described instead of matching a predetermined region composed of a plurality of pixels.

  The imaging system shown in FIG. 1 has an imaging device 510a and an imaging device 510b arranged in parallel equiposition. The imaging devices 510a and 510b respectively include lenses 511a and 511b that refract incident light and form an image of an object on an image sensor (not shown) that is a solid-state imaging device. The images captured by the imaging device 510a and the imaging device 510b are referred to as a comparison image Ia and a reference image Ib, respectively. In FIG. 1, a point S on the object E in the three-dimensional space is mapped to a position on a straight line parallel to a straight line connecting the lens 511a and the lens 511b in each of the comparison image Ia and the reference image Ib. Here, the point S mapped to each image is a point Sa (x, y) in the comparative image Ia and a point Sb (X, y) in the reference image Ib. At this time, the parallax value dp is expressed by the following equation (1) using the coordinate values of the point Sa (x, y) at the coordinates on the comparison image Ia and the point Sb (X, y) at the coordinates on the reference image Ib. It is expressed as

  dp = X−x (1)

  In FIG. 1, the distance between the point Sa (x, y) in the comparative image Ia and the intersection of the perpendicular line taken from the imaging lens 511a on the imaging surface is Δa, and the point Sb (X, y) in the reference image Ib is If the distance from the intersection of the perpendicular line taken from the imaging lens 511b on the imaging surface is Δb, the parallax value dp can also be expressed as dp = Δa + Δb.

  Next, a distance Z between the imaging devices 510a and 510b and the object E is derived by using the parallax value dp. Here, the distance Z is a distance from a straight line connecting the focal position of the lens 511a and the focal position of the lens 511b to the point S on the object E. As shown in FIG. 1, using the focal length f of the imaging lens 511a and the imaging lens 511b, the baseline length B that is the length between the lens 511a and the lens 511b, and the parallax value dp, the following equation (2) Thus, the distance Z can be calculated.

  Z = (B × f) / dp (2)

  From this equation (2), it can be seen that the larger the parallax value dp, the smaller the distance Z, and the smaller the parallax value dp, the larger the distance Z.

(SGM method)
Next, a distance measuring method using the SGM method will be described with reference to FIGS.

  FIG. 2 is a conceptual diagram illustrating a reference image, a high-density parallax image, and an edge parallax image. 2A shows a reference image, FIG. 2B shows a high-density parallax image obtained using the reference image shown in FIG. 2A, and FIG. FIG. 3 is a conceptual diagram showing an edge parallax image obtained using the reference image shown in FIG. Here, the reference image corresponds to the reference image Ib shown in FIG. 1 and is an image in which the captured subject is represented by a luminance value. Further, the high-density parallax image indicates an image in which each pixel of the reference image is represented by a parallax value corresponding to each pixel in the reference image derived by the SGM method. The edge parallax image indicates an image in which each pixel of the reference image is represented by a parallax value corresponding to each pixel in the reference image derived by the block matching method. However, as will be described later, the disparity value that can be derived by the block matching method is a relatively textured portion such as an edge portion in the reference image, and when the disparity value cannot be derived such as a weak texture portion, for example, An image is configured with a parallax value of “0”.

  The SGM method is a method for appropriately deriving a parallax value even for a portion having a weak texture in an image. Based on the reference image shown in FIG. 2A, the high-density parallax shown in FIG. A method for generating an image. The block matching method is a method for deriving the edge parallax image shown in FIG. 2C based on the reference image shown in FIG. According to the SGM method, as can be seen by comparing the inside of the dashed ellipse in FIGS. 2B and 2C, the high-density parallax image is detailed even on roads where the texture is weaker than the edge parallax image. Since the distance based on the parallax value can be expressed, more detailed distance measurement can be performed.

  In the SGM method, the cost value as the degree of coincidence on the comparison image with respect to the reference image is not calculated and the parallax value is not derived immediately, but the parallax value is derived by calculating the combined cost value after calculating the cost value. It is a method to do. The SGM method finally derives a parallax image (here, a high-density parallax image) represented by parallax values corresponding to almost all pixels in the reference image.

  In the case of the block matching method, the cost value is calculated in the same way as the SGM method. However, as in the SGM method, a comparatively strong texture such as an edge portion is calculated without calculating a synthesis cost value. Only the parallax value is derived.

<Calculation of cost value>
FIG. 3 is an explanatory diagram for obtaining a corresponding pixel in a comparison image corresponding to a reference pixel in the reference image. FIG. 4 is a graph showing an example of the relationship between the shift amount and the cost value. A method for calculating the cost value C (p, d) will be described with reference to FIGS. In the following description, C (p, d) represents C (x, y, d).

  3A is a conceptual diagram showing reference pixels in the reference image, and FIG. 3B is a corresponding pixel candidate in the comparison image corresponding to the reference pixel shown in FIG. It is a conceptual diagram at the time of calculating a cost value while shifting (shifting) sequentially. Here, the corresponding pixel indicates a pixel in the comparison image that is most similar to the reference pixel in the reference image. The cost value is an evaluation value that represents the degree of coincidence of each pixel in the comparison image with respect to the reference pixel in the reference image. The cost value (and synthesis cost value) shown below indicates that the smaller the value is, the more similar the pixel in the comparison image is to the reference pixel.

  As shown in FIG. 3A, a candidate that is a candidate for a reference pixel p (x, y) in the reference image Ib and a corresponding pixel on the epipolar line EL in the comparison image Ia with respect to the reference pixel p (x, y). Based on each luminance value of the pixel q (x + d, y), the cost value C (p, d) of the candidate pixel q (x + d, y) corresponding to the reference pixel p (x, y) is calculated. d is a shift amount (shift amount) between the reference pixel p and the candidate pixel q, and the shift amount d is shifted in units of pixels. That is, in FIG. 3, the candidate pixel q (x + d, y) and the reference pixel p (x) are sequentially shifted by one pixel within a predetermined shift range (for example, 0 <d <25). A cost value C (p, d) that is the degree of coincidence of the luminance value with x, y) is calculated.

  As described above, since the imaging devices 510a and 510b are arranged in parallel equiposition, the comparison image Ia and the reference image Ib are also in parallel equivalence relations. Therefore, the corresponding pixel in the comparison image Ia corresponding to the reference pixel p in the reference image Ib exists on the epipolar line EL shown as a horizontal line in FIG. 3, and the corresponding pixel in the comparison image Ia is In order to obtain it, the pixel on the epipolar line EL of the comparison image Ia may be searched.

  The cost value C (p, d) calculated in this way is represented by the graph shown in FIG. 4 in relation to the shift amount d. In the example of FIG. 4, the cost value C is “0” when the shift amount d = 5, 12, and 19, and thus the minimum value cannot be obtained. For example, when there is a portion where the texture is weak in the image, it is difficult to obtain the minimum value of the cost value C in this way.

<Calculation of synthetic cost value>
FIG. 5 is a conceptual diagram for calculating the synthesis cost. FIG. 6 is a graph showing an example of the relationship between the shift amount and the synthesis cost value. A method for calculating the synthesis cost value Ls (p, d) will be described with reference to FIGS.

  The calculation of the combined cost value is not only the calculation of the cost value C (p, d), but also the cost value when the pixels around the reference pixel p (x, y) are used as the reference pixel is the reference pixel p (x, d The combined cost value Ls (p, d) is calculated by concentrating on the cost value C (p, d) in y). In order to calculate the combined cost value Ls (p, d), first, the route cost value Lr (p, d) is calculated. The route cost value Lr (p, d) is calculated by the following equation (3).

Lr (p, d) = C (p, d) + min (Lr (p−r, k) + P (d, k))
(3)
(P = 0 (when d = k))
P = P1 (when | d−k | = 1),
P = P2 (> P1) (when | dk |> 1))

  As shown in Expression (3), the route cost value Lr is obtained recursively. Here, r indicates a direction vector in the aggregation direction, and has two components in the x direction and the y direction. min () is a function for obtaining the minimum value. Lr (p−r, k) is the respective path when the shift amount is changed (the shift amount in this case is k) for the pixel having the coordinate shifted by one pixel in the r direction from the coordinate of the reference pixel p. The cost value Lr is shown. Then, based on the relationship between the shift amount d of the route cost value Lr (p, d) and the shift amount k, the value P (d, k) as shown in (a) to (c) below. Lr (p−r, k) + P (d, k) is calculated.

(A) When d = k, P = 0. That is, Lr (p−r, k) + P (d, k) = Lr (p−r, k).
(B) When | d−k | = 1, P = P1. That is, Lr (p−r, k) + P (d, k) = Lr (p−r, k) + P1.
(C) When | d−k |> 1, P = P2 (> P1). That is, Lr (p−r, k) + P (d, k) = Lr (p−r, k) + P2.

  Then, min (Lr (p−r, k) + P (d, k)) is Lr (p−r) calculated in the above (a) to (c) when k is changed to various values. , K) + P (d, k) is a value obtained by extracting the minimum value. In other words, the value P1 or the value P2 (> P1) when the pixel located at the coordinate position of the reference pixel p in the comparison image Ia moves away from the pixel shifted by the shift amount k from the pixel (pr) adjacent in the r direction. ) Is added so that the shift amount d for the pixels away from the coordinates of the reference pixel p in the comparison image Ia is not affected by the discontinuous path cost value Lr. Further, the value P1 and the value P2 are fixed parameters determined in advance by experiments, and are parameters such that the parallax values of the reference pixels adjacent on the path are likely to be continuous. Thus, in order to obtain the path cost value Lr for each pixel in the r direction in the comparison image Ia, first, the path cost value Lr from the extreme end pixel in the r direction from the coordinates of the reference pixel p (x, y). And the route cost value Lr is obtained along the r direction.

Then, as shown in FIG. 5, Lr ₀ , Lr ₄₅ , Lr which are route cost values Lr in eight directions (r ₀ , r ₄₅ , r ₉₀ , r ₁₃₅ , r ₁₈₀ , r ₂₂₅ , r ₂₇₀ and r ₃₁₅ ). ₉₀ , Lr ₁₃₅ , Lr ₁₈₀ , Lr ₂₂₅ , Lr ₂₇₀ and Lr ₃₁₅ are obtained, and finally, a combined cost value Ls (p, d) is obtained based on the following equation (4).

  Ls (p, d) = ΣLr (4)

  As described above, the calculated combined cost value Ls (p, d) can be represented by the graph shown in FIG. 6 in relation to the shift amount d. In the example of FIG. 6, the combined cost value Ls is derived as a parallax value dp = 3 because the minimum value is obtained when the shift amount d = 3. In the above description, the number in the r direction is eight, but the present invention is not limited to this. For example, the eight directions may be further divided into two to be 16 directions, or may be divided into three to be 24 directions. Alternatively, the combined cost value Ls may be calculated by obtaining the route cost value Lr in any one of the eight directions and adding the route cost values Lr.

[Specific Description of this Embodiment]
Hereinafter, this embodiment will be described in detail with reference to FIGS. In the present embodiment, a case where a parallax value deriving system that derives a parallax value by the stereo matching process as described above is mounted on a transport system including a robot arm will be described as an example. It should be noted that the present invention is not limited by the following description of the present embodiment, and constituent elements in the following embodiment can be easily conceived by those skilled in the art, substantially the same, and so-called The equivalent range is included. Furthermore, various omissions, substitutions, changes, and combinations of the constituent elements can be made without departing from the scope of the following embodiments.

(Overall configuration of transfer system)
FIG. 7 is a diagram illustrating an example of the overall configuration of the transport system according to the present embodiment. The overall configuration of the transport system 1 according to the present embodiment will be described with reference to FIG.

  As shown in FIG. 7, the transport system 1 according to the present embodiment is a system that grips a part installed on a table (hereinafter referred to as “background part 21”) by a robot arm and transports it to a transport destination. . As illustrated in FIG. 7, the transport system 1 includes a parallax value derivation system 15 that is an example of an information processing system, a recognition processing unit 7, an arm control unit 10, and an arm 11.

  The parallax value derivation system 15 is a system that generates a parallax image for a subject from luminance images obtained by imaging the subject with two imaging units in a state in which pattern light is irradiated on a component that is the subject. The parallax value deriving system 15 includes a stereo camera 2, a pattern light projecting unit 8 (light projecting unit), and a light control unit 9.

  The stereo camera 2 is a device that generates a parallax image for a subject from a luminance image obtained by imaging a component that is a subject by two imaging units. The stereo camera 2 includes imaging units 3 and 4, a matching unit 5, and a parallax image generation unit 6. The functions of the imaging units 3 and 4, the matching unit 5, and the parallax image generation unit 6 will be described later with reference to FIG.

  The pattern light projecting unit 8 is a device that irradiates the component 20 that is the subject of the imaging units 3 and 4 with pattern light having a special pattern. The special pattern is preferably, for example, a two-dimensional random number pattern or a pattern having a plurality of brightness values. On the other hand, as a special pattern, a pattern having a repetition period is not preferable because the cost value C may be repeated. As described above, the pattern light can be formed on the component 20 by irradiating the component 20 with the pattern light by the pattern projecting unit 8, so that the accuracy of the parallax value derived by the stereo matching process can be improved. Can be improved.

  The dimming control unit 9 analyzes whether the luminance image and the parallax image output from the stereo camera 2 are abnormal, and when detecting that there is an abnormality, the light amount of the pattern light emitted by the pattern light projecting unit 8 The adjustment amount is determined, and the determined adjustment amount is transmitted to the pattern projector 8. For example, the dimming control unit 9 increases the amount of pattern light in order to eliminate the blacked-out state when the luminance image received from the stereo camera 2 detects that there is a blacked-out state region. Is sent to the pattern projector 8. Further, when the dimming control unit 9 detects that there is a saturated region in the luminance image received from the stereo camera 2, the dimming control unit 9 sets an adjustment amount for reducing the amount of pattern light in order to eliminate the saturated state. Transmit to the pattern projector 8. In addition, although the light control part 9 has received the captured image of the imaging part 3 as a luminance image to receive, it is not limited to this, It is good also as what receives the captured image of the imaging part 4. .

  The functions of the stereo camera 2, the pattern light projecting unit 8, and the dimming control unit 9 of the parallax value deriving system 15 will be described in detail with reference to FIGS.

  The recognition processing unit 7 is a device that recognizes the shape, position, distance, and the like of the component 20 captured by the stereo camera 2 based on the luminance image and the parallax image output from the stereo camera 2. The arm control unit 10 is a device that controls the arm 11, such as an articulated robot, to grip the component 20 and convey it to the conveyance destination based on the recognition result of the component 20 by the recognition processing unit 7.

(Hardware configuration of disparity value derivation system)
FIG. 8 is a diagram illustrating an example of a hardware configuration of the disparity value deriving system according to the present embodiment. FIG. 9 is a diagram illustrating an example of reflection of pattern light in an image. FIG. 10 is a diagram illustrating an example of discontinuity of parallax values caused by reflection. A hardware configuration of the disparity value deriving system 15 according to the present embodiment will be mainly described with reference to FIGS.

  As shown in FIG. 8, the stereo camera 2 of the parallax value derivation system 15 includes a CPU (Central Processing Unit) 100, a ROM (Read Only Memory) 101, a ROM I / F 102, a RAM (Random Access Memory) 103, A RAM I / F 104, an image processing unit 105, an imaging unit 106 (imaging unit), and an imaging unit control I / F 107 are provided.

  The CPU 100 is an arithmetic device that controls each function of the stereo camera 2. The ROM 101 is a non-volatile storage device that stores a program executed by the CPU 100 to control each function of the stereo camera 2. The ROM I / F 102 is an interface for connecting the CPU 100 and the ROM 101 for realizing read and write operations to the ROM 101 by instructions of the CPU 100.

  The RAM 103 is a volatile storage device that functions as a work memory of the CPU 100 and a buffer for a captured image input from the image processing unit 105 via the RAM I / F 104. The RAM I / F 104 is an interface that connects the CPU 100 and the RAM 103 for realizing read and write operations to the RAM 103 by instructions of the CPU 100.

  The image processing unit 105 is a hardware circuit such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) that generates a parallax image from a captured image (luminance image) captured by the imaging unit 106.

  Although only one image pickup unit 106 is shown in FIG. 2 for the sake of brevity, the image pickup unit 106 is composed of two image pickup units. The two image pickup units are fixed by a fixing jig in a state of being separated by a predetermined distance, Each device captures an image of a subject, generates analog image data, and further converts it into digital image data. The imaging unit 106 includes a lens 106a, a diaphragm 106b, and an image sensor 106c.

  The lens 106a is an optical element that refracts incident light and forms an image of an object on the image sensor 106c. The diaphragm 106b is a member that adjusts the amount of light that forms an image on the image sensor 106c by blocking part of the light that has passed through the lens 106a. The image sensor 106c is a solid-state imaging device that converts light incident on the lens 106a and passing through the aperture 106b into electrical analog image data. The image sensor 106c is realized by, for example, a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor).

  The imaging unit control I / F 107 relays register setting values for controlling the exposure time of the imaging unit 106 from the dimming control unit 9 to the imaging unit 106, or conversely, register setting values from the imaging unit 106. Is an interface that relays The imaging unit control I / F 107 is, for example, an interface compliant with Ethernet (registered trademark).

  As shown in FIG. 8, the dimming control unit 9 of the parallax value deriving system 15 includes an analyzing unit 200 (analyzing unit) and a dimming unit 201 (dimming unit).

  The analysis unit 200 is a circuit that analyzes whether the luminance image or the parallax image received from the image processing unit 105 of the stereo camera 2 has an image abnormality. For example, the analysis unit 200 detects the presence or absence of a blackened state or a saturated region in the subject or the background portion around the subject from the received luminance image. Further, the analysis unit 200 detects from the received parallax image whether or not there is a discontinuous region of the parallax value associated with the reflection of the pattern light emitted from the pattern light projecting unit 8 in the subject or background. Here, in FIG. 9, for example, when the analysis unit 200 receives the reference image Ib_ex as the luminance image from the image processing unit 105, the reflection is reflected on the background region 400 around the component reflected in the reference image Ib_ex. The example of the state currently imaged is shown. As described above, since the imaging unit 106 is separated from the imaging unit by a predetermined distance, the reflection of the region 400 of the reference image Ib_ex illustrated in FIG. 9 is captured at a different position in the comparison image. Become. As a result, in the parallax image Ip_ex shown in FIG. 10 generated by the image processing unit 105, discontinuous parallax values are generated in the background area 401. Note that the cause of discontinuous parallax values in the parallax image is not limited to the fact that the reflection of the pattern light is reflected in the luminance image as described above.

  Further, as will be described later, the analysis unit 200 determines whether the texture is properly formed in the luminance image received from the image processing unit 105 of the stereo camera 2 as a result of the light control by the light control unit 201, that is, the texture. Analyzes whether or not the image is blurred. In this case, for example, the analysis unit 200 calculates the sharpness of the texture with respect to the luminance image, and determines that the texture is properly formed when the texture is equal to or greater than the predetermined threshold. do it.

  The light control unit 201 receives the analysis result from the analysis unit 200, and from the analysis result, a blackened state or a saturated state occurs in the luminance image, or a discontinuous region of the parallax value occurs in the parallax image. This is a circuit that transmits an adjustment amount according to the content of the image abnormality to the pattern projector 8 when it is determined that the image abnormality is occurring. Specifically, the dimming unit 201 associates the content of the image abnormality (black crushing state, saturation state, disparity value discontinuity, etc.) with the adjustment amount for the light amount of the pattern light projecting unit 8. An adjustment amount corresponding to the content of the image abnormality is determined from (not shown) and transmitted to the pattern projector 8. The light adjustment unit 201 determines the adjustment amount from the content of the image abnormality by the light amount adjustment table based on the light amount adjustment data, but is not limited to the table format, and the content of the image abnormality and the adjustment. Any form of information may be used as long as the information is associated with the quantity. The dimming unit 201 performs dimming of the pattern light by transmitting the adjustment amount to the pattern projecting unit 8, and then receives the light amount data from the pattern projecting unit 8.

  The light control unit 201 has reached the light control limit from the light amount data received from the pattern light projecting unit 8, that is, when it is determined that light control cannot be performed any more, or from the analysis result received from the analysis unit 200, When it is determined that an image abnormality in which the texture is not properly formed in the luminance image has occurred, exposure adjustment is executed. Specifically, the light control unit 201 transmits adjustment instruction information for adjusting the exposure time to the imaging unit 106 as exposure adjustment. The imaging unit 106 adjusts the exposure time in imaging according to the adjustment instruction information received from the light control unit 201.

  In addition, the dimming unit 201 performs the dimming operation on the pattern light projecting unit 8 and the imaging unit 106 when the analysis result received from the analyzing unit 200 indicates that there is no image abnormality in the luminance image and the parallax image. The exposure adjustment operation for is stopped. However, the analysis unit 200 continues the analysis of whether there is an image abnormality in the luminance image and the parallax image, and the dimming unit 201 receives the analysis result from the analysis unit 200 and receives from the pattern light projecting unit 8. Continue receiving light intensity data.

  In addition, although the light control part 201 adjusts the exposure time with respect to the imaging part 106 as exposure adjustment, it is not limited to this. For example, the light control unit 201 simply adds a plurality of frames obtained by changing a plurality of types of exposure in units of captured images (frames) of the imaging unit 106 as exposure adjustment, or performs pixel processing such as gamma processing. It is also possible to execute a dynamic range expansion process to be added after performing value conversion or the like.

  As shown in FIG. 8, the pattern light projecting unit 8 of the parallax value deriving system 15 includes a light amount adjusting unit 300, a light source unit 301, a texture filter 302, and a lens 303.

  The light amount adjustment unit 300 is a device that adjusts the amount of light emitted from the light source unit 301 using the adjustment amount received from the light control unit 201. The light amount adjustment unit 300 adjusts the light amount of the light from the light source unit 301 using the adjustment amount received from the dimming unit 201, and then transmits the light amount data to the dimming unit 201. The light source unit 301 is a light source that emits light with the light amount adjusted by the light amount adjusting unit 300. The texture filter 302 is a filter for causing the light emitted from the light source unit 301 to have a special pattern. The light from the light source unit 301 that has passed through the texture filter 302 is directed to the lens 303 as pattern light. The lens 303 is an optical element that refracts the pattern light so that the pattern light transmitted through the texture filter 302 is diffused and applied to the subject.

  In addition, although the analysis part 200, the light control part 201, and the light quantity adjustment part 300 shall be implement | achieved by the hardware circuit, it is not limited to this. That is, at least one of the analysis unit 200, the light control unit 201, and the light amount adjustment unit 300 may be realized by a program that is software executed by a CPU or the like.

  Further, the hardware configurations of the analysis unit 200, the light control unit 201, and the light amount adjustment unit 300 illustrated in FIG. 8 are merely examples, and are not limited to such hardware configurations.

(Function block configuration of stereo camera and operation of each function block)
FIG. 11 is a diagram illustrating an example of a functional block configuration of the stereo camera according to the present embodiment. FIG. 12 is a diagram illustrating the parallax of the subject and the parallax of the background in the transport system according to the present embodiment. FIG. 13 is a diagram illustrating an example of the relationship between the shift amount and the synthesis cost value in the present embodiment. FIG. 14 is a diagram for explaining a graph of the synthesis cost value when the pattern light is irradiated and when the pattern light is not irradiated. FIG. 15 is a diagram for explaining subpixel estimation by parabolic fitting. FIG. 16 is a diagram for explaining subpixel estimation by the method of least squares. With reference to FIGS. 11 to 16, the functional block configuration of the stereo camera 2 according to the present embodiment and the operation of each functional block will be described.

  As shown in FIG. 11, the stereo camera 2 includes imaging units 3 and 4, a matching unit 5, and a parallax image generation unit 6.

  In addition, the imaging units 3 and 4 are fixed in a state where they are separated by a predetermined distance, and each captures an image of a subject, generates analog image data, and further converts it into digital image data to generate a luminance image. It is a functional part to do. The imaging units 3 and 4 correspond to the imaging unit 106 shown in FIG.

  The matching unit 5 uses the luminance images generated by the imaging units 3 and 4 as the reference image and the comparison image, respectively, and calculates a cost value C that is the degree of coincidence of each pixel in the comparison image with respect to the reference pixel of the reference image. Part. The matching unit 5 includes filter units 51 and 52 and a cost calculation unit 53 (first calculation means). The matching unit 5 is realized by the image processing unit 105 shown in FIG. Note that the luminance image generated by the imaging unit 3 may be used as a comparative image, and the luminance image generated by the imaging unit 4 may be used as a reference image.

  The parallax image generation unit 6 is a functional unit that generates a parallax image using the cost value C received from the matching unit 5. The parallax image generation unit 6 includes a synthesis cost calculation unit 61 (second calculation unit), a subpixel estimation unit 62 (derivation unit), and a generation unit 63 (generation unit). The parallax image generation unit 6 is realized by the image processing unit 105 illustrated in FIG.

  The filter units 51 and 52 are functional units that remove noise from the luminance images generated by the imaging units 3 and 4, respectively. Here, the luminance image from which noise has been removed by the filter unit 51 is referred to as a reference image Ib, and the luminance image from which noise has been removed by the filter unit 52 is referred to as a comparative image Ia.

  The cost calculation unit 53 performs the reference on the epipolar line in the comparison image Ia based on the luminance value of the reference pixel p (x, y) (first reference region) in the reference image Ib and the reference pixel p (x, y). Based on each luminance value of the candidate pixel q (x + d, y), which is a candidate for the corresponding pixel (corresponding region), specified by shifting from the pixel corresponding to the position of the pixel p (x, y) by the shift amount d. The functional unit calculates the cost value C (p, d) (degree of coincidence) of each candidate pixel q (x + d, y). As a method for calculating the cost value C by the cost calculation unit 53, for example, SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), or NCC (Normalized Cross-Correlation) may be used.

  Further, in the case of the transportation of the component 20 in the transportation system 1 shown in FIG. 7, the distance from the stereo camera 2 to the background portion 21 is fixed as shown in FIG. The parallax value dp21 of the point 21b of the background region 21a in the reference image Ib and the comparison image Ia obtained by capturing 21 is known information in advance. Therefore, the cost calculation unit 53 can determine a search range of the candidate pixel q (x + d, y) based on the shift amount d to a predetermined finite range with respect to the reference pixel p (x, y). It is possible to calculate the cost value C within the search range. Thus, by determining the search range of the candidate pixel q in advance, the parallax value dp20 of the component area 20a in the reference image Ib and the comparison image Ia can be derived with high accuracy, and erroneous detection of the disparity value dp20 that is far away can be performed. In addition, the calculation load on the cost calculation unit 53 can be reduced.

  Further, an example of a graph showing the relationship between the shift amount d and the cost value C calculated by the cost calculation unit 53 is the graph shown in FIG. 4 described above. In the graph shown in FIG. 4, since the shift value d = 5, 12, 19 is an approximate value as the minimum value of the cost value C, by obtaining the minimum value of the cost value C, the reference image Ib It is difficult to obtain a corresponding pixel in the comparison image Ia corresponding to the reference pixel.

  In order to calculate the combined cost value Ls (p, d) (degree of coincidence), the combined cost calculation unit 61 firstly calculates the route cost value Lr (p, d) in a predetermined r direction according to the above equation (3). Is a functional unit for calculating In the path cost value Lr (p, d), the pixel adjacent in the r direction of the reference pixel p (x, y) in the reference image Ib is set as the reference pixel (second reference region) as shown in the equation (3). The cost value C in the comparison image Ia for the reference pixel in the case is a value calculated by aggregating the cost value C (p, d) of the candidate pixel q (x + d, y) calculated by the cost calculation unit 53. is there.

As illustrated in FIG. 5, the synthesis cost calculation unit 61 performs Lr which is a route cost value Lr in eight directions (r ₀ , r ₄₅ , r ₉₀ , r ₁₃₅ , r ₁₈₀ , r ₂₂₅ , r _270, and r ₃₁₅ ). ₀ , Lr ₄₅ , Lr ₉₀ , Lr ₁₃₅ , Lr ₁₈₀ , Lr ₂₂₅ , Lr ₂₇₀ and Lr ₃₁₅ are calculated, and finally the synthesis cost value Ls (p, d) is calculated based on the above-described equation (4). To do. And the graph which shows the relationship between the shift amount d and the synthetic | combination cost value Ls calculated by the synthetic | combination cost calculation part 61 is a graph shown in FIG. As shown in FIG. 13, the combined cost value Ls becomes the minimum value when the shift amount d = 3. In the above description, the number in the r direction is eight, but the present invention is not limited to this. For example, the eight directions may be further divided into two to be 16 directions, or may be divided into three to be 24 directions. Or it is good also as what calculates | requires the path | route cost value Lr in any of eight directions, and calculates the synthetic | combination cost value Ls.

  Further, the graph showing the relationship between the shift amount d and the synthesis cost value Ls shown in FIG. 14 is a graph in the case where the pattern projecting unit 8 irradiates the component 20 with pattern light and forms a texture, and the pattern light is irradiated. It is a graph at the time of not forming a texture without doing. In the graph when the texture is not formed in FIG. 14, the parallax value dp1 corresponding to the minimum synthesis cost value Ls is obtained. However, the synthesis cost value Ls other than the synthesis cost value Ls in the parallax value dp1 is also approximated. Therefore, it cannot be said that a clear parallax value can be obtained. On the other hand, in the graph when the texture of FIG. 14 is formed, the contrast of the calculated synthesis cost value Ls becomes large and the minimum synthesis cost value Ls becomes clear, so that the parallax value dp can be obtained with high accuracy. Become.

  The sub-pixel estimation unit 62 calculates the shift amount d corresponding to the minimum value of the combined cost value Ls of the pixels in the comparison image Ia for the reference pixel in the reference image Ib calculated by the combined cost calculating unit 61, and the shift adjacent thereto. Sub-pixel estimation is performed based on the combined cost value Ls in the quantity d. The graph of the synthesis cost value Ls shown in FIG. 13 is a graph of the synthesis cost value Ls with respect to the shift amount d in pixel units. Therefore, the minimum value of the synthesis cost value Ls in the graph shown in FIG. 13 is the synthesis cost value Ls at the shift amount d = 3 in pixel units. That is, in the graph of the synthesis cost value Ls with respect to the shift amount d in pixel units as shown in FIG. 13, it is only possible to derive a value in pixel units as the parallax value dp. Here, sub-pixel estimation is to derive the parallax value dp by estimating the parallax value dp in units smaller than the pixel unit (hereinafter referred to as sub-pixel unit) instead of the pixel unit value. From this parallax value dp, the distance to the subject of the corresponding pixel can be calculated from the above-described equation (2). That is, the parallax value dp is an index value of the distance to the subject.

  First, a case where the subpixel estimation unit 62 performs subpixel estimation by parabolic fitting will be described with reference to FIG. The sub-pixel estimation unit 62 obtains the value of the shift amount d that minimizes the combined cost value Ls in the graph of the combined cost value Ls calculated by the combined cost calculating unit 61 (see FIG. 13). In the example of FIG. 13, the combined cost value Ls is minimum when the shift amount d = 3. Next, the subpixel estimation unit 62 obtains a shift amount d adjacent to the shift amount d = 3. Specifically, the shift amount d = 2,4. Next, in the graph of the shift amount d and the combined cost value Ls shown in FIG. 13, the sub-pixel estimation unit 62 passes through the three points where the shift amount d = 2, 3, and 4 as shown in FIG. A convex quadratic curve is obtained. Then, the subpixel estimation unit 62 estimates that the shift amount d in subpixel units corresponding to the minimum value (extreme value) of the quadratic curve is the parallax value dp.

  Next, a case where the subpixel estimation unit 62 performs subpixel estimation by the least square method will be described with reference to FIG. The sub-pixel estimation unit 62 obtains the value of the shift amount d that minimizes the combined cost value Ls in the graph of the combined cost value Ls calculated by the combined cost calculating unit 61 (see FIG. 13). In the example of FIG. 13, the combined cost value Ls is minimum when the shift amount d = 3. Next, the sub-pixel estimation unit 62 obtains four shift amounts d in the vicinity of the shift amount d = 3. Specifically, the shift amount d = 1, 2, 4, 5. Next, as shown in FIG. 16, in the graph of the shift amount d and the combined cost value Ls shown in FIG. 13, the subpixel estimation unit 62 uses the least square method to shift the amount d = 1, 2, 3, 4, and so on. A convex quadratic curve passing through the vicinity of 5 points, which is 5, is obtained. Then, the subpixel estimation unit 62 estimates that the shift amount d in subpixel units corresponding to the minimum value (extreme value) of the quadratic curve is the parallax value dp.

  The subpixel estimation unit 62 estimates and derives the parallax value dp by either subpixel estimation by parabolic fitting shown in FIG. 15 or subpixel estimation by the least square method shown in FIG. As a result, the parallax value dp can be derived in units of sub-pixels, which is a unit smaller than the pixel unit, so that it is possible to derive a precise and dense parallax value dp.

  The sub-pixel estimation is not limited to the above-described parabolic fitting or the least-square method, and the sub-pixel estimation may be performed by other methods. For example, the sub-pixel estimation unit 62 uses the three points shown in FIG. 15 to perform sub-pixel estimation by equiangular straight fitting that estimates the disparity value dp by obtaining a conformal straight line that passes through the three points instead of the quadratic curve. It may be executed.

  Further, in the subpixel estimation by the least square method, the quadratic curve is obtained using the five points on the graph shown in FIG. 16, but the present invention is not limited to this. A quadratic curve may be obtained.

  The subpixel estimation by the subpixel estimation unit 62 is not limited to calculating the parallax value dp in units of subpixels, and the parallax value dp in units of pixels is calculated without performing subpixel estimation. It may be a thing. In this case, the sub-pixel estimation unit 62 calculates the shift amount d corresponding to the minimum value of the combined cost value Ls of the pixels in the comparison image Ia for the reference pixel in the reference image Ib calculated by the combined cost calculating unit 61 as the parallax value. dp may be used.

  Further, as the cost value C calculated by the cost calculation unit 53 and the synthesis cost value Ls calculated by the synthesis cost calculation unit 61, the smaller the value, the more similar the evaluation value the pixel in the comparison image Ia is to the reference pixel in the reference image Ib. However, the present invention is not limited to this, and a similar evaluation value may be used as the value increases. In this case, in the quadratic curve obtained by the subpixel estimation unit 62 described above, the shift amount d in subpixel units corresponding to the local maximum value may be the parallax value dp.

  The generation unit 63 is an image in which the luminance value of each pixel of the reference image Ib is represented by a parallax value dp corresponding to the pixel based on the sub-pixel unit parallax value dp derived by the sub-pixel estimation unit 62. This is a functional unit that generates a parallax image Ip (high-density parallax image). The parallax image Ip (image based on distance information) may be an image represented by distance information calculated from the parallax value dp instead of the parallax value dp.

  In addition, although the cost calculation part 53, the synthetic | combination cost calculation part 61, the subpixel estimation part 62, and the production | generation part 63 shall be implement | achieved by the hardware circuit, it is not limited to this. That is, at least one of the cost calculation unit 53, the synthesis cost calculation unit 61, the sub-pixel estimation unit 62, and the generation unit 63 may be realized by the CPU 100 executing a program that is software.

  Further, the block configurations of the filter unit 51, the filter unit 52, the cost calculation unit 53, the synthesis cost calculation unit 61, the subpixel estimation unit 62, and the generation unit 63 illustrated in FIG. 11 conceptually illustrate functions. Thus, the present invention is not limited to such a block configuration.

(Overall operation of transport system)
FIG. 17 is a flowchart illustrating an example of the operation of the transport system according to the present embodiment. The overall operation of the transport system 1 including image processing based on the stereo matching processing (SGM method) of the disparity value deriving system 15 according to the present embodiment will be described with reference to FIG.

<Step S11>
As shown in FIG. 1, it is assumed that a component 20 is installed on a background portion 21 that is a table. The part 20 may be assumed to be installed in the background part 21 from the equipment in the previous process of the transport system 1 or may be assumed to be installed in the background part 21 by the operator. Then, the process proceeds to step S12.

<Step S12>
The pattern light projecting unit 8 of the parallax value deriving system 15 irradiates the component 20 installed on the background unit 21 with pattern light having a special pattern. Thereby, a texture is formed on the surfaces of the component 20 and the background portion 21. Then, the process proceeds to step S13.

<Step S13>
The imaging units 3 and 4 of the stereo camera 2 respectively image a subject to generate analog image data, and further convert the image data to digital image data to generate a luminance image. The filter unit 51 of the matching unit 5 of the stereo camera 2 removes noise from the luminance image generated by the imaging unit 3 (imaging unit, first imaging unit) and outputs a reference image Ib. The filter unit 52 of the matching unit 5 of the stereo camera 2 removes noise from the luminance image generated by the imaging unit 4 (imaging unit, second imaging unit), and outputs a comparison image Ia. Then, the process proceeds to step S14.

<Step S14>
The cost calculation unit 53 of the matching unit 5 of the stereo camera 2 performs the luminance value of the reference pixel p (x, y) in the reference image Ib and the epipolar line in the comparison image Ia based on the reference pixel p (x, y). Based on each luminance value of the candidate pixel q (x + d, y), which is a candidate for the corresponding pixel, specified by shifting from the pixel corresponding to the position of the reference pixel p (x, y) by the shift amount d, The cost value C (p, d) of the candidate pixel q (x + d, y) is calculated. As described above, since the distance from the stereo camera 2 to the background portion 21 is fixed, the search range in which the cost calculation unit 53 calculates the cost value C of the candidate pixel q (x + d, y) based on the shift amount d is A predetermined finite range may be used.

When the synthesis cost calculation unit 61 of the parallax image generation unit 6 of the stereo camera 2 uses, as the reference pixel, a pixel adjacent in the r direction of the reference pixel p (x, y) in the reference image Ib according to the above-described equation (3). The cost value C in the comparison image Ia for the reference pixel is calculated by aggregating the cost value C (p, d) of the candidate pixel q (x + d, y) calculated by the cost calculation unit 53. Lr (p, d) is calculated. As illustrated in FIG. 5, the synthesis cost calculation unit 61 performs Lr which is a route cost value Lr in eight directions (r ₀ , r ₄₅ , r ₉₀ , r ₁₃₅ , r ₁₈₀ , r ₂₂₅ , r _270, and r ₃₁₅ ). ₀ , Lr ₄₅ , Lr ₉₀ , Lr ₁₃₅ , Lr ₁₈₀ , Lr ₂₂₅ , Lr ₂₇₀ and Lr ₃₁₅ are calculated, and finally the synthesis cost value Ls (p, d) is calculated based on the above-described equation (4). To do. Then, the process proceeds to step S15.

<Step S15>
The sub-pixel estimation unit 62 of the parallax image generation unit 6 of the stereo camera 2 corresponds to the minimum value of the pixel synthesis cost value Ls in the comparison image Ia calculated by the synthesis cost calculation unit 61 for the reference pixel in the reference image Ib. Sub-pixel estimation is executed based on the combined cost value Ls of the shift amount d to be performed and the shift amount d adjacent thereto. The sub-pixel estimation unit 62 uses the shift amount d in sub-pixel units corresponding to the minimum value of the approximate curve obtained by the sub-pixel estimation (a downwardly convex quadratic curve in FIGS. 15 and 16) as the parallax value dp. And the parallax value dp is derived.

  The generation unit 63 of the parallax image generation unit 6 of the stereo camera 2 corresponds to the luminance value of each pixel of the reference image Ib based on the sub-pixel unit parallax value dp derived by the sub-pixel estimation unit 62. A parallax image Ip (high-density parallax image) that is an image represented by the parallax value dp to be generated is generated. The stereo camera 2 outputs the luminance image output from the imaging unit 3 or the imaging unit 4 and the parallax image Ip generated by the generation unit 63. Then, the process proceeds to step S16.

<Step S16>
The analysis unit 200 of the dimming control unit 9 analyzes whether there is an image abnormality in the luminance image or the parallax image Ip received from the stereo camera 2. As a result of the analysis, if there is an image abnormality (step S16: Yes), the process proceeds to step S17. If there is no image abnormality (step S16: No), the process proceeds to step S18.

<Step S17>
When the analysis unit 200 determines that there is an image abnormality in the luminance image or the parallax image Ip, the dimming control unit 9 performs dimming control on the pattern light that the pattern light projecting unit 8 irradiates the component 20. . Details of the light control will be described later with reference to FIG. After executing the dimming control by the dimming control unit 9, the process returns to step S13.

<Step S18>
When the analysis unit 200 determines that there is no image abnormality in the luminance image or the parallax image Ip, the recognition processing unit 7 is a component imaged by the stereo camera 2 based on the luminance image and the parallax image output from the stereo camera 2. Recognize 20 shapes, positions and distances. Then, the process proceeds to step S19.

<Step S19>
Based on the recognition result of the component 20 by the recognition processing unit 7, the arm control unit 10 controls the arm 11, such as an articulated robot, to grip the component 20 and transport it to the transport destination.

  The overall operation of the transport system 1 including image processing based on the stereo matching processing (SGM method) of the parallax value derivation system 15 is performed by the operation as described above.

(Dimming control)
FIG. 18 is a flowchart illustrating an example of the operation of the light control according to the present embodiment. The dimming control for the pattern light projecting unit 8 by the dimming control unit 9 will be described with reference to FIG.

  As described above, the light adjustment control unit 9 adjusts the pattern light emitted from the pattern light projecting unit 8 to the component 20 when the analysis unit 200 determines that the luminance image or the parallax image Ip has an image abnormality. Perform light control. In this case, as will be described later, the dimming control unit 9 analyzes the presence or absence of an image abnormality in the luminance image received from the stereo camera 2 in steps S171, S172, and S173 shown in FIG. Further, the dimming control unit 9 analyzes the presence / absence of an image abnormality in the parallax image Ip received from the stereo camera 2 in step S174 shown in FIG.

<Step S171>
The analysis unit 200 of the dimming control unit 9 analyzes the presence or absence of a blackened state or a saturated region in the background portion around the subject from the luminance image received from the stereo camera 2. If there is a blackened or saturated region in the background (step S171: Yes), the process proceeds to step S175, and if not (step S171: No), the process proceeds to step S172.

<Step S172>
The analysis unit 200 analyzes the presence or absence of a blackened state or a saturated region in the subject from the luminance image received from the stereo camera 2. When there is a blackened area or a saturated area in the subject (step S172: Yes), the process proceeds to step S175, and when there is no area (step S172: No), the process proceeds to step S173.

<Step S173>
The analysis unit 200 analyzes whether or not the texture is properly formed in the luminance image received from the stereo camera 2, that is, whether or not the texture is blurred. If the texture is blurred (step S173: Yes), the process proceeds to step S177, and if not (step S173: No), the analysis unit 200 determines that there is no image abnormality in the luminance image.

<Step S174>
The analysis unit 200 analyzes from the parallax image Ip received from the stereo camera 2 whether or not there is a discontinuous region of the parallax value associated with the reflection of the pattern light emitted from the pattern light projecting unit 8 in the subject or background. When there is a discontinuous region with a parallax value (step S174: Yes), the process proceeds to step S175, and when there is no disparity value (step S174: No), the analysis unit 200 determines that there is no image abnormality in the parallax image.

<Step S175>
The analysis unit 200 dimmes an analysis result indicating that an image abnormality such as a blackened state or a saturated state occurs in the luminance image or a discontinuous region of the parallax value occurs in the parallax image. Send to part 201. The light control unit 201 transmits an adjustment amount according to the content of the image abnormality indicated by the analysis result received from the analysis unit 200 to the pattern light projecting unit 8. Specifically, the light control unit 201 determines an adjustment amount according to the content of the image abnormality from the light amount adjustment table in which the content of the image abnormality and the adjustment amount for the light amount of the pattern light projecting unit 8 are associated with each other. Transmit to the light projecting unit 8. The light amount adjustment unit 300 of the pattern light projecting unit 8 adjusts the amount of light emitted from the light source unit 301 using the adjustment amount received from the light control unit 201. The light emitted from the light source unit 301 passes through the texture filter 302 and the lens 303 and is irradiated to the subject as pattern light. Then, control goes to a step S176.

<Step S176>
The light control unit 201 performs light control of the pattern light by transmitting the adjustment amount to the light amount adjustment unit 300 of the pattern light projecting unit 8, and then receives light amount data from the light amount adjustment unit 300. The light control unit 201 determines from the light amount data received from the pattern light projecting unit 8 whether or not the light control limit has been reached, that is, whether or not light control can be performed any more. When the light control limit has been reached (step S176: Yes), the process proceeds to step S177. When the light control limit has not been reached (step S176: No), the light control unit 201 can still perform light control on the pattern light projecting unit 8. Recognize.

<Step S177>
The light control unit 201 has reached the light control limit from the light amount data received from the pattern light projecting unit 8, that is, when it is determined that light control cannot be performed any more, or from the analysis result received from the analysis unit 200, When it is determined that an image abnormality in which the texture of the luminance image is blurred has occurred, exposure adjustment is executed. Specifically, the light control unit 201 transmits adjustment instruction information for adjusting the exposure time to the imaging unit 106 as exposure adjustment. The imaging unit 106 adjusts the exposure time in imaging according to the adjustment instruction information received from the light control unit 201.

  In a series of operations as described above, there is no image abnormality in either the luminance image or the parallax image, or an image abnormality occurs in at least one of the luminance image or the parallax image, and the dimming operation is performed. When the light limit is not reached, or when the exposure adjustment operation is executed, the light control is terminated.

  As described above, the pattern projecting unit 8 irradiates the subject with pattern light having a special pattern, and the imaging units 3 and 4 respectively capture the subject and display luminance images (reference image Ib, comparison image). Ia) is generated, and the stereo camera 2 derives the parallax value dp from the reference image Ib and the comparison image Ia by stereo matching processing, and generates the parallax image Ip from the parallax value dp. Then, the analysis unit 200 analyzes whether or not there is an image abnormality for the luminance image or the parallax image Ip, and when it is determined that there is an image abnormality, the dimming control unit 9 performs the pattern irradiated by the pattern light projecting unit 8. Dimming control is performed on the light. As a result, even if an image abnormality occurs in the luminance image obtained by illuminating the pattern light and the parallax image based on the luminance image, the pattern light to be irradiated is again controlled in the state in which the image abnormality is eliminated by dimming control. The subject can be imaged. Therefore, a highly accurate parallax value can be derived by stereo matching processing. Therefore, the recognition accuracy of the subject can be improved.

  Specifically, as the above-described dimming control, the analysis unit 200 detects that there is a blackened state or a saturated region in the luminance image, or the parallax value in the parallax image does not reflect the reflection of pattern light. When it is detected that there is a continuous area, the dimming unit 201 performs a dimming operation on the pattern light projecting unit 8. Further, the analysis unit 200 has a texture on the luminance image when the light control limit is reached from the light amount data received from the pattern light projecting unit 8 as a result of the dimming operation or from the analysis result received from the analysis unit 200. When it is determined that an image abnormality that is not properly formed has occurred, exposure adjustment is executed. Such an operation can eliminate or reduce the image abnormality of the luminance image and the parallax image, and can contribute to the derivation of the parallax value with high accuracy.

  In addition, as described above, the stereo camera 2 can calculate the parallax value with high accuracy even when there is a portion where the texture is weak in the image by calculating the synthesis cost value by the stereo matching process by the SGM method. It is said. In the present embodiment, since the texture is formed on the subject by the pattern light subjected to the light control, the stereo camera 2 does not necessarily need to execute the stereo matching process by the SGM method. Alternatively, a stereo matching process such as normal block matching may be executed.

(Appearance structure of stereo camera)
FIG. 19 is an example of an external perspective view of the stereo camera of the present embodiment. FIG. 20 is an example of a plan view and a bottom view of the stereo camera of this embodiment. FIG. 21 is an example of a front view, a rear view, and a side view of the stereo camera of this embodiment. The external structure of the stereo camera 2 according to the present embodiment will be described with reference to FIGS.

  FIG. 19A is an example of a perspective view from the front of the stereo camera 2. FIG. 19B is an example of a perspective view from the rear of the stereo camera 2. FIG. 20A is an example of a plan view of the stereo camera 2. FIG. 20B is an example of a bottom view of the stereo camera 2. FIG. 21A is an example of a front view of the stereo camera 2. FIG. 21B is an example of a rear view of the stereo camera 2. FIG. 21C is an example of a right side view of the stereo camera 2. FIG. 21D is an example of a left side view of the stereo camera 2. As shown in FIGS. 19 to 21, the stereo camera 2 has, as its external structure, lenses 106a_1 and 106a_2, an imaging unit control I / F 107, a front cover 108, a side cover 109, a rear cover 110, and a power connector. 111 and a back cover 112.

  The lenses 106a_1 and 106a_2 are lenses corresponding to the lens 106a of the imaging unit 106 shown in FIG. 8, and are optical elements for refracting incident light to form an image of an object on the image sensor 106c (see FIG. 8). It is. As shown in FIGS. 19A and 21A, the lenses 106a_1 and 106a_2 are fixed to the front side of the front cover 108 at a predetermined interval.

  The imaging unit control I / F 107 is disposed on the rear cover 110 of the stereo camera 2 and the function thereof is as described above with reference to FIG.

  The front cover 108 is a cover that covers the interior from the top surface of the stereo camera 2 to the front surface (front surface) and part of the rear surface side. As shown in FIG. 19A, the front cover 108 has a shape in which a ridge line formed between the front surface and the upper surface and a ridge line formed between the front surface and the back surface are chamfered.

  The side cover 109 is a cover that covers both side surfaces (right side surface and left side surface) of the stereo camera 2. As shown in FIGS. 20A and 20B, the side cover 109 has a shape in which a ridgeline formed by a side surface, a front surface, and a back surface is chamfered. Further, as shown in FIGS. 21A and 21B, the side cover 109 has a shape in which a ridge formed by a side surface, an upper surface, and a bottom surface is chamfered. Further, the side cover 109 has heat radiation fins 109a. The heat radiation fin 109a is a fin that radiates heat generated by the CPU 100, the image processing unit 105, and the like (see FIG. 8) built in the stereo camera 2 to the outside of the stereo camera 2.

  The rear cover 110 is a cover that covers the rear surface of the stereo camera 2. As shown in FIGS. 19B, 21C, and 21D, the rear cover 110 and the front cover 108 are configured to be chamfered so that no ridge line is formed between the back surface and the top surface. As shown in FIGS. 19 (b), 20 (b), 21 (c) and 21 (d), the rear cover 110 and the back cover 112 are configured to be chamfered so that no ridge line is formed between the back surface and the bottom surface. Has been.

  The power connector 111 is disposed on the rear cover 110 of the stereo camera 2 and is a connector for supplying power to each unit included in the stereo camera 2 shown in FIG.

  The back cover 112 is a cover that covers a portion of the bottom surface of the stereo camera 2 that is not covered by the front cover 108.

  As described above, the stereo camera 2 has a structure in which the ridge lines of the peripheral edges of the front surface (front surface) and the back surface are chamfered over the entire periphery.

(Modification 1)
FIG. 22 is a diagram illustrating an example of a hardware configuration of the disparity value deriving system according to the first modification of the present embodiment. With reference to FIG. 22, the parallax value deriving system 15a according to the present modification will be described focusing on differences from the parallax value deriving system 15 of the above-described embodiment. Note that the stereo camera 2 and the dimming control unit 9 shown in FIG. 22 are the same as those shown in FIG. 8 of the above-described embodiment, so the same reference numerals are given and the description thereof is omitted here.

  As shown in FIG. 22, the parallax value deriving system 15a according to this modification includes a stereo camera 2, a pattern light projecting unit 8a, a dimming control unit 9, and a texture transmitting unit 12 (transmitting means). ing. Among these, the pattern light projecting unit 8a is a device that irradiates the component 20 (see FIG. 7) that is the subject of the imaging units 3 and 4 (see FIG. 7) with pattern light having a special pattern. The pattern light projecting unit 8 a includes a light amount adjusting unit 300 and a projector 304.

  The light amount adjustment unit 300 is a device that adjusts the light amount of light projected from the projector 304 using the adjustment amount received from the light control unit 201. Further, the light amount adjustment unit 300 adjusts the light amount of the light of the projector 304 using the adjustment amount received from the dimming unit 201, and then transmits the light amount data to the dimming unit 201.

  The projector 304 is an apparatus that includes an optical filter, a micromirror device, a lens, a lamp, and the like, and projects pattern light having a special pattern based on the texture data transmitted from the texture transmission unit 12. The texture transmitting unit 12 sends texture data (texture information) for determining a texture pattern, a pattern, a pattern, a color, a dot, or the like formed by irradiating the subject with the pattern light projected from the projector 304. It is a device that transmits.

  As described above, the development cost can be reduced by adopting the ready-made projector 304 as an apparatus for irradiating pattern light. Further, since the projector 304 generates pattern light based on the texture data transmitted from the texture transmission unit 12, the texture formed on the subject can be freely changed. Therefore, it is possible to form a texture on the subject in a manner in which image abnormality such as a blackout state, a saturated state, or disparity discontinuity hardly occurs. Therefore, the recognition accuracy of the subject can be improved.

(Modification 2)
FIG. 23 is a diagram illustrating an example of a functional block configuration of a stereo camera according to the second modification of the present embodiment. With reference to FIG. 23, the stereo camera 2a of the present modification will be described focusing on differences from the stereo camera 2 of the above-described embodiment. Note that the imaging units 3 and 4, the matching unit 5, and the parallax image generation unit 6 illustrated in FIG. 23 are the same as those illustrated in FIG. 11 of the above-described embodiment, and thus are denoted by the same reference numerals. The description of is omitted.

  As shown in FIG. 23, the stereo camera 2a includes an imaging unit 3, 4, a preprocessing unit 13, 14 (first preprocessing unit, second preprocessing unit), a matching unit 5, and parallax image generation. Part 6.

  The preprocessing units 13 and 14 are devices that perform image processing such as distortion correction or dynamic range expansion processing as preprocessing on the luminance images generated by the imaging units 3 and 4, respectively. The preprocessing units 13 and 14 transmit the preprocessed luminance images to the filter units 51 and 52, respectively. In FIG. 23, the preprocessing unit 13 and the preprocessing unit 14 are described as separate functional units, but may be integrated functional units.

  As described above, the preprocessing units 13 and 14 execute the dynamic range expansion process as the preprocessing on the luminance images generated by the imaging units 3 and 4, respectively, and thereby the matching unit 5 and the parallax image generation unit 6. In the step of deriving the parallax value in step S1, it is possible to suppress the occurrence of a black-out state and a saturated state in the reference image Ib or the comparison image Ia. In addition, the preprocessing units 13 and 14 perform distortion correction as preprocessing on the luminance images generated by the imaging units 3 and 4, respectively, thereby affecting the accuracy of calculation of the cost value C by the matching unit 5. It is possible to reduce the distortion state between the reference image Ib and the comparative image Ia. Therefore, the accuracy of calculation of the cost value C by the matching unit 5 can be improved. Therefore, the recognition accuracy of the subject can be improved.

(Modification 3)
FIG. 24 is an explanatory diagram for obtaining corresponding pixels in the comparison image corresponding to the reference pixels in the reference image of the disparity value deriving system according to Modification 3 of the present embodiment. With reference to FIG. 24, the operation of the parallax value derivation system according to the present modification will be described focusing on differences from the parallax value derivation system 15 of the above-described embodiment.

  As shown in FIG. 24, a case is considered where a saturated region 22 is formed by reflection of a component in a component region 20b that is a portion where a component is imaged in the comparison image Ia_2 (or the reference image Ib_2). When a search for calculating the cost value C from the candidate region q1 of the saturated region 22 of the comparative image Ia_2 is performed on the reference region p1 of the saturated region 22 of the reference image Ib_2 (in the direction of the arrow in FIG. 24), the candidate region Since the pixel values of all are saturated, the parallax value of the corresponding region (corresponding region p1a in FIG. 24) corresponding to the reference region p1 cannot be derived. Therefore, the cost calculation unit 53 (see FIG. 11) determines a predetermined range for the search range for calculating the cost value C, and the pixel values (luminance values) can be regarded as being saturated continuously beyond the predetermined range. When the value is equal to or greater than a predetermined value (first value), it is determined that the saturated state is reached in the predetermined range. Next, the cost calculation unit 53 transmits the determination result to the dimming control unit 9, and the analysis unit 200 of the dimming control unit 9 is a saturated region in the luminance image (comparison image Ia_2 (or reference image Ib_2)). Analyzes that there is. When the analysis unit 200 determines that there is an image abnormality in the luminance image, the dimming control unit 9 performs dimming control on the pattern light that the pattern projecting unit 8 irradiates the component.

  As described above, the case where the luminance value is saturated has been described. However, the present invention is not limited to this, and when the luminance value is low, that is, when a so-called blackened state occurs, the dimming control unit 9 The dimming control may be executed as follows. The cost calculation unit 53 (see FIG. 11) determines a predetermined range for the search range for calculating the cost value C, and is a predetermined range in which pixel values (luminance values) can be regarded as a black-out state continuously for a predetermined range or more. When the value is equal to or smaller than the value (second value), it is determined that the black area is in the predetermined range. Here, the second value is smaller than the first value. Next, the cost calculation unit 53 transmits the determination result to the dimming control unit 9, and the analysis unit 200 of the dimming control unit 9 is in a blackened state in the luminance image (the comparative image Ia_2 (or the reference image Ib_2)). Analyzes that there is a region. When the analysis unit 200 determines that there is an image abnormality in the luminance image, the dimming control unit 9 performs dimming control on the pattern light that the pattern projecting unit 8 irradiates the component.

  Thus, since it is possible to determine whether or not the luminance image is saturated in the stage of calculating the cost value C by the cost calculation unit 53, the dimming control is executed at an early timing, and the pattern light projecting unit 8 is informed of the subject. Pattern light that improves recognition accuracy can be irradiated.

(Modification 4)
FIG. 25 is a diagram illustrating an example of a component information selection screen in the disparity value deriving system according to the fourth modification of the present embodiment. FIG. 26 is an explanatory diagram for obtaining a corresponding pixel in a comparison image corresponding to a reference pixel in a reference image of the disparity value deriving system according to the fourth modification of the present embodiment. 25 and 26, the operation of the disparity value deriving system according to the present modification is different from the disparity value deriving system 15 of the above-described embodiment and the disparity value deriving system of the above-described modification 3. The explanation is centered.

  In the parallax value derivation system according to the present modification, an information processing apparatus such as a normal PC (Personal Computer) is connected to the imaging unit control I / F 107 of the stereo camera 2. As described above, since the dimming control unit 9 is connected to the imaging unit control I / F 107, when the information processing apparatus is connected, for example, the dimming is performed via a network device such as a switching hub. What is necessary is just to connect the control part 9 and information processing apparatus. The information processing apparatus includes an input device such as a mouse and a keyboard, a display device that displays a setting screen or the like (for example, the display device 150 illustrated in FIG. 25), an external storage such as a CPU, RAM, ROM, and HDD (Hard Disk Drive). Equipment.

  FIG. 25A is a diagram showing a state in which a component information selection screen 1500 for selecting a material (subject information) of a component that is a subject is displayed on the display device 150. A part irradiated with pattern light from the pattern light projecting unit 8 has a different area in a saturated state depending on the material of the part (for example, metal, acrylic, wood, glass, etc.). For example, when the material of the component is a metal, it is known in advance that the metal easily reflects light and is likely to be saturated. Therefore, a component that is a metal irradiated with pattern light from the pattern light projecting unit 8 There is a tendency that the area that becomes the saturated region becomes larger than that of the material. In this case, in the component information selection screen 1500, the search range for calculating the cost value C is selected according to the metal by selecting “metal” with a radio button or the like using the input device and selecting the OK button 1500a. Set to On the contrary, when the material of the component is wood, it is known in advance that the wood hardly reflects light and is not easily saturated. Therefore, the component made of wood irradiated with pattern light from the pattern light projecting unit 8 is Compared to other materials, the area that becomes the saturated region tends to be small.

  FIG. 25B is a diagram showing a state in which a component information selection screen 1501 for selecting a reflection level (subject information) of a component that is a subject is displayed on the display device 150. The area irradiated with the pattern light from the pattern light projecting unit 8 has a different area in a saturated state depending on the reflection level (for example, gloss, dark, normal, etc.) of the part. For example, when the surface of the component is glossy and the reflection level is high, it is known in advance that the component in the glossy state easily reflects light and is likely to be saturated, so that pattern light is emitted from the pattern light projecting unit 8. A part tends to have a larger area as a saturation region compared to a part having another reflection level. In this case, in the part information selection screen 1501, “glossy” is selected with a radio button or the like by the input device, and the OK button 1501a is selected, so that the search range for calculating the cost value C corresponds to “glossy”. Range. On the contrary, when the surface of the component is not glossy and dark, it is known in advance that it is difficult to reflect light and to be in a saturated state. Therefore, there is no gloss when pattern light is irradiated from the pattern light projecting unit 8. The part tends to have a smaller area that becomes a saturated region as compared to other reflection level parts. The component information selection screens 1500 and 1501 displayed on the display device 150 and the input device operated by the user correspond to the “setting unit” of the present invention.

  For example, it is assumed that the part is metal or has a glossy state, and “metal” or “gloss” is set on the part information selection screen 1500 or the part information selection screen 1501 of the display device 150. As shown in FIG. 26, it is assumed that a saturated region 23 is formed by reflection of a component in a component region 20c that is a portion where a component is imaged in the comparison image Ia_3 (or the reference image Ib_3). Here, the cost calculation unit 53 changes the reference region p2 of the saturation region 23 of the reference image Ib_3 to a size according to the material of the part or the reflection level. Here, since the component is set to have a metallic or glossy state, the size of the reference region p2 is made larger than the size corresponding to the other material or the component of the reflection level. A search for calculating the cost value C from the candidate region q2 (same size as the reference region p2) of the saturated region 23 of the comparison image Ia_3 is performed on the reference region p2 of the saturated region 23 of the reference image Ib_3 (FIG. 26). In this case, since the pixel values of the candidate areas are all saturated, the parallax value of the corresponding area corresponding to the reference area p2 (corresponding area p2a in FIG. 26) cannot be derived. Therefore, the cost calculation unit 53 (see FIG. 11) determines a predetermined range for the search range for calculating the cost value C, and the predetermined value is equal to or greater than the predetermined value that allows the pixel values to be continuously considered to be saturated. If it is a value, it is determined that the state is saturated within the predetermined range. Next, the cost calculation unit 53 transmits the determination result to the dimming control unit 9, and the analysis unit 200 of the dimming control unit 9 is a saturated region in the luminance image (comparison image Ia_3 (or reference image Ib_3)). Analyzes that there is. When the analysis unit 200 determines that there is an image abnormality in the luminance image, the dimming control unit 9 performs dimming control on the pattern light that the pattern projecting unit 8 irradiates the component.

  In this way, by setting the size of the reference region p2 of the reference image Ib_3 (and the candidate region q2 of the comparison image Ia_3) according to the material or reflection level of the component, the determination accuracy of the presence or absence of saturation in the component is increased. Can be improved.

(Modification 5)
FIG. 27 is a diagram illustrating an example of a hardware configuration of the disparity value deriving system according to the fifth modification of the present embodiment. With reference to FIG. 27, the disparity value deriving system 15b according to the present modification will be described focusing on differences from the disparity value deriving system 15 of the above-described embodiment. In FIG. 27, when the functions of the respective components of the parallax value deriving system 15 of the above-described embodiment are the same, the same reference numerals are given and the description thereof is omitted here.

  As shown in FIG. 27, the parallax value deriving system 15b according to this modification includes a stereo camera 2, a pattern light projecting unit 8, a dimming control unit 9, and a subject information setting unit 16 (setting unit). I have.

  The subject information setting unit 16 is a device that sets the material or reflection level of a component, which is subject information. The subject information setting unit 16 transmits the set subject information to the dimming unit 201 of the dimming control unit 9. Note that the subject information setting unit 16 may include a component information selection screen 1500, 1501 displayed on the display device 150 described in the above-described modification example 4, and an input device operated by the user.

  The light control unit 201 is a circuit that receives subject information from the subject information setting unit 16, receives an analysis result from the analysis unit 200, and transmits an adjustment amount according to the subject information or the analysis result to the pattern light projecting unit 8. It is. Specifically, the dimming unit 201 can predict the trend of change in the area of the region that is saturated or blacked out in advance from the subject information received from the subject information setting unit 16. In order to reduce the black crushing state, the amount of pattern light irradiated to the pattern light projecting unit 8 is adjusted. For example, when the subject information setting unit 16 sets “metal” as the material of the component or “gloss” as the reflection level of the component, the dimming unit 201 compares with the case of other materials or the reflection level, The light quantity of the pattern light irradiated to the pattern light projection part 8 is reduced.

  As described above, since the material or reflection level of the component that is subject information is set in advance, and the amount of pattern light to be irradiated to the pattern light projecting unit 8 is adjusted based on the setting, occurrence of a saturated state or a blackout state occurs. Can be suppressed. Accordingly, the recognition accuracy of the subject can be improved.

  As described above, the dimming unit 201 receives the analysis result from the analysis unit 200 and transmits the adjustment amount according to the analysis result to the pattern light projecting unit 8, but is not limited thereto. In addition, the light control unit 201 may not use the analysis result of the analysis unit 200.

  FIG. 28 is a flowchart illustrating an example of the operation of the transport system according to the fifth modification of the present embodiment. With reference to FIG. 28, description will be made centering on differences from the overall operation of the transport system 1 shown in FIG.

<Step S21>
Similarly to FIG. 1 of the above-described embodiment, it is assumed that the component 20 is installed on the background portion 21 that is a table. The part 20 may be assumed to be installed in the background part 21 from the equipment in the previous process of the transport system 1 or may be assumed to be installed in the background part 21 by the operator. Then, the process proceeds to step S22.

<Step S22>
The subject information setting unit 16 sets the material or reflection level of the component that is the subject information, and transmits the set subject information to the dimming unit 201 of the dimming control unit 9. Then, the process proceeds to step S23.

<Step S23>
The light control unit 201 adjusts the amount of pattern light to be irradiated to the pattern light projecting unit 8 according to the subject information received from the subject information setting unit 16. Then, the process proceeds to step S24.

<Steps S24 to S29>
The processes of steps S24 to S29 of this modification are the same as the processes of steps S12 to S15, S18, and S19 shown in FIG.

  In the above-described embodiment (including each modification example), the stereo camera is described. However, the present invention is not limited to this, and a monocular camera may be provided instead of the stereo camera. In this case, the monocular camera may be moved to pick up a part as a subject a plurality of times, and distance measurement may be performed using the picked-up image, or a phase shift method may be used.

  When at least one of the cost calculation unit 53, the synthesis cost calculation unit 61, the subpixel estimation unit 62, and the generation unit 63 according to the above-described embodiment is realized by executing a program, the program is stored in a ROM or the like in advance. Provided embedded. The programs executed by the stereo cameras 2 and 2a of the above-described embodiments are files in an installable format or an executable format, and are CD-ROM, flexible disk (FD), CD-R, DVD (Digital Versatile). It may be configured to be recorded on a computer-readable recording medium such as Disk). Further, the program executed by the stereo cameras 2 and 2a of the above-described embodiment may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. . Further, the program executed by the stereo cameras 2 and 2a of the above-described embodiment may be configured to be provided or distributed via a network such as the Internet. The program executed by the stereo cameras 2 and 2a of the above-described embodiment includes a module configuration including at least one of the above-described cost calculation unit 53, synthesis cost calculation unit 61, sub-pixel estimation unit 62, and generation unit 63. As actual hardware, the CPU 100 reads out and executes a program from the ROM 101, so that the above-described units are loaded onto the main storage device and generated.

DESCRIPTION OF SYMBOLS 1 Transport system 2, 2a Stereo camera 3, 4 Image pick-up part 5 Matching part 6 Parallax image generation part 7 Recognition processing part 8, 8a Pattern light projection part 9 Light control part 10 Arm control part 11 Arm 12 Texture transmission part 13, 14 Pre-processing unit 15, 15a, 15b Parallax value deriving system 16 Subject information setting unit 20 Component 20a, 20b, 20c Component region 21 Background portion 21a Background region 21b Point 22, 23 Saturation region 51, 52 Filter unit 53 Cost calculation unit 61 Composition Cost calculation unit 62 Subpixel estimation unit 63 Generation unit 100 CPU
101 ROM
102 ROMI / F
103 RAM
104 RAM I / F
DESCRIPTION OF SYMBOLS 105 Image processing part 106 Imaging part 106a, 106a_1, 106a_2 Lens 106b Aperture 106c Image sensor 107 I / F for imaging part control
DESCRIPTION OF SYMBOLS 108 Front cover 109 Side cover 109a Radiation fin 110 Rear cover 111 Power supply connector 112 Bottom cover 150 Display apparatus 200 Analysis part 201 Light control part 300 Light quantity adjustment part 301 Light source part 302 Texture filter 303 Lens 304 Projector 400, 401 area 510a, 510b Imaging device 511a, 511b Lens 1500, 1501 Component information selection screen 1500a, 1501a OK button B Base line length C Cost value d Shift amount dp, dp1, dp20, dp21 Parallax value E Object EL Epipolar line f Focal length Ia, Ia_2, Ia_3 Comparison image Ib , Ib_2, Ib_3, Ib_ex Reference image Ip, Ip_ex Parallax image Lr Path cost value Ls Composite cost value p Reference pixel p1, p2 Reference region p1 , P2a corresponding region q candidate pixels q1, q2 candidate region S, Sa, Sb point Z distance

JP 2012-181142 A

Claims (13)

A light projecting means for irradiating a subject with pattern light having a specific pattern to form a texture on the subject;
Imaging means for imaging the subject on which the texture is formed;
Derivation means for deriving distance information to the subject based on a captured image captured by the imaging means;
Analyzing means for analyzing the presence or absence of an image abnormality for at least either the captured image or the image based on the distance information;
A dimming unit that performs dimming control when it is analyzed that the image abnormality is present by the analyzing unit;
Information processing system with The image pickup means is arranged in a position different from the first image pickup means for picking up the subject and generating a reference image, and second image pickup for picking up the subject and generating a comparison image Means,
The derivation means further comprises calculation means for calculating the degree of coincidence between the area of the reference image and the area of the comparison image,
The deriving unit derives, as the distance information, a parallax value that specifies a corresponding region of the comparison image corresponding to a reference region of the reference image from the degree of coincidence calculated by the calculating unit,
A generating unit configured to generate a parallax image based on the parallax value;
The information processing system according to claim 1, wherein the analysis unit analyzes the presence / absence of the image abnormality in the captured image and the parallax image.   When the light control means is analyzed by the analysis means that there is blur of the texture in the captured image as the image abnormality, or when it is analyzed that the light control means has reached the light control limit, The information processing system according to claim 2, wherein an exposure adjustment operation for the first imaging unit and the second imaging unit is executed as the light control.   The information processing system according to claim 3, wherein the dimming unit performs dynamic range expansion processing on the reference image and the comparative image as the exposure adjustment operation.   The analysis means is configured such that the luminance value of the captured image is equal to or greater than the first value over a predetermined range of the captured image, or the second smaller than the first value over the predetermined range of the captured image. The information processing system according to any one of claims 2 to 4, wherein when the image is less than or equal to the value, the captured image is analyzed as having the image abnormality. Setting means for setting information on the subject;
The information processing system according to any one of claims 2 to 5, wherein the calculation unit changes a size of the reference region of the reference image according to information on the subject.   The information processing system according to claim 6, wherein the subject information is a reflection level of the subject.   The information processing system according to claim 6, wherein the subject information is a material of the subject.   The information processing system according to any one of claims 6 to 8, wherein the light control unit adjusts a light amount of the pattern light of the light projecting unit according to information on the subject set by the setting unit. . A light projecting means for irradiating a subject with pattern light having a specific pattern to form a texture on the subject;
Imaging means for imaging the subject on which the texture is formed;
Derivation means for deriving distance information to the subject based on a captured image captured by the imaging means;
Setting means for setting information of the subject;
Dimming means for performing dimming control based on information on the subject set by the setting means;
Information processing system with   The information processing system according to claim 10, wherein the subject information is a reflection level of the subject.   The information processing system according to claim 10, wherein the subject information is a material of the subject. A light projecting step of irradiating a subject with pattern light having a specific pattern to form a texture on the subject;
An imaging step of imaging the subject on which the texture is formed;
A derivation step for deriving distance information to the subject based on the captured image;
An analysis step of analyzing the presence or absence of an image abnormality for at least either the captured image or the image based on the distance information;
A dimming step for performing dimming control when analyzing that there is an image abnormality;
An information processing method comprising: