JP6054831B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program, and more particularly, to an image processing apparatus, an image processing method, and an image processing program for detecting a predetermined area included in an image.

  An image processing technique for detecting a desired shape or object from an image obtained by imaging a target space is known. For example, Patent Document 1 discloses a technique for detecting a rectangular area in an image. In Patent Document 1, edge pixels included in an input image are extracted, straight lines are detected based on the edge pixels, and a rectangular region is detected from four lines that are substantially orthogonal.

JP 2013-106160 A

  Since Patent Document 1 is a technique for reading a document from an image, it is assumed that an imaging target and a camera are substantially facing each other, and whether a straight line is substantially orthogonal on a two-dimensional image captured substantially facing each other. Judgment. That is, as shown in FIG. 32, in Patent Document 1, an image 900 captured from a position facing a paper surface (plane) of a document is acquired, and horizontal and vertical straight lines in a two-dimensional plane of the image 900 are detected. Then, a rectangular area 901 indicating a document is detected from these straight lines.

  However, Patent Document 1 does not consider a three-dimensional environment in which the imaging target and the camera do not face each other. For example, as shown in an image 910 in FIG. 33, when a rectangular plane (three-dimensional plane) 911 such as a cupboard in a three-dimensional environment is measured (imaged) from an oblique direction, a two-dimensional image (image plane) is obtained by perspective. The horizontal and vertical straight lines in the plane 911 do not form 90 degrees, and the corners 911a of the plane 911 are not necessarily orthogonal (not at right angles). For this reason, it is difficult to detect a rectangular region by applying the conventional technique such as Patent Document 1 to a three-dimensional environment. That is, in the prior art, it is assumed that the object and the camera are facing each other, and when not facing, the straight line detected by performing straight line detection is not orthogonal to the other straight lines, It is difficult to detect a rectangular area.

  In addition, although (standard) Hough transform is used for straight line detection as an embodiment of Patent Document 1, it is generally difficult to set a threshold value for whether or not to detect as a straight line in (standard) Hough transform. For example, when a straight line is detected using Hough transform for an image 910 including a plurality of small patterns 911b facing substantially the same direction as shown in FIG. 33, in addition to a straight line 912 corresponding to a rectangular edge as shown in FIG. Thus, the straight line 912a is also detected in the pattern 911b. For this reason, a plurality of rectangular “corner” candidates may be detected due to an unnecessary straight line, resulting in an increase in false detection. That is, in the prior art, the pattern of the object is extracted as a straight line by the straight line detection, and the shape of the object may not be accurately detected.

  Therefore, the conventional technique has a problem that a rectangular region cannot be accurately detected in a three-dimensional environment in which an image is acquired from a position that does not face the detection target.

  An image processing apparatus according to the present invention includes an edge pixel detection unit that detects a plurality of edge pixels in a three-dimensional plane included in measurement image information, and a plurality of edge line segments that connect the plurality of detected edge pixels. An edge line detection unit for detecting, a coordinate system conversion unit for converting the coordinate system of the detected plurality of edge line segments into a plane coordinate system viewed from the normal direction of the three-dimensional plane, and a plane coordinate system A rectangular area detecting unit for detecting a rectangular area in the three-dimensional plane based on a plurality of edge line segments.

  The image processing method according to the present invention detects a plurality of edge pixels in a three-dimensional plane included in measurement image information, detects a plurality of edge line segments connecting the plurality of detected edge pixels, and detects the detection. The coordinate system of the plurality of edge line segments is converted into a plane coordinate system viewed from the normal direction of the three-dimensional plane, and based on the distribution of the plurality of edge line segments in the plane coordinate system, The rectangular area is detected.

  An image processing program according to the present invention detects a plurality of edge pixels in a three-dimensional plane included in measurement image information, detects a plurality of edge line segments connecting the plurality of detected edge pixels, and detects the detection The coordinate system of the plurality of edge line segments is converted into a plane coordinate system viewed from the normal direction of the three-dimensional plane, and based on the distribution of the plurality of edge line segments in the plane coordinate system, This is for causing a computer to execute an image processing method for detecting a rectangular area.

  According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and an image processing program that can detect a rectangular region with high accuracy.

It is a flowchart which shows the outline | summary of the rectangular area detection method which concerns on embodiment. It is a figure which shows an example of the image processed with the rectangular area detection method which concerns on embodiment. It is a figure which shows an example of the image processed with the rectangular area detection method which concerns on embodiment. It is a figure which shows an example of the image processed with the rectangular area detection method which concerns on embodiment. It is a figure which shows an example of the image processed with the rectangular area detection method which concerns on embodiment. It is a figure which shows an example of the image processed with the rectangular area detection method which concerns on embodiment. It is a figure which shows an example of the image processed with the rectangular area detection method which concerns on embodiment. It is explanatory drawing for demonstrating the rectangular area detection method which concerns on embodiment. 1 is a block diagram illustrating a configuration of a rectangular area detection device according to a first embodiment. FIG. 6 is an explanatory diagram for explaining an operation of the rectangular area detection device according to the first embodiment. FIG. 6 is an explanatory diagram for explaining an operation of the rectangular area detection device according to the first embodiment. FIG. 6 is an explanatory diagram for explaining an operation of the rectangular area detection device according to the first embodiment. 3 is a flowchart illustrating a rectangular area detection method according to the first embodiment. 6 is a diagram illustrating an example of an image processed by the rectangular area detection method according to Embodiment 1. FIG. FIG. 6 is an explanatory diagram for explaining a rectangular area detection method according to the first embodiment. FIG. 6 is an explanatory diagram for explaining a rectangular area detection method according to the first embodiment. FIG. 6 is an explanatory diagram for explaining a rectangular area detection method according to the first embodiment. FIG. 6 is an explanatory diagram for explaining a rectangular area detection method according to the first embodiment. FIG. 6 is an explanatory diagram for explaining a rectangular area detection method according to the first embodiment. FIG. 6 is an explanatory diagram for explaining a rectangular area detection method according to the first embodiment. FIG. 6 is an explanatory diagram for explaining a rectangular area detection method according to the first embodiment. FIG. 6 is an explanatory diagram for explaining a rectangular area detection method according to the first embodiment. It is a block diagram which shows the structure of the furniture model processing system which concerns on Embodiment 2. FIG. It is a flowchart which shows the furniture model production | generation method which concerns on Embodiment 2. FIG. It is a flowchart which shows the furniture model production | generation method which concerns on Embodiment 2. FIG. It is a figure which shows an example of the image processed with the furniture model production | generation method concerning Embodiment 2. FIG. It is a figure which shows an example of the user interface used with the furniture model production | generation method concerning Embodiment 2. FIG. It is a figure which shows an example of the user interface used with the furniture model production | generation method concerning Embodiment 2. FIG. It is a figure which shows an example of the user interface used with the furniture model production | generation method concerning Embodiment 2. FIG. It is a figure which shows an example of the user interface used with the furniture model production | generation method concerning Embodiment 2. FIG. It is a figure which shows an example of the user interface used with the furniture model production | generation method concerning Embodiment 2. FIG. It is a figure which shows the conventional rectangular area detection method. It is a figure which shows the conventional rectangular area detection method. It is a figure which shows the conventional rectangular area detection method.

(Outline of the embodiment)
Prior to the description of the embodiment, an outline of features of the embodiment will be described. In the embodiment, a rectangular area in the environment is detected from information of a distance image sensor that measures the environment. FIG. 1 shows an outline of a rectangular area detecting method according to the embodiment. This rectangular area detecting method is executed by a rectangular area detecting device or the like as will be described later.

  As shown in FIG. 1, first, a three-dimensional attention plane is detected from a distance image obtained by measuring the environment with a distance image sensor (S11). For example, when a distance image as shown in FIG. 2 is input, the rectangular area detection device detects a target plane (a white background portion in FIG. 3) as shown in FIG.

  Subsequently, a planar mask image is generated from the three-dimensional target plane detected in S11 (S12), and edge detection is performed on the generated mask image (S13). For example, the rectangular area detection device generates a mask image obtained by extracting a planar image from the distance image based on the distance image of FIG. 2 and the three-dimensional attention plane of FIG. The detected edge image is detected.

  Subsequently, the rectangular area detection apparatus detects an edge fragment group (a set of edge line segments) using line segment detection means such as probabilistic Hough transform for the edge image detected in S13 (S14). For example, the rectangular area detection device performs a probabilistic Hough transform on the edge image of FIG. 4 and detects an edge fragment group as shown in FIG. FIG. 8A schematically shows an edge fragment group of a two-dimensional measurement coordinate system (a coordinate system viewed from a sensor).

  Subsequently, the rectangular area detection apparatus converts the two-dimensional edge fragment group detected in S14 into a three-dimensional plane coordinate system (a coordinate system viewed from the normal direction of the plane) (S15). For example, the rectangular area detection apparatus converts the edge fragment group of the two-dimensional measurement coordinate system in FIG. 8A into a three-dimensional plane coordinate system as shown in FIG.

  Subsequently, the rectangular area detection apparatus creates an edge presence position histogram in the X and Y axis directions in the three-dimensional plane coordinate system converted in S15, and uses an average shift method or the like to detect edge presence peaks in the X and Y axis directions. The position is obtained (S16). For example, the rectangular area detection apparatus uses the edge presence position histogram indicating the distribution of the edge fragments in the X and Y axis directions as shown in FIG. 8C for the edge fragment group in the three-dimensional plane coordinate system of FIG. And the peak position of this histogram is obtained.

  Subsequently, the rectangular area detecting device obtains the minimum line segment (peak line segment) connecting the edge presence peak positions in the X and Y axis directions obtained in S16 (S17). For example, the rectangular area detection apparatus connects the peak positions of the edge presence position histogram of FIG. 8C, and generates a minimum line segment as shown in FIG. 8D or FIG.

  Subsequently, the rectangular area detection device detects a correct rectangular (rectangular) area by collating whether or not the edge fragment group detected in S14 exists on the minimum line segment obtained in S17 (S18). . For example, the rectangular area detection apparatus collates the minimum line segment in FIG. 8D or FIG. 6 with the edge fragment group in FIG. 8A or FIG. 5 to obtain an area surrounded by the effective minimum line segment. Thus, a rectangular area as shown in FIG. 8E or FIG. 7 is detected.

  As described above, in the embodiment, in a method for detecting a rectangular region in an environment from information of a distance image sensor that measures the environment, an edge fragment group extracted from a two-dimensional image is converted into a three-dimensional plane coordinate system. Then, a rectangular region can be detected with high reliability by creating a histogram indicating the distribution of edge presence positions in two axial directions orthogonal to each other in the plane coordinate system.

(Embodiment 1)
The first embodiment will be described below with reference to the drawings. FIG. 9 shows a configuration of an image processing system (rectangular area detection system) according to the present embodiment.

  As shown in FIG. 9, the image processing system 1 according to the present embodiment includes a rectangular area detection device 100 and a distance image sensor 200. The rectangular area detection device 100 and the distance image sensor 200 may be an integrated device or may be independent devices. For example, the rectangular area detection device 100 and the distance image sensor 200 are provided in a robot or the like.

  The distance image sensor (distance image measurement unit) 200 generates a distance image (distance image data, measurement image data) obtained by measuring a three-dimensional environment. The distance image includes image information (input image) obtained by imaging the measurement target from the sensor (measurement point) and distance information from the sensor to the measurement target. For example, “RGBD (color + distance)” per pixel. Or, “Grayscale (luminance value) + distance” is included. The distance image sensor 200 includes a stereo camera (three-dimensional camera), an LRF (laser range finder) + camera, Microsoft Kinect (registered trademark), and the like.

  The rectangular area detection apparatus 100 includes a distance image acquisition unit 101, a plane detection unit 102, a plane mask image generation unit 103, a plane coordinate system calculation unit 104, an edge detection unit 105, an edge fragment detection unit 106, a coordinate system conversion unit 107, an edge An existence position histogram generation unit 108, a histogram peak estimation unit 109, a line segment generation unit 110, a neighboring edge fragment extraction unit 111, an edge matching unit 112, and a rectangular area detection unit 113 are provided.

  Note that other functional blocks may be used as long as a rectangular area detection method according to the present embodiment described later can be realized. Each function (each process) in the rectangular area detection apparatus 100 in FIG. 9 is configured by hardware and / or software, and may be configured by one piece of hardware or software, or by a plurality of pieces of hardware or software. It may be configured. Each function of the rectangular area detection device 100 may be realized by a computer having a CPU (Central Processing Unit), a memory, and the like. For example, a rectangular area detection program for performing a rectangular area detection method described later is stored in the storage device, and each function of the rectangular area detection device 100 is executed by the CPU executing the rectangular area detection program stored in the storage device. It may be realized.

  The distance image acquisition unit 101 acquires a distance image (distance image data) generated by the distance image sensor 200 measuring the environment. The plane detection unit 102 performs plane detection processing on the input image included in the distance image acquired by the distance image acquisition unit 101 to detect a target plane (three-dimensional plane). The plane mask image generation unit (extraction image generation unit) 103 extracts a mask image (extraction image) corresponding to the target plane from the input image using the region of the target plane detected by the plane detection unit 102 as a mask. The plane coordinate system calculation unit 104 calculates a plane coordinate system based on the plane for the target plane detected by the plane detection unit 102.

  The edge detection unit (edge pixel detection unit) 105 performs edge detection processing on the mask image generated by the planar mask image generation unit 103, and generates an edge image including the detected edge pixel. An edge fragment detection unit (edge line segment detection unit) 106 performs a probabilistic Hough transform process on the edge image detected by the edge detection unit 105, and an edge fragment group (edge line) that connects edge pixels included in the edge image. Subgroup) is detected.

  The coordinate system conversion unit 107 converts the coordinate system of the two-dimensional edge fragment group detected by the edge fragment detection unit 106 into a three-dimensional plane coordinate system. The coordinate system conversion unit 107 converts the two-dimensional coordinate system into a three-dimensional coordinate system, and also converts the three-dimensional coordinate system into a two-dimensional coordinate system.

The conversion processing from the three-dimensional coordinates (X, Y, Z) to the two-dimensional coordinates (u, v) in the coordinate system conversion unit 107 is performed by, for example, perspective projection conversion using a pinhole camera model. In the pinhole camera model, as shown in FIG. 10, a camera that uses a camera coordinate system (sensor coordinate system), a global coordinate system (world coordinate system), and a simultaneous conversion matrix that expresses the position and orientation between coordinate systems as a matrix. Corresponds to a sensor (distance image sensor), and the camera coordinate system is a coordinate system based on the camera. The global coordinate system is a coordinate serving as a three-dimensional reference, and is a coordinate system based on a measurement target (for example, a target plane). Formula 1 below shows a conversion formula for converting three-dimensional coordinates into two-dimensional coordinates using a simultaneous conversion matrix. The camera external parameter matrix (simultaneous conversion matrix) indicates the position and orientation of the global coordinate system viewed from the camera coordinate system.

The conversion processing from the two-dimensional coordinates to the three-dimensional coordinates in the coordinate system conversion unit 107 is also performed using a pinhole camera model. As shown in FIG. 11, when the focal distance from the camera to the image plane is f and the distance from the camera to the object (measurement target) is D, the two-dimensional coordinates (u, v) viewed from the camera are expressed by the following equation (2). It becomes.

In the distance image sensor according to the present embodiment, the distance D between the pixel and the target object can be measured, and the value of (u, v, D) for one pixel can be obtained. It can be calculated by the following equation 3.

  An edge presence position histogram generation unit (histogram generation unit) 108 generates an edge presence position histogram indicating the distribution of edge fragment groups in the three-dimensional planar coordinate system converted by the coordinate system conversion unit 107. A histogram peak estimation unit (peak detection unit) 109 estimates a peak at a position where an edge exists in the edge presence position histogram generated by the edge presence position histogram generation unit 108. The line segment generation unit (peak line segment generation unit) 110 generates a minimum line segment (peak line segment) that connects the peak positions of the histogram estimated by the histogram peak estimation unit 109.

  The neighborhood edge fragment extraction unit (neighboring edge line segment extraction unit) 111 refers to the edge fragment group detected by the edge fragment detection unit 106 and extracts the edge fragment group near the minimum line segment generated by the line segment generation unit 110. . The neighboring edge fragment extraction unit 111 extracts an edge fragment group near the minimum line segment based on the distance obtained by projecting the minimum line segment onto the edge image of the edge fragment group. The distance at which the minimum line segment (edge fragment group) is projected onto the image can be obtained as the distance from the point to the line segment (straight line) in the two-dimensional image or the three-dimensional image.

When projecting onto a two-dimensional image (two-dimensional coordinate system), the distance from the point to the straight line is obtained by the following equation (4). Formula 4 shows the length (distance) d of the perpendicular line drawn from the point A (x ₀ , y ₀ ) to the straight line L (ax + by + c = 0).

  The distance between a point and a line segment is divided according to whether or not a perpendicular line can be dropped from the point to the line segment. If a perpendicular line can be dropped on the line segment, the distance can be calculated by the above equation 4, and the perpendicular line can be lowered on the line segment. If not, the distance between the end point (start point or end point) of the line segment and the point is the distance between the line segment and the point. The neighboring edge fragment extraction unit 111 calculates the distance between the center position of the edge fragment and the minimum line segment by this calculation method.

When projecting to a three-dimensional image (three-dimensional coordinate system), as shown in FIG. 12, a perpendicular line drawn from the point of interest P to the minimum line segment E can be written as VP _{→ H,} and the distance between the point and the line segment | V _{P → H} | can be calculated as shown in Equation 5 below.

  The edge collating unit 112 collates the neighboring edge fragment group extracted by the neighboring edge fragment extracting unit 111 with the minimum line segment, and determines whether or not it is a valid minimum line segment (success / failure of collation). The edge matching unit 112 determines the validity of the minimum line segment based on the length of the edge fragment group of the three-dimensional image projected onto the minimum line segment, and removes the invalid minimum line segment (matching has failed). . The rectangular area detection unit 113 determines that the area surrounded by the line segment that the edge matching unit 112 determines to be the effective minimum line segment (successful verification) is a rectangular area.

  FIG. 13 shows a rectangular area detection method (rectangular area detection process) executed by the image processing system (rectangular area detection apparatus) according to the present embodiment.

  As shown in FIG. 13, first, the distance image sensor 200 measures a three-dimensional environment (S101). The distance image sensor 200 measures a three-dimensional environment to generate a distance image including the input image and the distance as shown in FIG. 2, and the distance image acquisition unit 101 acquires the distance image.

  Subsequently, the plane detection unit 102 detects the attention plane from the input image included in the distance image acquired in S101, and specifies a distance point group on the attention plane (S102). For example, the plane detection unit 102 uses a method such as RANSAC (RANdom SAmple Consensus) on the input image as shown in FIG. 2 to obtain the plane parameters (ax + by + cz + d = 0) a, b, c as shown in FIG. , D) and a distance point group on the plane (white background portion in FIG. 3). In the RANSAC method, a plane parameter is obtained from an arbitrary point, and a plane that includes many points on the plane parameter is specified as a plane. By using the RANSAC algorithm, a plane can be detected with high accuracy from a noisy input image.

  Subsequently, the plane mask image generation unit 103 extracts only the image of the target plane area detected in S102 from the input image acquired in S101 (S103). The plane mask image generating unit 103 sets “1” (a white background portion in FIG. 3) for pixels where the distance point group exists on the detected plane as shown in FIG. 3, and “0” for pixels where the distance point group for the plane does not exist. By creating (a black background portion in FIG. 3) a binary mask and by ANDing the input image and this binary mask as shown in FIG. 2 (leaving only the color information of the pixel “1”), A planar mask image as shown in FIG. 14 is generated.

  Subsequently, the plane coordinate system calculation unit 104 calculates the plane coordinate system of the plane of interest detected in S102 (S104). The plane coordinate system calculation unit 104 calculates a plane coordinate system as shown in FIG. 15 based on the plane parameters obtained in S102. Here, as an example, the normal direction of the plane is the Z axis, the vertical direction (gravity direction) is the Y axis, and the horizontal direction is the X axis. For these three-dimensional coordinate axes, the order of X, Y, and Z may be changed. Any two axes (in the rectangular plane) in the direction in which the rectangular area is to be detected may be set as the X and Y axes. At least the plane normal is selected as one axis (out of three axes). For example, when a rectangular area such as a furniture door or drawer is detected from the input image as shown in FIG. 2, the vertical direction (gravity direction) is preferably the Y axis. By detecting the rectangular area on the vertical plane, the rectangular area can be assumed as a furniture candidate.

  Subsequently, the edge detection unit 105 detects an edge image from the mask image generated in S103 (S105). The edge detection unit 105 can use an arbitrary edge filter. For example, the edge detection unit 105 detects an edge using a Canny filter or a Laplacian filter, and generates an edge image as shown in FIG. The Canny filter can detect an edge image resistant to noise by using a Gaussian differential filter. FIG. 16 is an enlarged view of the edge image. As shown in FIG. 16, the edge image is an image (for example, white background) in which the pixel (position) where the edge is detected is shown as the edge pixel. That is, the edge image indicates whether each pixel is an edge.

  Subsequently, the edge fragment detection unit 106 detects an edge fragment group for the edge image generated in S105 (S106). The edge fragment detection unit 106 detects a group of edge fragments (set of edge line segments) as shown in FIG. 5 by using probabilistic Hough transform for the edge image as shown in FIG. FIG. 17 shows a group of edge fragments for the edge image (edge pixel) of FIG. As shown in FIG. 17, an edge fragment is a line segment (edge line segment) connecting edge pixels. That is, the edge fragment is a line segment (straight line) starting from the first edge pixel and ending at the second edge pixel. Note that although there are a plurality of line segments connecting the edge pixels, they are referred to as edge fragment groups, but one or more of the line segments may be referred to as edge fragments.

  The Hough transform includes a standard Hough transform and a probabilistic Hough transform, and the present embodiment uses the stochastic Hough transform. In the probabilistic Hough transform, a randomly selected edge pixel is used for voting on a parameter plane, and features on the image are detected based on the voting result. By using the probabilistic Hough transform, a line segment having an edge pixel as an end point can be detected from the edge image. For this reason, as shown in FIG. 34 of the prior art, it is possible to prevent detection of an unnecessary straight line as compared with the case where standard Hough transform is used. In addition, you may use not only a probabilistic Hough transformation but the other line segment detection method of detecting the line segment which connects an edge pixel.

  Subsequently, the coordinate system conversion unit 107 converts the (two-dimensional) edge fragment group detected in S106 into a (three-dimensional) plane coordinate system (S107). The coordinate system conversion unit 107 uses a pinhole camera model as in Equation 1 above, and uses a (two-dimensional) coordinate system (camera coordinate system) of an edge fragment viewed from the distance image sensor as shown in FIG. 8) is converted into a (three-dimensional) plane coordinate system (global coordinate system) as seen from the Z direction (normal direction) as shown in FIG. 8B.

In the distance image measured by the distance image sensor 200, distance information is also included in one pixel. Therefore, the three-dimensional positions corresponding to the “center position”, “edge start point”, and “edge end point” pixels of the edge fragment are described above. Coordinate transformation is performed for each of Equation 1 as Equation 6 below.
The simultaneous transformation matrix of Expression 6 is calculated by the plane coordinate system calculation unit 104. Here, the coordinates of the three points of the “edge position”, “edge start point”, and “edge end point” of the edge fragment are converted. However, in order to convert the edge fragment into three-dimensional coordinates, at least the “edge start point”, The coordinates of the two points “edge end point” may be converted. In the present embodiment, since the center position of the edge fragment is used in the next S108, the coordinates of this center position are also converted into three-dimensional coordinates.

  Subsequently, the edge presence position histogram generation unit 108 generates an edge presence position histogram indicating the edge presence position for the edge fragment group in the plane coordinate system converted in S107 (S108). The edge presence position histogram generation unit 108 generates an edge presence position histogram as shown in FIG. 8C for the edge fragment group in the planar coordinate system as shown in FIG.

Here, the center position of the edge fragment is ^3D C = (x _c , y _c , z _c ), the vector connecting the start point and the end point is ^3D S, the unit vector of the three-dimensional vertical axis (Y axis), and the horizontal The unit vectors of the axis (X axis) are n _v and n _h , respectively. As shown in the following Expression 7, the inner product of the “vector connecting the start point and the end point” of the edge fragment and the “X, Y axis direction unit vector” is _h w and _v w.

As shown in FIG. 18, the inner product _{h w,} a _v w, each position _x c, the weight of edge fragments in _{y c.} In the example of FIG. 18, since the edge fragment is oriented (tilted) in the X direction (horizontal direction), the weight in the X direction is increased. As shown in FIG. 8C, a histogram indicating the distribution in the X direction and the Y direction using the inner products _h w and _v w as weights is referred to as an edge presence position histogram. That is, the edge presence position histogram shows the distribution of the inclination of the edge fragment with respect to the X axis and the Y axis (the distribution of the weight according to the inclination).

Subsequently, the histogram peak estimation unit 109 estimates the peak position of the edge presence position in the edge presence position histogram generated in S108 (S109). The histogram peak estimation unit 109 uses an arbitrary peak detection method. For example, the histogram peak estimation unit 109 uses a mean shift method, which is a peak detection method, for the edge presence position histogram as shown in FIG. 8C, and a set of peak positions in the X and Y axis directions ( _h P, _v P) is estimated. A peak portion (vertex portion) of the distribution in the X direction and the Y direction in FIG. 8C is the peak position.

Subsequently, the line segment generation unit 110 generates the minimum line segment that connects the X and Y axis peak positions in the edge presence position histogram estimated in S109 (S110). As shown in FIG. 19, the line segment generation unit 110 obtains an intersection point I from the X and Y axis peak positions in the edge presence position histogram, and sets a minimum line segment connecting the intersection points I in the X and Y axis directions ( _hL , _vL ).

Thereafter, all the minimum line segments in _h L and _v L generated in S110 are sequentially selected, and the processes of S111 to S114 are repeated. First, the neighboring edge fragment extraction unit 111 extracts an edge fragment group near the selected minimum line segment (S111). The neighboring edge fragment extraction unit 111 obtains the distance between the minimum line segment and the edge fragment on the two-dimensional image or the three-dimensional image, and extracts all the edge fragment groups E in which the distance is less than or equal to the threshold pixel.

  When obtaining the distance on the two-dimensional image, the neighboring edge fragment extraction unit 111 projects the minimum line segment (three-dimensional) onto the two-dimensional image, and the edge fragment whose distance from the minimum line segment on the two-dimensional image is a threshold pixel or less. Ask for. Specifically, for example, the coordinate system conversion unit 107 converts the three-dimensional coordinates of the minimum line segment (start point and end point) into two-dimensional coordinates using Equation 1 above. Next, as shown in FIG. 20, the distance between the minimum line segment and the point (edge fragment center) is calculated by the calculation method of Equation 4, and if the distance is smaller than a threshold value (for example, 8 pixels), the edge fragment is Extracted as “neighboring edge fragment of minimum line segment”.

  Further, when obtaining the distance on the 3D image, the neighboring edge fragment extraction unit 111 projects the edge fragment group (2D) onto the 3D image, and the distance from the minimum line segment on the 3D image is equal to or less than the threshold pixel. Find edge fragments. Specifically, for example, the coordinate system conversion unit 107 converts the two-dimensional coordinates of the edge fragment group into the three-dimensional coordinates using the above Equation 3. Next, as shown in FIG. 21, the distance between the center point of the edge fragment converted into three dimensions and the minimum line segment is obtained by the calculation of the above formula 5, and if the distance is smaller than a threshold (for example, 0.02 mm), the edge Fragments are extracted as “neighboring edge fragments of the minimum line segment”.

  Subsequently, the edge collating unit 112 collates the edge fragment group extracted in S111 with the minimum line segment (S112). The edge matching unit 112 determines that the minimum line segment or the minimum line segment whose length projected from the edge fragment on the two-dimensional image or the three-dimensional image is equal to or greater than a threshold value is valid. The minimum line segment and the edge fragment may be collated in either the two-dimensional image or the three-dimensional image, but it is preferable to collate on the three-dimensional image in order to evaluate the three-dimensional edge inclination. Therefore, the case of a three-dimensional image will be described.

  Note that when the distance is obtained on the two-dimensional image in S111, after extracting as the “closest edge fragment of the minimum line segment”, the edge fragment center position and the edge fragment start point / end point are respectively obtained by the coordinate system conversion unit 107. Using the calculation of Equation 3, the two-dimensional coordinates are converted into three-dimensional coordinates, and the converted coordinates are input to the edge collating unit 112. Further, when the distance is obtained on the three-dimensional image in S111, since the extracted “neighboring edge fragment of the minimum line segment” is already the three-dimensional coordinate, the coordinate as it is is input to the edge matching unit 112.

  In S112, as shown in FIG. 22, the edge matching unit 112 projects the edge fragment of the three-dimensional coordinates onto the minimum line segment, and calculates the “projected length” of the edge fragment on the minimum line segment. If the projected length is equal to or greater than a certain threshold with respect to the length of the minimum line segment, it is determined to be an effective minimum line segment (a line segment where an edge exists). For example, the length of the minimum line segment is set to 1, and the projected length of 0.3 is set as the threshold value.

  Subsequently, if there is no edge fragment on the line segment as a result of the collation in S112, the edge collating unit 112 removes the minimum line segment (S113). Subsequently, the edge matching unit 112 determines whether or not there is an unmatched minimum line segment (S114), and if there is an unmatched minimum line segment, selects the next line segment and repeats the processing from S111 onward. .

  Subsequently, when all the minimum line segments have been collated, the rectangular area detection unit 113 detects an area surrounded by the four minimum line segments as a rectangular area based on the remaining minimum line segments (S115).

  As described above, in the present embodiment, the edge fragment group is detected based on the edge image, the edge presence position histogram indicating the distribution of the edge fragment group is generated, and the minimum line segment connecting the peaks of the edge presence position histogram is Rectangle detection in a three-dimensional environment was made possible by collating with edge fragment groups. In the present embodiment, a rectangle can be detected with high reliability by taking a peak of a histogram of orthogonal two axes.

  That is, by representing the object as a three-dimensional coordinate item, the line segments connecting the edge fragments are orthogonal to each other, and a rectangular region appears. In addition, since the line segment connects the peak positions of the edge presence position histogram, the pattern of the object is not recognized as a straight line. Therefore, even in a three-dimensional environment in which an image is acquired from a position that does not face the target, a rectangular area can be detected, and a rectangular area can be accurately detected without recognizing a pattern or the like as a straight line.

(Embodiment 2)
The second embodiment will be described below with reference to the drawings. In the present embodiment, in order for the robot to operate the furniture, it is possible to set the operation information of the furniture candidate using the rectangular area detected in the first embodiment as the furniture candidate.

  FIG. 23 shows a configuration of the furniture model processing system according to the present embodiment. As shown in FIG. 23, the furniture model processing system 2 according to the present embodiment includes a furniture model generation device 300 in addition to the rectangular area detection device 100 and the distance image sensor 200 of the first embodiment.

  The furniture model generation apparatus (3D model generation assumption) 300 includes an HMI (Human Machine Interface) unit 301, an operation information addition unit 302, an operation display unit 303, and a model storage unit 304. For example, the furniture model generation apparatus 300 includes a PC (personal computer), a tablet PC, a smartphone, and the like. Each function of the furniture model generation device 300 is configured by hardware or software similarly to the rectangular area detection device 100, and each function may be realized by executing a program stored in the storage device by a computer. . The furniture model generation device 300 includes one or a plurality of devices. For example, the furniture model generation device 300 includes a HMI unit 301, an operation information addition unit 302, and an operation display unit 303 in a terminal device operated by a user, and a model storage unit 304 as a robot You may prepare.

  The HMI unit 301 is a user interface configured with a touch panel, a mouse, or the like. The HMI unit 301 displays (outputs) information to the user and accepts an input operation from the user.

  The operation information adding unit (operation specifying unit) 302 adds operation information to the furniture model in response to a user operation instruction via the HMI unit 301. The operation display unit 303 displays the furniture model to which the operation information adding unit 302 has added the operation information to the user via the HMI unit 301.

  The model storage unit 304 is a storage unit configured with a memory, a hard disk, and the like, and stores a furniture model. The model storage unit 304 stores the furniture model to which the operation information adding unit 302 has given operation information and the operation display unit 303 has displayed. The furniture model (furniture model information) includes the three-dimensional position and shape of furniture (object), operation information, and the like.

  FIG. 24 shows a furniture model processing method executed by the furniture model processing system (furniture model processing apparatus) according to the present embodiment, and FIG. 25 shows details of the operation method instruction processing (S203) of FIG. Show.

  As shown in FIG. 24, first, the rectangular area detection device 100 identifies a rectangular area as in the first embodiment (S201). A distance image sensor 200 mounted on the robot measures a three-dimensional environment around the robot, and generates a distance image as shown in FIG. 26, for example. Then, the rectangular area detection device 100 detects a rectangular area included in the distance image.

  Subsequently, the rectangular area detected in S201 is presented as a furniture candidate to the HMI unit 301 (S202). Since a rectangular plane perpendicular to the floor surface in a three-dimensional environment is likely to constitute furniture or a part thereof, the detected rectangular area is set as a furniture candidate. For example, the rectangular area detection apparatus 100 or the furniture model generation apparatus 300 may store a plurality of furniture candidates (furniture templates) and select a candidate that matches the detected rectangular area as a furniture candidate.

  The rectangular area detection device 100 specifies the detected rectangular area as a furniture candidate, and the HMI unit 301 displays the specified furniture candidate 400 to the user as shown in FIG. In FIG. 26, rectangular areas 401 and 402 are detected rectangular areas. For example, the rectangular area 401 is a plane of a revolving door, and the rectangular area 402 is a plane of a drawer. Here, an example in which an operation of the rectangular area 401 that is a revolving door is designated will be described.

  Subsequently, the user instructs the operation method of the furniture (furniture candidate) displayed in S202 through the HMI unit 301, and the operation information adding unit 302 adds the operation information (S203).

  In the process of S203, as shown in FIG. 28, the user selects a movable axis through the HMI unit 301 (S211). For example, as shown in FIG. 28, by performing a mouse click or a touch operation on the touch panel, the end (side) of the rectangular area 401 is selected and designated as the rotation axis 401a. In FIG. 28, as an example, by selecting the right end (right side) of the top, bottom, left, and right four ends (four sides) of the rectangular area 401, the selected end (side) rotates about the hinge around the selected end (side). Instruct.

  Next, the user instructs the handle position through the HMI unit 301 (S212). For example, as shown in FIG. 29, the selected portion is designated as the handle position 401b by selecting the upper left portion of the rectangular area 401 by clicking or touching. Note that a handle may be displayed at the handle position 401b when the user selects the handle position 401b. As shown in FIG. 29, when the rotation axis 401a and the handle position 401b are selected, a rotation ring 401c for instructing the rotation amount of the rectangular area 401 is displayed.

  Next, the user instructs the amount of rotation through the HMI unit 301 (S213). For example, as shown in FIG. 30, the “rotation amount 401d around the rotation axis 401a of the rectangular area 401” is instructed by dragging the displayed rotation ring 401c (S213).

  Subsequent to S203 in FIG. 24, the operation amount display unit 303 displays to the user via the HMI unit 301 how the rectangular area moves in response to the “user operation instruction” in S203 (S204). For example, as shown in FIG. 31, when the handle position 401b is operated, it is displayed that the door of the rectangular area 401 is opened with the specified rotation amount 401d around the rotation axis 401a.

  Subsequently, when the user finally finalizes the operation instruction, “the position / shape of the rectangular area” and “user operation instruction information” are stored in the model storage unit 304 (S205). When the user performs an operation indicating that the operation displayed in S204 has been confirmed, the position and shape of the rectangular area 401, the rotation axis 401a, the handle position 401b, and the rotation amount 401d are stored as a furniture model (three-dimensional model). To do.

  Thereafter, the robot uses the furniture model information to operate the furniture. For example, the robot includes a control unit, a storage unit, an actuator, and the like, and stores the generated furniture model information in the storage unit of the robot. The control unit controls the operation of the actuator and the like according to the furniture model information. For example, according to the furniture model information, the robot opens and closes the revolving door in the rectangular area 401 by operating the handle at the handle position 401b as shown in FIG.

  As described above, in the present embodiment, a rectangular model detected from an input image is set as a furniture candidate, and an operation of the furniture candidate is instructed via the HMI, thereby building a furniture model including furniture estimation and structure. It can be done easily.

  In recent years, the aging of society is rapidly progressing, and a life support robot HSR (Human Support Robot) for supporting caregivers and the like has been attracting attention. For example, in general robots, when manipulating objects by remote control for life support, it is difficult to understand the distance to the target object and the space with the environmental object only from the camera image of the robot head. There is a problem that it is necessary to attach a marker for recognizing the image and a problem that it is difficult to adjust the hand of the robot to the target position in the object operation.

  Therefore, by applying this embodiment to such a robot, furniture candidates can present a wide range of environmental information around the robot in an easy-to-understand manner, and by using rectangle detection, furniture without a marker can be provided. By using a furniture model (interactive furniture model) provided with operation information, the robot can accurately recognize and operate the furniture.

  In particular, in order to model the structure of furniture, it is necessary for a human to move the furniture or actually move it with a robot. Therefore, it is very difficult to remotely teach the furniture structure to the robot. By applying this embodiment to a robot, furniture arrangement estimation and structure teaching in an unknown home environment can be facilitated, and remote control can be greatly labor-saving.

  Note that the present invention is not limited to the above-described embodiment, and can be changed as appropriate without departing from the spirit of the present invention.

DESCRIPTION OF SYMBOLS 1 Image processing system 2 Furniture model processing system 100 Rectangular area detection apparatus 101 Distance image acquisition part 102 Plane detection part 103 Plane mask image generation part 104 Plane coordinate system calculation part 105 Edge detection part 106 Edge fragment detection part 107 Coordinate system conversion part 108 Edge presence position histogram generation unit 109 Histogram peak estimation unit 110 Line segment generation unit 111 Neighborhood edge fragment extraction unit 112 Edge collation unit 113 Rectangular area detection unit 200 Distance image sensor 300 Furniture model generation device 301 HMI unit 302 Operation information addition unit 303 Operation Display unit 304 Model storage unit

Claims (17)

An edge pixel detection unit for detecting a plurality of edge pixels in a three-dimensional plane included in the measurement image information;
An edge line segment detection unit for detecting a plurality of edge line segments connecting the detected plurality of edge pixels;
A coordinate system conversion unit that converts the coordinate system of the detected plurality of edge line segments into a plane coordinate system viewed from the normal direction of the three-dimensional plane;
A rectangular area detector for detecting a rectangular area in the three-dimensional plane based on a distribution of a plurality of edge line segments in the planar coordinate system;
An image processing apparatus comprising: The measurement image information includes image information obtained by imaging the three-dimensional plane from a measurement point, and distance information from the measurement point to the three-dimensional plane.
The image processing apparatus according to claim 1. A plane detection unit that detects the three-dimensional plane included in the measurement image information;
The edge pixel detection unit detects the plurality of edge pixels based on the detected three-dimensional plane;
The image processing apparatus according to claim 1. An extracted image generation unit that generates an extracted image obtained by extracting the image of the detected three-dimensional plane region from the measurement image information;
The edge pixel detection unit performs edge detection processing on the generated extracted image and detects the plurality of edge pixels.
The image processing apparatus according to claim 3. The edge line segment detection unit performs a probabilistic Hough transform on the plurality of edge pixels and detects the plurality of edge line segments;
The image processing apparatus according to claim 1. A histogram generation unit that generates a histogram indicating a distribution of positions of a plurality of edge line segments in the planar coordinate system;
The rectangular area detection unit detects the rectangular area based on the generated histogram.
The image processing apparatus according to claim 1. The histogram is a distribution having a weight as an inclination of the plurality of edge line segments with respect to an axial direction of the planar coordinate system.
The image processing apparatus according to claim 6. The histogram weight is an inner product of the plurality of edge line segments and an axial unit vector of the planar coordinate system.
The image processing apparatus according to claim 7. A peak detector for detecting a plurality of peak positions of the histogram;
A peak line segment generation unit that generates a plurality of peak line segments that connect between the detected peak positions in the axial direction of the planar coordinate system, and
The rectangular area detecting unit detects the rectangular area based on the plurality of generated peak line segments;
The image processing apparatus according to claim 6. The rectangular area detection unit detects the rectangular area based on peak line segments corresponding to the plurality of edge line segments among the generated plurality of peak line segments.
The image processing apparatus according to claim 9. An edge matching unit for matching the generated plurality of peak line segments and the plurality of edge line segments;
The rectangular area detection unit detects the rectangular area based on the peak line segment that has been successfully verified.
The image processing apparatus according to claim 9 or 10. A neighborhood edge line segment extraction unit that extracts edge line segments in the vicinity of the generated plurality of peak line segments from the plurality of edge line segments,
The edge matching unit matches the plurality of peak line segments with the extracted edge line segments;
The image processing apparatus according to claim 11. A three-dimensional model generation unit that generates a three-dimensional model that can rotate around one side of the detected rectangular area;
The image processing apparatus according to claim 1. An operation designating unit that designates the rotation axis and the rotation operation with respect to the rectangular region according to a user input operation;
The image processing apparatus according to claim 13. The three-dimensional model generation unit recognizes the detected rectangular area as a furniture candidate, and generates a furniture model as the three-dimensional model.
The image processing apparatus according to claim 13 or 14. Detecting a plurality of edge pixels in a three-dimensional plane included in the measurement image information;
Detecting a plurality of edge line segments connecting the detected plurality of edge pixels;
Converting the detected coordinate system of the plurality of edge line segments into a plane coordinate system viewed from the normal direction of the three-dimensional plane;
Detecting a rectangular region in the three-dimensional plane based on a distribution of a plurality of edge line segments in the plane coordinate system;
Image processing method. Detecting a plurality of edge pixels in a three-dimensional plane included in the measurement image information;
Detecting a plurality of edge line segments connecting the detected plurality of edge pixels;
Converting the detected coordinate system of the plurality of edge line segments into a plane coordinate system viewed from the normal direction of the three-dimensional plane;
Detecting a rectangular region in the three-dimensional plane based on a distribution of a plurality of edge line segments in the plane coordinate system;
An image processing program for causing a computer to execute an image processing method.